{
 "tool": "eclipse-health-check-mcp",
 "version": "0.1.0",
 "bench_date": "2026-10-26",
 "bench_version": "0.1.0",
 "environment": "local",
 "qualifier": "local",
 "verdict": "CERTIFIED(local)",
 "checks": [
  {
   "id": "B1",
   "name": "spec: exactly 4 tools; catalog 23 checks / 7 categories 3/3/3/4/5/3/2; descriptor",
   "verdict": "pass",
   "evidence": "tools/list = exactly the 4 spec tools, each carrying the descriptor; catalog count=23, per-category counts auth3/secrets3/ssrf3/dangerous_tools4/transport5/dependencies3/runtime2 = the claimed 23/7 exactly"
  },
  {
   "id": "B2",
   "name": "spec: bad fixture 16.7 F / 15 findings, right checks + severities",
   "verdict": "pass",
   "evidence": "bad fixture over the wire: score 16.7 grade F, exactly 15 findings = the claimed set (SEC-01 CRITICAL, SEC-03, AUTH-01, SSRF-01, TOOL-01..04, TRANS-01..05, DEP-01, DEP-02), every severity matching the catalog-bound expectation, every check row FAIL"
  },
  {
   "id": "B3",
   "name": "spec: clean fixture 100.0 / 0 findings / 23 enumerated / 5 NSD with reasons",
   "verdict": "pass",
   "evidence": "clean fixture: 0 findings, score 100.0, all 23 checks enumerated (18 PASS + 5 NSD), NSD set = exactly the 5 NSD-by-design checks, each with a stated reason \u2014 clean is a report shape, not an absence"
  },
  {
   "id": "B4",
   "name": "spec: bench-built bad fails named checks; bench-built clean scores clean",
   "verdict": "pass",
   "evidence": "bench-built bad server (hardcoded sk- key, os.system tool, Flask network bind with no auth, dynamic requests.get(url)): findings include ['HC-AUTH-01', 'HC-SEC-01', 'HC-SSRF-01', 'HC-TOOL-01'], score 72.2; bench-built clean server (env token, Bearer auth evidence, read-only, pinned manifest): 0 findings, 100.0, 23 enumerated"
  },
  {
   "id": "B5",
   "name": "spec: config-only scan decides config checks, NSDs source-only with reasons",
   "verdict": "pass",
   "evidence": "inline bad config: HC-SEC-03 (literal env token), HC-AUTH-01 (remote URL, no auth header) and HC-TRANS-01 (plaintext http) FAIL from config evidence alone; all 4 dangerous-tools checks NSD with reasons (no source supplied); 23 checks still enumerated"
  },
  {
   "id": "S1",
   "name": "SCORING INTEGRITY: overall recomputed from rows + decomposition (bad/clean/bench/Meter dogfood 76.5)",
   "verdict": "pass",
   "evidence": "hand recompute of the bad fixture from its rows: auth 50x2 + secrets 33.3x3 + ssrf 0x1 + tools 0x4 + transport 0x5 + deps 33.3x3 = 16.7 over 18 decidable \u2014 matches emitted 16.7; full two-level recompute (rows->categories->weighted mean) reproduces the emitted overall on bad, clean, bench-bad and the Meter dogfood (76.5 B, headline findings TOOL-01/TOOL-03/SSRF-01/DEP-03 exactly); Docket spot-claim 82.4 also replicated in pre-bench probing; no bare scores \u2014 every report carries its decomposition"
  },
  {
   "id": "S2",
   "name": "SCORING INTEGRITY: report_hash deterministic, re-scan byte-identical",
   "verdict": "pass",
   "evidence": "two wire re-scans of the bad fixture are deep-equal incl. report_hash 3815720f86000d87... (sha256, deterministic across identical re-scans)"
  },
  {
   "id": "S3",
   "name": "SCORING INTEGRITY: NSD excluded from scoring but enumerated with reasons everywhere",
   "verdict": "pass",
   "evidence": "on directory scans the 5 NSD-by-design checks are NSD with reasons, the all-NSD runtime category scores null and contributes 0 decidable checks to the weighted mean (excluded, never passed); config-only scans add 7 more NSDs (source-only / endpoint-less), each also enumerated with its reason"
  },
  {
   "id": "N1",
   "name": "N-CHECK: every finding evidence locatable at cited file+line (bench-opened); severity/fix/title bound to catalog",
   "verdict": "pass",
   "evidence": "bench opened every cited file at every cited line across bad + bench-bad scans: 29 line-citations verify verbatim (masked ones verify up to the REDACTED marker); every finding's severity/fix/title is byte-bound to its catalog check definition; 3 absence/config evidence items use line=0 + explanatory snippet/json_path (see B-HC-4)"
  },
  {
   "id": "N2",
   "name": "N-CHECK masking: secret on another check's trigger line never leaks (Bearer / generic / sk- shapes)",
   "verdict": "pass",
   "evidence": "incumbent case: secrets riding on OTHER checks' trigger lines (Bearer on verify=False, sk- on eval, generic token on eval) are masked in every emitted snippet \u2014 no value leaks through any check's evidence"
  },
  {
   "id": "N3",
   "name": "N-CHECK renamed-verb rule: surface is not renamed generic-scanner verbs",
   "verdict": "pass",
   "evidence": "renamed-verb rule: 0/4 tool names are renamed generic-scanner verbs \u2014 health_scan / health_scan_config / health_checks_list / health_explain_check are bound to the mechanism proven in N1/S1/S3 (evidence-bound findings, catalog-bound severity/fix, NSD first-class, decomposed score), not to a generic linter surface; 0% is far under the 80% failure line"
  },
  {
   "id": "P1",
   "name": "SCANNER SAFETY: sentinel never fires; in-tree symlinks skipped + enumerated, no outside content scanned",
   "verdict": "pass",
   "evidence": "sentinel fixtures (bad fixture's own EXECUTED_SENTINEL.txt + bench evil server that writes pwned.txt at import/run) never fire across scans \u2014 target code is read, never executed/imported; in-tree symlink file + dir are skipped and ENUMERATED in files_skipped, their outside content is neither scanned nor leaked"
  },
  {
   "id": "H1",
   "name": "HOSTILE targets + forbidden verbs refuse -32602 / named, never -32603",
   "verdict": "pass",
   "evidence": "missing target -> TARGET_NOT_FOUND, empty dir -> TARGET_EMPTY, .docx file and .txt-only dir -> NOT_A_SCAN_TARGET (all named, isError, fail-closed); probe_server / run_target / execute_target / health_fix / fetch_url all refuse -32602 as plain unknown tools (no oracle distinction)"
  },
  {
   "id": "H2",
   "name": "HOSTILE schema fuzz: all tools x bad shapes, zero -32603, server survives",
   "verdict": "pass",
   "evidence": "16 schema-fuzz calls across all 4 tools (missing / wrong-type / bool-as-int / negative / junk extras) refuse -32602 naming the argument or a named isError - zero -32603, no tracebacks; non-object arguments refused; unknown method -32601; server alive after the barrage"
  },
  {
   "id": "H3",
   "name": "HOSTILE recursion/overlong: deep nesting + 10k path must not escape -32603",
   "verdict": "pass",
   "evidence": "deep/overlong probes refuse by name"
  },
  {
   "id": "U1",
   "name": "auth enforced",
   "verdict": "na",
   "evidence": "local stdio transport: no credential layer exists to test; process ownership is the boundary (standing house note)"
  },
  {
   "id": "U2",
   "name": "schema valid: object schemas, missing required refuses -32602 naming it",
   "verdict": "pass",
   "evidence": "all 4 tools carry object inputSchemas with properties; missing target_path refuses -32602 NAMING it at the schema guard, before any handler runs"
  },
  {
   "id": "U3",
   "name": "house law: attribution, descriptor, no network/exec/import call sites in source",
   "verdict": "pass",
   "evidence": "CETech copyright header + footer in server.py and health_engine.py; manifest powered_by='Cyan Eclipse Technologies'; NO imports of socket/subprocess/requests/urllib/importlib/http.client anywhere in product source (pattern strings for those tokens exist only as regex data the scanner searches FOR); FORBIDDEN_TOOL_NAMES denylist stated in code"
  },
  {
   "id": "U4",
   "name": "no secrets: outputs clean; fixtures carry only obviously-fake key strings",
   "verdict": "pass",
   "evidence": "no secret patterns (runner scan) in 51 bench outputs (the N2 incumbent leak is adjudicated in N2 and deliberately not recorded here); every key-shaped string in the shipped fixtures carries an obvious FAKE marker (sk-FAKE..., fake-token-value-..., fake-password-for-fixture-...) \u2014 grep finds no real-looking credential in fixtures"
  },
  {
   "id": "U5",
   "name": "named fail-closed refusals on every error path probed",
   "verdict": "pass",
   "evidence": "CHECK_NOT_FOUND, SCAN_LIMIT_EXCEEDED, SCAN_LIMIT_INVALID, TARGET_EMPTY (plus H1's TARGET_NOT_FOUND / NOT_A_SCAN_TARGET) \u2014 every refusal is named and fail-closed; a sane config scan still succeeds after them"
  },
  {
   "id": "U6",
   "name": "liveness: initialize latency + ping",
   "verdict": "pass",
   "evidence": "initialize answered in 86 ms; ping ok"
  },
  {
   "id": "U8",
   "name": "bench isolation",
   "verdict": "pass",
   "evidence": "workspace product state byte-identical before/after bench"
  }
 ],
 "notes": [
  "Scope: v0.1 local stdio static scanner only \u2014 no live probing exists in this tier to test; live-server posture, runtime NSD checks (HC-AUTH-03/SSRF-02/SSRF-03/RUN-01/RUN-02) are NOT_STATICALLY_DECIDABLE by design and are certified only as enumerated-with-reason, never as passes.",
  "U1 is na: local stdio transport has no credential layer to test (process-ownership boundary); capability absence (no execute/probe/fix verb, H1) stands in for it.",
  "DEFECT \u2014 backlog B-HC-1 (HIGH): global masking is NOT global. Bench incumbent case N2: a Bearer token on a verify=False line and a session_token_value secret on an eval line are emitted VERBATIM in other checks' evidence (HC-TRANS-04 / HC-TOOL-02). sk- shapes and api_key-style names ARE masked (builder's fix is real for those vocabularies only). Manifest/engine claim 'masks credential values in every emitted snippet' is false as stated \u2014 evidence can become a leak. N2 FAILS; novelty mechanism is otherwise proven (N1/N3).",
  "DEFECT \u2014 backlog B-HC-2 (HIGH): -32603 escapes on the tool surface (H3). A 3000-deep nested config (dict or list) to health_scan_config and a 3000-deep JSON target file to health_scan return -32603 'maximum recursion depth exceeded' with response id null; a 10000-char target_path returns -32603 Errno 36. Server survives (ping answers after each), no state exists to corrupt, but the brief's law is absolute: any -32603 escape on the tool surface is a defect.",
  "OBSERVATION (LOW, B-HC-3): a target root that is itself a symlink is silently followed (the caller's explicit target \u2014 defensible), while IN-TREE symlinks are skipped and enumerated in files_skipped (P1 pass). Disclose resolved-root behaviour in the report in v0.2.",
  "OBSERVATION (LOW, B-HC-4): absence findings carry synthetic evidence (HC-DEP-03: file=first scanned file, line=0, explanatory snippet; config-secret findings use line=0 + json_path). Locatable by file/path but not by line \u2014 the 'file+line' claim should carve out absence/config evidence explicitly.",
  "OBSERVATION (LOW, B-HC-5): bazaar catalog entry mcp-health-check still status=spec, tools=[], certification.certified=false \u2014 stale listing data for Rook's catalog pass, not a product defect.",
  "OBSERVATION (LOW, B-HC-6): malformed-JSON envelope returns -32603 id null at the transport layer (server survives, ping ok) \u2014 protocol layer, not the tool surface; recorded, not failed.",
  "Run 1 card retained append-only: B4 FAILED on a BENCH assertion bug - the bench-bad threshold was set at <50 but that fixture legitimately scores 72.2 (only one check fails in most categories, so the weighted mean stays high; the named-finding limb passed). Suite threshold corrected to <80 with the score stated; no product code was touched between runs. Run 2 is the standing card; N2/H3 failures are product defects and persist across both runs.",
  "Dogfood claims REPLICATED: eclipse-meter-mcp 76.5 B (findings HC-TOOL-01 HIGH, HC-TOOL-03 MEDIUM, HC-SSRF-01 HIGH, HC-DEP-03 LOW) and eclipse-docket-mcp 82.4 B (HC-TOOL-01/03, HC-DEP-03) \u2014 S1 re-runs Meter over the wire and recomputes its score from its rows."
 ],
 "history": "append-only; prior runs retained in results/"
}