{
 "tool": "eclipse-bidmatrix-mcp",
 "version": "0.1.0",
 "bench_date": "2026-10-26",
 "bench_version": "0.1.0",
 "environment": "local",
 "qualifier": "local",
 "verdict": "CERTIFIED(local)",
 "checks": [
  {
   "id": "B1",
   "name": "spec: full shred of sample \u2014 sections, matrix, checklist, deadlines, set-asides, gap, stats",
   "verdict": "pass",
   "evidence": "full shred of the sample: 11 matrix rows incl. Section L + Section M, deadlines {2027-03-17, 2027-03-03}, both date-flag kinds present, SILENT_EMPTY gap incl. Section H, set-aside summary with NAICS 541512 and size standard '$34.0 million'"
  },
  {
   "id": "B2",
   "name": "spec: single-concern tools return their slice, consistent with the full shred",
   "verdict": "pass",
   "evidence": "each single-concern tool returns exactly its slice of the full shred, byte-equal (matrix+gap, checklist, deadlines+flags, set-aside summary); compliance_matrix carries no deadlines key \u2014 the 5-tool surface behaves per spec \u00a71 (L/M extraction, matrix, checklist, deadlines, set-asides)"
  },
  {
   "id": "B3",
   "name": "evidence binding across bench fixtures (bullets, CRLF, BOM): every quote substring-true + locator line correct",
   "verdict": "pass",
   "evidence": "programmatic substring verification over 39 matrix rows across 4 bench documents (bullet styles -/*/\u2022/numbered, CRLF, BOM-prefixed, sample): every requirement IS its source_quote, every quote is a true substring of the source, every line locator points at the line containing the quote; deadline and set-aside quotes likewise verbatim \u2014 zero fabricated/paraphrased requirements"
  },
  {
   "id": "B4",
   "name": "set-aside summary: type variants, NAICS, size standard verbatim",
   "verdict": "pass",
   "evidence": "all set-aside variants correct"
  },
  {
   "id": "B5",
   "name": "checklist mirrors matrix (ids, quotes, done=false)",
   "verdict": "pass",
   "evidence": "checklist mirrors the matrix 1:1 (11 items): same ids, same verbatim quotes, same line locators, done=false on every item"
  },
  {
   "id": "B6",
   "name": "deadlines: strict formats parse; impossible / canonical vague+relative shapes flag by name",
   "verdict": "pass",
   "evidence": "strict formats parse to ISO (long + abbreviated month-first, long day-first, ISO, US slash); impossible dates (Feb 30, 2027-13-45), the canonical relative ('30 days after award'), on-or-about and NLT-vague shapes all flag by name with NO fabricated deadline entry"
  },
  {
   "id": "B7",
   "name": "HOSTILE deadline traps: vague/relative dates must flag, never fabricate, never vanish; unambiguous ordinal + abbreviated day-first parse (adjudicated run 3)",
   "verdict": "pass",
   "evidence": "all 6 vague/relative traps flag by name with NO fabricated date (soon, TBD, next Friday, close of business, within 30 days of award, 2-digit-year March 17, 27); the 2 unambiguous shapes the run-2 suite wrongly bucketed as fabrication \u2014 ordinal 'March 17th, 2027' and abbreviated day-first '17 Mar 2027' (declared format %d %b %Y) \u2014 parse to exactly the date present, 2027-03-17 (suite expectation corrected run 3, fixer adjudicated right)"
  },
  {
   "id": "B8",
   "name": "completeness pass under repeated section ids",
   "verdict": "pass",
   "evidence": "repeated section ids are occurrence-keyed: explicit-H doc yields 3 sections [1, 1, 0] and descriptive-C doc yields 2 sections [1, 0]; each empty occurrence counted 0 and emits exactly one SILENT_EMPTY gap (run-2 suite wrongly demanded 3 sections for the 2-heading C doc \u2014 expectation corrected run 3)"
  },
  {
   "id": "B9",
   "name": "deadline time_verbatim carries the timezone verbatim",
   "verdict": "pass",
   "evidence": "time_verbatim carries timezone verbatim"
  },
  {
   "id": "H1",
   "name": "HOSTILE fuzz: every tool x every field x bad shapes over the wire \u2014 named refusals, zero -32603",
   "verdict": "pass",
   "evidence": "261 wire fuzz cases (5 tools x 3 fields x 17 bad shapes + garbage argument containers): every case refused by name (-32602 schema error naming the argument, or a result body carrying a SCREAMING_CASE ToolError); ZERO -32603 escapes, ZERO fabricated successes"
  },
  {
   "id": "H2",
   "name": "HOSTILE path tricks: ENAMETOOLONG, /etc/passwd, dir, device, bad UTF-8, traversal/symlink",
   "verdict": "pass",
   "evidence": "path battery: ENAMETOOLONG (5000-char path) -> FILE_NOT_FOUND (builder fix verified, no OSError/-32603 escape); /etc/passwd -> NO_SOLICITATION_STRUCTURE with zero content leak; directory and /dev/null -> FILE_NOT_A_FILE; invalid UTF-8 -> FILE_UNREADABLE; missing file with ../ -> FILE_NOT_FOUND; symlink and ../ traversal to a real solicitation shred normally (by-design local scope, recorded in NOTES)"
  },
  {
   "id": "H3",
   "name": "HOSTILE sources: both/neither/empty/huge documents refuse by name",
   "verdict": "pass",
   "evidence": "ambiguous dual source -> AMBIGUOUS_DOCUMENT_SOURCE; neither -> MISSING_DOCUMENT; empty/whitespace -> EMPTY_DOCUMENT; 2.2M-char document -> DOCUMENT_TOO_LARGE (2,000,000 limit); prose with no solicitation signal -> NO_SOLICITATION_STRUCTURE \u2014 all named, all fail-closed, no empty plausible matrix ever returned"
  },
  {
   "id": "H4",
   "name": "forbidden fetch/submit verbs refuse in the same -32602 shape as unknown tools",
   "verdict": "pass",
   "evidence": "30 forbidden names (builder denylist: fetch_solicitation, sam_gov_fetch, download_pdf, fetch_url, submit_proposal/bid, file_proposal, publish, send, email, pay, purchase, sign, legal_advice, eligibility_determination, award + bench additions get_solicitation, upload_document, scrape_url, http_get, award_contract et al.) plus unknown/empty/case-variant names: every one refused -32602 in the identical 'unknown tool' shape \u2014 no oracle distinguishing forbidden from nonexistent, no fetch or submit verb exists"
  },
  {
   "id": "H5",
   "name": "builder claim: max_requirements refusal layering",
   "verdict": "pass",
   "evidence": "builder claim VERIFIED \u2014 layering is exactly as stated: non-numeric string 'abc' and out-of-range 0/-1/99999 ride to the engine and refuse MAX_REQUIREMENTS_INVALID by name; wrong-typed shapes (2.5, [3], True, dict) refuse at the schema guard -32602 NAMING max_requirements; int-shaped string '3' is accepted and truncates to 3 rows with truncated=true; boundary 5000 accepted"
  },
  {
   "id": "H6",
   "name": "HOSTILE protocol envelopes: malformed initialize params / non-object / garbage line \u2014 no -32603 escape",
   "verdict": "pass",
   "evidence": "malformed envelopes fail closed with clean protocol errors, zero -32603, no raw exception text: initialize params 'x'/[]/42 -> -32602 'params must be an object'; garbage line -> -32700 Parse error; non-object line 42 -> -32600 Invalid Request; server alive and in sync after all of it"
  },
  {
   "id": "H7",
   "name": "injection text is data; NUL/unicode/surrogate inputs survive without -32603",
   "verdict": "pass",
   "evidence": "prompt-injection text inside the document is stored VERBATIM as a matrix quote (data, not instruction) and every checklist item still ships done=false; NUL/control + unicode document shreds normally; a lone-surrogate document produces no -32603; server alive after all of it"
  },
  {
   "id": "N1",
   "name": "N-CHECK: evidence-bound rows + SILENT_EMPTY gap on incumbent-shaped fixture",
   "verdict": "pass",
   "evidence": "N-CHECK \u2014 novel mechanism PROVEN on a bench-built incumbent-shaped fixture (5 sections, 2 silent): every one of the 5 rows is a verbatim substring of the source with requirement == source_quote and a correct section + line locator (programmatically re-verified); the completeness pass emits SILENT_EMPTY for EXACTLY the two silent sections (H reserved-by-amendment, Q administrative annex) with line spans and zero counts, and flags no productive section. A generic summarizer fails both limbs: it paraphrases (substring check kills it) and it silently drops empty sections (gap report kills it)"
  },
  {
   "id": "N2",
   "name": "N-CHECK renamed-verb rule: surface is not renamed generic-summarizer verbs",
   "verdict": "pass",
   "evidence": "renamed-verb rule: 0/5 tool names are renamed generic-summarizer verbs (every name carries a domain noun \u2014 solicitation, compliance_matrix, requirements, deadline, set_aside; the two generic tokens present, extract/summary, are bound to domain objects, never standalone) \u2014 far under the 80-percent failure line; combined with N1's substring + gap evidence the surface is a mechanism, not a mirror"
  },
  {
   "id": "U1",
   "name": "auth enforced",
   "verdict": "na",
   "evidence": "local stdio transport: no credential layer exists to test; process ownership is the boundary (standing house note)"
  },
  {
   "id": "U2",
   "name": "schema valid: exactly 5 tools, object schemas",
   "verdict": "pass",
   "evidence": "tools/list returns exactly the 5 spec tools, each with an object inputSchema + properties; initialize returns serverInfo eclipse-bidmatrix 0.1.0"
  },
  {
   "id": "U3",
   "name": "no network/git/publish behavior (source-level)",
   "verdict": "pass",
   "evidence": "source-level: server.py + bidmatrix_engine.py contain no network call sites (urllib/socket/requests/http.client/urlopen), no subprocess, no os.system/popen, no eval/exec, and no file-write calls \u2014 the only file access is Path.read_text of the caller-supplied path; the capability denylist (FORBIDDEN_TOOL_NAMES) is stated in code; no git/publish behavior exists in the product"
  },
  {
   "id": "U4",
   "name": "no secret leak",
   "verdict": "pass",
   "evidence": "no secret patterns in 510 bench outputs"
  },
  {
   "id": "U5",
   "name": "sane errors: named fail-closed refusals, -32601/-32602 shapes",
   "verdict": "pass",
   "evidence": "unknown method -> -32601; missing document -> MISSING_DOCUMENT; empty -> EMPTY_DOCUMENT; wrong-typed document_text -> -32602 schema refusal naming the argument \u2014 every error is named and fail-closed, no stack traces on the tool surface"
  },
  {
   "id": "U6",
   "name": "process health",
   "verdict": "pass",
   "evidence": "initialize answered in 73 ms; ping ok"
  },
  {
   "id": "U7",
   "name": "attribution: headers/footers + powered-by descriptor",
   "verdict": "pass",
   "evidence": "attribution headers on all 7 product .py files + footers on every file >40 lines; server.py carries the 'Powered by Cyan Eclipse Technologies (cetech.ai)' line (the house pattern \u2014 cf. Meter's certified server); the bazaar manifest states powered_by Cyan Eclipse Technologies and lists exactly the 5 tools"
  },
  {
   "id": "U8",
   "name": "bench isolation",
   "verdict": "pass",
   "evidence": "workspace product state byte-identical before/after bench"
  }
 ],
 "notes": [
  "Scope: v0.1 local stdio only \u2014 caller-supplied text / local file path. No live SAM.gov fetch, PDF parsing, payment, or hosted surface exists to test; catalog entry still lists the product as spec-stage with tools=[] (stale listing data, not a product defect \u2014 bench backlog for the catalog pass).",
  "RE-BENCH run 3 (post D1-D6 fix): FULL suite re-run, not just failed checks \u2014 no regressions anywhere outside the two adjudicated suite expectations. Builder claims REPLICATED: own unittest suite 47/47 pass (incl. 21 new regression tests in tests/test_bench_regressions.py), smoke passes. Fixes additionally verified on the wire with bench-built fixtures independent of the builder's tests (43/44 probes; the 1 miss is new backlog D7 below).",
  "D1 VERIFIED FIXED: 8(a) matches in prose positions \u2014 mid-sentence 'This is an 8(a) set-aside procurement.', sentence-start, 'Reserved for 8(a) participants.', 'set aside for 8(a) firms', bare 8(a) mention.",
  "D2 VERIFIED FIXED (mandated shapes): multi-code NAICS lists extract EVERY code \u2014 'NAICS codes: 541512 and 541513', comma/and 3-code lists, singular anchor with 2 codes, 'NAICS:' anchor.",
  "D3 VERIFIED FIXED: all 6 true vague/relative traps flag by name with no fabricated date (soon, TBD, next Friday, close of business, within 30 days of award, March 17, 27); bench extra traps (tomorrow, ASAP, to be determined, in two weeks, end of month) also flag. See B7 adjudication.",
  "D4 VERIFIED FIXED: completeness is occurrence-keyed \u2014 bench Hx3 fixture (1 productive + 2 empty annexes) yields counts [1, 1, 0, 0] and exactly 2 SILENT_EMPTY gaps, one per empty occurrence; the sample now yields exactly 7 sections (Section M double-split gone), 11 matrix rows intact. See B8 adjudication.",
  "D5 VERIFIED FIXED: time_verbatim keeps 2-letter zones verbatim \u2014 ET/PT/CT/MT all return '2:00 PM <zone>'; EST/PST/EDT/UTC still work; sample keeps '2:00 PM ET'.",
  "D6 VERIFIED FIXED: malformed protocol envelopes fail closed with clean protocol errors and NO raw exception text \u2014 initialize params 'x'/[]/42 -> -32602 'params must be an object'; garbage line -> -32700 Parse error; non-object lines (42, string, array) -> -32600 Invalid Request; zero -32603 escapes; server alive after all of it.",
  "SUITE-CONFLICT ADJUDICATION (a) B7 \u2014 FIXER RIGHT: the run-2 'fabricated' bucket contained exactly the 2 parses the fix brief mandated: ordinal 'March 17th, 2027' and abbreviated day-first '17 Mar 2027', both -> 2027-03-17, the unambiguous date verbatim-present in the text (the latter is the product's declared strict format %d %b %Y, which run 2's own D3 note said should parse). Parsing an unambiguous present date is not fabrication. Suite expectation corrected in c_b7 (6 flag-traps + 2 parse-cases).",
  "SUITE-CONFLICT ADJUDICATION (b) B8 \u2014 FIXER RIGHT: the run-2 descriptive-C fixture has 2 headings (Section C + Scope of Work - Annex), so the suite's demand for 3 sections was arithmetically wrong; the product's occurrence-keyed output [1, 0] + one SILENT_EMPTY gap is correct. Suite expectation corrected in c_b8 (per-fixture counts 3 and 2).",
  "NEW DEFECT D7 (minor, bench backlog, non-blocking): NAICS lists separated only by ' / ' extract just the first code ('NAICS codes: 541512 / 541519' -> [541512]), and the unanchored phrasing 'Applicable NAICS codes are 541512 and 541513' extracts none. Outside the engine's stated anchored scope ('NAICS code:' / 'NAICS codes:' lists) \u2014 logged for the next fix wave, does not reopen D2.",
  "Prior observations carried (not defects): (a) sample's Total Small Business line also emits a redundant generic 'Small Business Set-Aside' hit; (b) stats.truncated true when max_requirements exactly equals the count; (c) undeclared arguments tolerated by shred but honored by the shared engine on deadline_extract (fail-closed); (d) file_path symlink/../ reads by design in local v0; (e) ambiguous slash dates parse US month-first per the declared format list.",
  "Run 2 card retained append-only (results/eclipse-bidmatrix-mcp-2026-10-26-2.json); its D1-D6 defect notes are historical. Run 1 card likewise retained.",
  "Run -3 card (results/eclipse-bidmatrix-mcp-2026-10-26-3.json) was a HARNESS flake, retained append-only: c_h6's fixed-window drain() raced after the garbage-line probes and the response queue desynced, cascading false fails into H7/N1/N2/U5/U6 (each check read the previous request's response). Product evidence in that run (B1-B9, H1-H6) matches runs 2 and 4 exactly; c_h6 hardened to consume until the ping response itself is seen; run 4 is the standing card.",
  "N-check ruling STANDS: NOVELTY_PROVEN \u2014 N1/N2 pass again; with D4 fixed, the completeness limb now also holds under repeated section ids, strengthening (not merely restoring) the mechanism evidence."
 ],
 "history": "append-only; prior runs retained in results/"
}