{
 "tool": "eclipse-fedscope-mcp",
 "version": "0.1.0",
 "bench_date": "2026-10-26",
 "bench_version": "0.1.0",
 "environment": "local",
 "qualifier": "local",
 "verdict": "CERTIFIED(local)",
 "checks": [
  {
   "id": "B1",
   "name": "spec: exactly 4 read-only tools, initialize descriptor",
   "verdict": "pass",
   "evidence": "tools/list returns exactly the 4 spec tools (opportunity_search, opportunity_score, opportunity_summarize, validate_dataset), each description carrying the 'Not a raw API wrapper' descriptor; initialize returns eclipse-fedscope 0.1.0"
  },
  {
   "id": "B2",
   "name": "spec: bundled search \u2014 counts, quarantine, top hit, filters, sort",
   "verdict": "pass",
   "evidence": "bundled fixture over the wire: 8 records, 6 scored, exactly 2 quarantined at indices [5,6] each named RECORD_MALFORMED (builder claim verified); top hit fs-fixture-001 fit=100 newcomer=95; results sorted by fit desc; query / naics_filter / min_fit_score filters each isolate exactly 1 record; validate_dataset agrees (6 valid, 2 quarantined)"
  },
  {
   "id": "B3",
   "name": "SCORING INTEGRITY fit axis: bench-hand-computed per-factor points across all bands",
   "verdict": "pass",
   "evidence": "fit axis hand-recomputed on 34 bench cases (NAICS 35/28/12/0/missing, set-aside 25/15/0/8, runway band edges -1..45d incl. floor + bare-date limbs, past-performance required/evaluated/none x has/none incl. explicit-field form, PoP 10/6/0/4): every per-factor point matches the documented bands, factor sums equal totals, no factor exceeds its max, every factor carries a reason"
  },
  {
   "id": "B4",
   "name": "SCORING INTEGRITY newcomer axis: own 5 factors, separate from fit",
   "verdict": "pass",
   "evidence": "newcomer axis hand-recomputed on 24 bench cases (access 25/15/0, barrier 25/12/3, runway mirrors the band table, openness 15/13/12/10/2/8 by notice type, scale 15/10/5/8 by value incl. '$1,200,000' string + garbage/missing flags): all points match, sums equal the newcomer total, factor names are disjoint from the fit axis \u2014 a separate axis, never folded into fit"
  },
  {
   "id": "B5",
   "name": "determinism: repeated wire runs byte-identical",
   "verdict": "pass",
   "evidence": "3 repeated wire runs of bundled opportunity_search + opportunity_score at a fixed as_of produce exactly 2 distinct payload texts \u2014 every re-run BYTE-IDENTICAL to its first run (stronger than the smoke's dict-equality claim)"
  },
  {
   "id": "B6",
   "name": "profile independence: WOSB construction + past performance flips scores; no CETech hardcoding",
   "verdict": "pass",
   "evidence": "non-CETech profile (WOSB construction, WITH past performance, TX) flips fs-fixture-003 exactly as hand-computed: NAICS 0->35 (primary), set-aside 0->25 (SDVOSB held), past-performance 2->10 (has it), PoP ->10 (home state TX); fs-fixture-001 (VOSB) flips the other way to SET_ASIDE_INELIGIBLE with fit < 100; engine+server source contains NO CETech NAICS code, state, or fixture name as a scoring constant \u2014 no hardcoding"
  },
  {
   "id": "B7",
   "name": "PoP country-only match: reason + flag must not misstate profile inputs",
   "verdict": "pass",
   "evidence": "D2 FIXED: country-only USA vs profile home_country=US/home_state=FL -> points=5 (neutral 5 unchanged), reason states the country-granularity match truthfully, no PROFILE_LOCATION_UNSPECIFIED flag: \"record country USA matches the profile's home country US at country granularity; state comparison unavailable because the record is country-only \u2014 neutral\""
  },
  {
   "id": "H1",
   "name": "HOSTILE fuzz: every tool x every param x bad shapes \u2014 zero -32603 on tool surface",
   "verdict": "pass",
   "evidence": "294 wire fuzz cases (4 tools x every declared param x 18 bad shapes + garbage argument containers): every case refused by name (-32602 schema error naming the argument, or an isError body carrying a SCREAMING_CASE refusal); the only successes are semantically inert (null=unset, string filters, min_fit_score=0, validate reporting a quarantine); ZERO -32603 escapes on the tool surface; server alive"
  },
  {
   "id": "H2",
   "name": "HOSTILE datasets/profiles: quarantine, all-malformed, empty, profile refusals, has_past_performance never defaults",
   "verdict": "pass",
   "evidence": "20 hostile dataset/profile probes, all named + fail-closed: EMPTY_DATASET (empty + absent), DATASET_INVALID (object without a records list), DATASET_ALL_MALFORMED, PROFILE_REQUIRED (explicit null; missing refuses -32602 naming the argument at the schema guard), PROFILE_INVALID for pp-absent / pp-string / pp-int \u2014 has_past_performance NEVER defaults \u2014 plus bad NAICS / set_asides shapes; RECORD_REQUIRED / RECORD_MALFORMED; LIMIT + MIN_FIT range/invalid guards; mixed dataset quarantines indices [1,2] and still returns 2 scored results (per-record quarantine holds)"
  },
  {
   "id": "H3",
   "name": "HOSTILE deadline traps: naive/unparseable/past + as_of traps refuse or flag by name",
   "verdict": "pass",
   "evidence": "11 deadline traps: unparseable ('TBD', int) -> DEADLINE_UNPARSEABLE + 0 runway + no fabricated days; offset-naive -> DEADLINE_NAIVE_NO_OFFSET (no offset ever guessed); past -> DEADLINE_PASSED; missing -> DEADLINE_MISSING; the 3 bundled fixture traps flag identically; as_of naive/garbage/impossible refuse -32602 naming AS_OF_OFFSET_REQUIRED / AS_OF_UNPARSEABLE"
  },
  {
   "id": "H4",
   "name": "forbidden fetch/live/write verbs refuse as plain unknown tools",
   "verdict": "pass",
   "evidence": "27 forbidden names (fetch/fetch_opportunities/sam_search_live/download/submit_bid/apply/create/write/delete/set_profile/api_key_set/login + bench additions get_opportunities, search_sam, http_get, scrape_url, publish, pay, empty, nonsense): every one refused -32602 in the identical 'unknown tool' shape \u2014 no oracle, and NO fetch/live/write verb exists on the surface at all"
  },
  {
   "id": "H5",
   "name": "HOSTILE protocol envelopes: no -32603 escape",
   "verdict": "pass",
   "evidence": "D3 FIXED: initialize params str/list/int refused -32602 (params must be an object), garbage line -32700 Parse error, non-object line '42' -32600 Invalid Request \u2014 ZERO -32603 escapes, no raw exception text; server alive"
  },
  {
   "id": "N1",
   "name": "N-CHECK: traced summaries, never-invent, named flags on every summary surface",
   "verdict": "pass",
   "evidence": "all summary limbs hold on ALL surfaces: rich-traced, no-description-leak, sparse-flags-no-invention, score-flags, summarize-with-profile, and standalone-summarize-flags (D1 FIXED: 'TBD' -> DEADLINE_UNPARSEABLE, naive -> DEADLINE_NAIVE_NO_OFFSET, past -> DEADLINE_PASSED, no profile required); never-invent holds everywhere"
  },
  {
   "id": "N2",
   "name": "N-CHECK renamed-verb rule + no fetch/live verb exists",
   "verdict": "pass",
   "evidence": "renamed-verb rule: 0/4 tool names are renamed raw-search/fetch verbs (the surface verbs \u2014 search, score, summarize, validate \u2014 are bound to the scoring/tracing mechanism proven in B3/B4/N1 limbs i-iv, not to a raw SAM.gov wrapper: there is no fetch, no live call, and no API key anywhere in the surface) \u2014 far under the 80% failure line"
  },
  {
   "id": "U1",
   "name": "auth enforced",
   "verdict": "na",
   "evidence": "local stdio transport: no credential layer exists to test; process ownership is the boundary (standing house note)"
  },
  {
   "id": "U2",
   "name": "schema valid: object schemas, required fields enforced by name",
   "verdict": "pass",
   "evidence": "all 4 tools carry object inputSchemas with properties; a missing required argument refuses -32602 NAMING it (profile) at the schema guard, before any handler runs"
  },
  {
   "id": "U3",
   "name": "no network behavior (source-level)",
   "verdict": "pass",
   "evidence": "source-level: server.py + fedscope_engine.py contain no network call sites (urllib/socket/requests/http.client/urlopen/api.sam.gov), no subprocess, no os.system/popen, no eval/exec, no os.environ (no API key is read anywhere), and no file-write calls \u2014 the only file read is the bundled fixture; the FORBIDDEN_TOOL_NAMES denylist is stated in code"
  },
  {
   "id": "U4",
   "name": "no secret leak: outputs + source/fixtures key-shape grep",
   "verdict": "pass",
   "evidence": "no secret patterns in 430 bench outputs, and a key-shape grep (private-key blocks, AKIA/sk-/ghp_ tokens, api_key/secret/password assignments) over all product .py/.json/.md incl. fixtures finds nothing \u2014 the only api_key-shaped strings are the forbidden-verb NAME 'api_key_set' and smoke's dummy refusal probe"
  },
  {
   "id": "U5",
   "name": "sane errors: named fail-closed refusals, -32601 shape",
   "verdict": "pass",
   "evidence": "unknown method -> -32601; missing dataset -> EMPTY_DATASET; empty profile object -> PROFILE_INVALID \u2014 every error is named and fail-closed, no stack traces on the tool surface"
  },
  {
   "id": "U6",
   "name": "process health",
   "verdict": "pass",
   "evidence": "initialize answered in 56 ms; ping ok"
  },
  {
   "id": "U7",
   "name": "attribution: headers/footers + powered-by + manifest",
   "verdict": "pass",
   "evidence": "attribution headers on all 6 product .py files + footers on every file >40 lines; server.py carries the 'Powered by Cyan Eclipse Technologies (cetech.ai)' line; bazaar-manifest powered_by + tool list match the shipped surface"
  },
  {
   "id": "U8",
   "name": "bench isolation",
   "verdict": "pass",
   "evidence": "workspace product state byte-identical before/after bench"
  }
 ],
 "notes": [
  "Scope: v0.1 local stdio only \u2014 caller-supplied / bundled LOCAL dataset, no live SAM.gov fetch exists to test. Catalog entry fedscope-mcp still lists status=spec with tools=[] and certification.certified=false (stale listing data, not a product defect \u2014 bench backlog for Rook's catalog pass).",
  "RE-BENCH run 4 (post D1-D3 fix): FULL suite re-run, not just failed checks \u2014 B7/H5/N1 flipped fail->pass, zero regressions elsewhere. Builder claims REPLICATED: own unittest suite 45/45 pass (33 pre-fix + 12 new) and smoke.py passes, bench-run before this suite; scoring snapshot claim VERIFIED \u2014 B2/B3/B4 totals and per-factor points match run-3 verified values exactly and B5 re-runs are byte-identical (scoring untouched by fix). Self-caught defect 1 (deadline floor-days math) VERIFIED fixed: a deadline 30d-1h out floors to 29 days / 17 runway points, an exact 30d deadline earns 30 days / 20 points, a bare-date deadline 30 calendar days out earns 20 (B3). Self-caught defect 2 (smoke probe layering) VERIFIED: smoke asserts over the wire (tools/list, quarantine counts, factor sums, byte-equal dict re-run, named refusals, forbidden verb) and passes end-to-end.",
  "D1 VERIFIED FIXED (run 4): summarize_record now runs the same _deadline_state regardless of profile \u2014 standalone opportunity_summarize flags DEADLINE_UNPARSEABLE ('TBD'), DEADLINE_NAIVE_NO_OFFSET (offset-naive) and DEADLINE_PASSED (past) on bench records AND on Verity's own independently built records, value echoed as recorded, good deadline earns no spurious flag, summarize WITH profile unchanged (same flags + scores).",
  "D2 VERIFIED FIXED (run 4): neutral-branch PoP reasons now split by actual inputs \u2014 country-only USA vs US/FL profile: 5 points, reason states the country-granularity match truthfully, NO PROFILE_LOCATION_UNSPECIFIED; converse cases hold: profile with truly no home location earns the flag legitimately, country-only profile vs state record states 'no home state' truthfully with no false flag, city-only record takes the generic granularity branch; foreign still 0.",
  "D3 VERIFIED FIXED (run 4): envelope battery expanded beyond H5 \u2014 initialize/tools/call/tools/list params as str/list/int/float/bool all -32602, garbage lines -32700, non-object JSON lines ('42', string, array, null, true) -32600, ZERO raw exception text anywhere, catch-all is -32603 INTERNAL_ERROR only, server alive and tools/list intact after the battery.",
  "Observations (not defects, bench backlog): (a) an unknown notice type and a missing notice type both earn 8/15 newcomer openness (missing additionally flags TYPE_UNSPECIFIED) \u2014 declared-neutral, defensible; (b) validate_dataset on an all-malformed dataset SUCCEEDS with valid_records=0 + quarantine report (its job is reporting; DATASET_ALL_MALFORMED is search-only by design); (c) undeclared extra arguments are ignored (tolerated, house pattern); (d) agency names in summaries are title-cased from the recorded path with the verbatim path carried alongside \u2014 a formatting transform, traced, not an invented fact; (e) set-aside canonicalization treats EDWOSB and WOSB as distinct designations (no cross-eligibility) \u2014 conservative, fail-closed.",
  "Runs 1-2 cards retained append-only: both carried BENCH-SUITE defects, not product defects. Run 1 H2 demanded the engine-layer PROFILE_REQUIRED / RECORD_REQUIRED strings for MISSING arguments, but the schema guard correctly refuses those first with -32602 naming the argument (house layering, cf. BidMatrix H5); the engine strings fire for explicit null values, as run 1's own profile-null probe showed. Run 2 H5 used a drain-based resync after garbage lines that raced and desynced the wire, cascading false untestable/fail verdicts into N1/N2/U2/U5/U6 (run 1, with the same product, passed all five). Suite fixed both times (layered expectations; H5 now reads garbage-line responses directly with no intervening request); run 3 was the standing FAILED card for the pre-fix build. Run 4 is the post-fix re-bench; product code was changed only by the builder's D1-D3 fix between runs 3 and 4 (bench touched only NOTES/evidence strings).",
  "N-check ruling (run 4, RE-EARNED in full, not inherited from run 3): NOVELTY_PROVEN. Scoring limb: B3/B4 every point hand-recomputed matches on both axes (~58 bench cases) + B5 byte-identical determinism + B6 profile independence + Verity own-record sample (4-digit NAICS family 12 + eligible 25 + runway 20 + pp 10 + PoP 6 = fit 73; newcomer 88). Summary limb: ALL limbs now hold on EVERY surface incl. standalone summarize (D1 fix) \u2014 traced basis, never-invent, named flags. Renamed-verb rule passes (N2: 0/4). Run-3 NOVELTY_FAILED superseded."
 ],
 "history": "append-only; prior runs retained in results/"
}