{
 "tool": "eclipse-environment-mcp",
 "version": "0.1.0",
 "bench_date": "2026-10-25",
 "bench_version": "0.1.0",
 "environment": "simulator",
 "qualifier": "simulator",
 "verdict": "CERTIFIED(simulator)",
 "checks": [
  {
   "id": "E1",
   "name": "unenrolled hub-visible device invisible on all surfaces",
   "verdict": "pass",
   "evidence": "the bench first confirms the hub itself serves light.den (the unenrolled den light); registry_list and state_snapshot each return exactly the 6 enrolled devices with no trace of it in either payload, and policy_evaluate / command_execute for the unenrolled device both deny with DEVICE_NOT_ENROLLED \u2014 unenrolled means invisible, not merely uncommandable"
  },
  {
   "id": "E2",
   "name": "tier-floor violations rejected at load (exit 2)",
   "verdict": "pass",
   "evidence": "both shipped rejection fixtures stop the server at load: the override form (oven start_heat forced to Tier 1) exits 2 naming TIER_FLOOR_VIOLATION, the declared form (classified below the heating_start floor) exits 2 naming TIER_BELOW_FLOOR \u2014 and in both cases stdout carries no JSON-RPC response: not one tool call was served against a ladder-violating config"
  },
  {
   "id": "E3",
   "name": "Tier 3 confirmation gate: deny without token",
   "verdict": "pass",
   "evidence": "Tier 3 (kitchen_oven.start_heat, heating_start floor 3): policy_evaluate without a confirmation token denies CONFIRMATION_REQUIRED, command_execute denies the same way and the denial is receipted as command_denied; with the owner's fixture confirmation token (matched by hash only \u2014 the policy stores no token) the verdict becomes ALLOW at tier 3, still execution_enabled_v0_1=false \u2014 confirmation opens the verdict, not the phase gate"
  },
  {
   "id": "E4",
   "name": "Tier 1 command receipts only on state confirmation",
   "verdict": "pass",
   "evidence": "Tier 1 toggle on the enrolled living-room light (hub-reported off -> on) returns executed_confirmed with a receipt_seq; exactly one command_executed receipt exists and its confirmation_basis records the hub-reported state transition \u2014 the receipt asserts the hub's reported state, not the command's acceptance"
  },
  {
   "id": "E5",
   "name": "accepted-but-unacted command: alert, NO receipt",
   "verdict": "pass",
   "evidence": "with the stub in its test-only mismatch mode (service calls return success but change nothing), the same Tier 1 toggle returns unconfirmed with receipt_seq null: the receipt ledger length is unchanged (8 -> 8) and the attempt is recorded as exactly one STATE_MISMATCH alert instead \u2014 acceptance is not proof, and no receipt asserts an effect the hub does not confirm"
  },
  {
   "id": "E6",
   "name": "snapshot freshness + provenance",
   "verdict": "pass",
   "evidence": "snapshot freshness is honest: the hallway motion sensor (fixture ships 2h old vs the policy's 1800s threshold) reports stale=true with its real age, the just-actuated living-room light reports stale=false, and every device entry carries provenance naming this bench's loopback hub URL and its entity_id \u2014 states are never silently freshened"
  },
  {
   "id": "E7",
   "name": "receipt chain valid; tamper detected",
   "verdict": "pass",
   "evidence": "receipt_list reports chain_valid over 9 receipts (registry/snapshot/evaluate/denied/executed); the product verifier confirms it, and a deep copy with one altered receipt detail verifies false"
  },
  {
   "id": "U1",
   "name": "auth enforced",
   "verdict": "na",
   "evidence": "local stdio transport: no credential layer exists to test; process ownership is the boundary"
  },
  {
   "id": "U2",
   "name": "schema valid",
   "verdict": "pass",
   "evidence": "tools/list returns exactly the 5 wrapper tools, each with inputSchema type object + properties + required"
  },
  {
   "id": "U3",
   "name": "network surface is the bench loopback stub only",
   "verdict": "pass",
   "evidence": "the wrapper's only hub address is env-supplied (ECLIPSE_ENVIRONMENT_HA_URL) with a loopback default (127.0.0.1:18123); this run wired it to the bench's stub on 127.0.0.1:44339 and no other host literal exists in server.py \u2014 no real Home Assistant was contacted"
  },
  {
   "id": "U4",
   "name": "no secret leak",
   "verdict": "pass",
   "evidence": "no secret patterns in 14 tool outputs"
  },
  {
   "id": "U5",
   "name": "sane errors",
   "verdict": "pass",
   "evidence": "snapshot of an unenrolled device -> error; executing a Tier 0 sense capability -> error; a would-be enroll_device tool -> -32602 (enrollment is not an agent action); malformed line survived (registry_list answers); unknown method -> -32601"
  },
  {
   "id": "U6",
   "name": "process health",
   "verdict": "pass",
   "evidence": "initialize answered in 105 ms; ping ok"
  },
  {
   "id": "U7",
   "name": "audit completeness",
   "verdict": "pass",
   "evidence": "ledger covers the run (['registry_list', 'state_snapshot', 'policy_evaluate', 'command_denied', 'command_executed'] all present); 10 receipts, seq strictly increasing, chain valid \u2014 and denials are receipted alongside allows (policy_evaluate receipts include verdict=deny entries): the audit records refusals, not just successes"
  },
  {
   "id": "U8",
   "name": "bench isolation",
   "verdict": "pass",
   "evidence": "workspace product state byte-identical before/after bench"
  }
 ],
 "notes": [
  "Simulator scope: the hub is the product's own HA-shaped stub (ha_stub.py) started by this bench on 127.0.0.1; no real Home Assistant instance exists in this environment and none was contacted. Real-HA parity (auth, service payloads, state semantics) is [verify] and untested.",
  "v0.1 phase scope: Tier 2/3 EXECUTION is unbuilt by design (evaluate + deny-by-phase); the bench probes the verdict paths and the Tier 3 confirmation gate, not execution. Vehicle rails (Smartcar/Tesla) do not exist in v0.1."
 ],
 "history": "append-only; prior runs retained in results/"
}