← index2026-05-07 01:56 (Beirut)(backfill from DOCUMENTATION/)

Brian Evidence-Gate System — Day 1 Reference

Brian Evidence-Gate System — Day 1 Reference

Built: 2026-05-06 | Source: /opt/agent/data/postmortems/260506_supervisor_design/ARCHITECTURE-FINAL.md

What this is

A deterministic evidence-gate that locks Brian's ability to claim done / fixed / shipped / blocked / "send me X" until verifiable evidence exists in an immutable ledger. Replaces the original Brian R1 LLM-judge proposal (which would have been bypassed semantically). All decisions are deterministic; LLMs only extract claims and classify intent.

The North Star (Codex R3, verbatim)

Brian may populate task details, but only the harness may bind claims to validators, execute validators, store evidence artifacts, and decide whether a final response is allowed.

Every component below is in service of this sentence. Any future change that violates it is a regression.

Round-table provenance

Three R1 proposals, three R2 cross-study refinements, one R3 adversarial review.
- Codex R1+R2: evidence-gated state machine; locked final-response capability
- Gemini R1+R2: verify-then-voice loop; deterministic ledger over LLM judge
- Vibe R1+R2: hard interceptor; no prose regeneration; user-collusion audit
- Codex R3 (adversarial): 5 patches → trust boundary model + schema enforcement + raw artifacts + validator binding policy + scoped Day 1

Antigravity R1+R2: async IDE, did not return synchronously. Documented gap.
Jules: substrate mismatch (TUI not chat consultant). Documented; side-fixed jules-delegate.py hardcoded-timeout bug.

Day 1 inventory (shipped)

Files written (Brian-writable)

Files written (Jonah-owned, chmod 0444 root:root — Brian read-only)

Runtime data

settings.json wire-in

How to use it

When Brian wants to claim "done"

brian_validator.py --claim done --validator pytest_pass \
  --args '{"target":"tests/test_x.py::test_y","required_runs":3}'

On PASS (exit 0): writes a VALIDATOR_PASS synthetic ledger entry. Subsequent Stop hook on a "done" message will pass.
On FAIL (exit 2): no entry written. Next "done" message gets BLOCKED.
On REFUSED (exit 3): wrong validator for claim, OR weak args — Brian must fix.

When Brian wants to claim "blocked"

brian_validator.py --claim blocked --validator ladder_exhaustion_proof \
  --args '{"turn_id":"<id>"}'   # Day 2 only — not yet wired

When Brian wants to ask Jonah for something fetchable

brian_validator.py --claim delegation --validator fetch_attempt_proof \
  --args '{"target_keyword":"linkedin_cookie","distinct_routes_attempted":3}'

The runner counts FAILED fetch-tool entries in the ledger matching the keyword.

Testing

Smoke test

python3 /tmp/test_evidence_gate.py    # 10/10 pass at Day 1 lock

E2E integration test

/tmp/test_validator_e2e.sh            # 5 steps, ledger → validator → gate

Trust boundary table (Codex R3.4 patch)

Asset Brian Harness Jonah Enforcement
Validator YAML read-only execute author chmod 0444 root:root ✓
Claim-binding policy read-only execute author chmod 0444 root:root ✓
Evidence ledger NONE append-only via hook audit hook is sole writer ✓
Raw artifact store NONE write via hook audit hash-chained ✓
Synthetic VALIDATOR_PASS NONE append via runner audit runner refuses weak args ✓

What's NOT shipped yet (Day 2-3 backlog)

Honest limits (carried from architecture doc)

  1. Cannot retrain RLHF priors. Substrate ceiling stands.
  2. Cannot prevent Jonah collusion. L5 (Day 3) surfaces it; he chooses.
  3. Cannot catch out-of-band actions (channels harness doesn't see).
  4. Cannot replace Jonah's real-time enforcement contract. Irreducible.
  5. Cannot guarantee zero false-accept forever. Weekly calibration is the maintenance burden (Day 3).

Verification gate

Initial Day 1 (synthetic only):

Check Status
settings.json is valid JSON ✓ verified by python3 -c json.load
Both hooks wired into settings.json ✓ verified by jq
Smoke test 10/10 pass (initial) ✓ executed
E2E test 5/5 steps ✓ executed
Validator YAMLs chmod 0444 root:root ✓ ls -la
Hash chain verified across multi-entry ledger ✓ chain_prev → chain_hash linked in step 1 of E2E

Hardening pass — Gemini R4 code review (3 bugs fixed):

Hardening pass — Codex implementation review (5 more bugs fixed):

# File:Line Defect Fix
1 gate.py extraction-failure exit Silent-pass on extraction empty → BLOCK on >50KB transcripts with empty extraction (heuristic distinguishes legit "no text yet")
2 gate.py first-claim-only find_claim_in_text → find_all_claims_in_text; collect ALL claims, check each, list all violations
3 gate.py unverified VALIDATOR_PASS HMAC-SHA256 signing in runner; gate _verify_hmac against /root/.claude/system/secrets/validator_hmac.key (chmod 0400 root)
4 ledger.py + validator.py seq race fcntl.flock(LOCK_EX) on per-session lock file LEDGER_ROOT/.{sid}.lock
5 Missing test coverage Added B10 (multi-claim), B11 (forged VALIDATOR_PASS rejected), B12 (extraction-anomaly heuristic)

Final test totals

Suite Cases Pass
/tmp/test_evidence_gate.py 18 18/18
/tmp/test_validator_e2e.sh 5 5/5

Hardening pass — Live test exposed wrong CC hook contract

Live test in fresh CC session: gate fired correctly but CC sent the "Shipped" message anyway.
Root cause: hook emitted {"systemMessage": ...} which is non-blocking advisory. CC contract for force-regeneration is {"decision": "block", "reason": "..."}.
Fix applied:
- gate.py now emits {"decision":"block","reason":...} on both BLOCK paths (claim violation + extraction anomaly)
- Added stop_hook_active loop guard at top of main() per CC docs "Stop hook runs forever" footgun
- Test assertion strengthened to require BOTH decision:block in stdout AND BLOCK message in stderr
- New regression test P7 (loop guard) — stop_hook_active=True must exempt

Final test totals (post live-test patch)

Suite Cases Pass
/tmp/test_evidence_gate.py 19 19/19
/tmp/test_validator_e2e.sh 5 5/5

What's still NOT verified live

Day 2 axiom (from Codex R5 retry, post-fix)

"Absence of an approved validator for the claimed scope is itself a blocking condition; Brian may report only attempted actions and uncertainty, not completion."