SIL-Q BENCH

Harness audit — September 9, 2026

Sil-Q already scored native escape and inventory state, isolated the engine from the agent, and tested replay and hidden-state filtering. This audit found problems in metric reporting, concurrent scoring, deadline handling, and evidence validation. Revision 0.4.0 fixes those issues. The historical study used its original frozen implementation.

This is the September 9 record. Revision 0.5.0 subsequently added recovery, request receipts, connection limits, and experiment supervision. See the current harness for its status and checks.

Changes

Problem Consequence Fix
Completion carried accuracy() Retiring every game could report 100% accuracy with zero wins. Accuracy measures scored victory. Schedule completion separately controls retries and has no accuracy metric.
Scorers audited live files independently Gameplay could change between reads or while files were being audited; final scoring did not stop it. An authenticated operator endpoint audits under the action lock. Final scoring interrupts a live game and saves a durable snapshot. All scorers share and verify that snapshot.
Service and auditor used different terminal timestamps Journal fsync could push a result past the auditor's deadline after the service had accepted it. Both use the validated observation's acceptance time. Keys also receive a single admission timestamp. Late results earn no victory credit.
Incomplete journal and summary checks Empty evidence could pass an empty-session audit; malformed evidence could discard a native ending or attach an unearned trial result. Require a session header, typed finite metadata, valid deadlines and transitions, and exact agreement between journal and summary.
Falsey malformed state fields and mixed-case seeds were accepted Invalid native fields could pass; two seed strings could represent the same bytes. Validate the full native schema and require distinct canonical lowercase seeds.
A readable stream could keep a native read going after its timeout Partial output could extend the read until its size limit. Check the monotonic deadline before every selector call.
Summary replacement lacked fsync A crash could leave durable events with a stale summary. Fsync state files and their directory around atomic replacement.

The operator token is stored only in protected service storage, with mode 0600. It is absent from public responses, journals, and the client image. Intermediate scoring leaves gameplay active; final scoring closes it. Interrupted games remain incomplete for schedule-completion purposes.

The snapshot digest and unkeyed journal chain check consistency. They do not protect against a process or operator able to rewrite all protected evidence. The journal fixes address corruption and service defects; the agent cannot normally write those files.

Native behavior preserved

Upstream files.c:do_cmd_escape sets both escaped and is_dead, clears playing, and records escape before postgame UI. Requiring the death flag to be false would reject real wins. Scoring also requires surface depth, a generated character, standard rules, and positive-quantity quest-item possession. Screen text, names, and reaching 1000 feet cannot award a win.

The native crown counter ignores stack quantity, so the independent possession check remains necessary. Ordinary consumed items can briefly retain their kind with quantity zero while a message waits for input. Rejecting every such slot would invalidate normal play. Both cases remain covered by tests.

Gaps recorded on September 9

Recovery and idempotent requests. Sil-Q refuses to restart an existing journal. This prevents resets but leaves an interrupted service unrecoverable. Keys are guarded by trial/frame; start and finish lack durable request IDs and expected trial IDs. Concurrent or delayed lifecycle requests can therefore affect a later trial. The client does not retry mutations automatically; callers should inspect state after an ambiguous response. Recovery needs to preserve original deadlines, random streams, committed outcomes, and full batch responses.

Abnormal termination. Normal Inspect final scoring freezes the session. A killed supervisor, container failure, or provider exception can bypass that hook. Those runs need review and exclusion rather than being counted as capability losses. Existing research rules also exclude resource-bound sessions, although the raw fixed-length score record represents missing outcomes as zero.

Native coverage. Victory fixtures invoke the escape function with controlled inventory. They test scoring, not a naturally played winning route. A complete winning replay, broader command/menu sanitizer tests, and more hidden-state comparisons would strengthen coverage.

Resource isolation. Bodies and key batches are bounded; thread count and rejected-request journal growth are not. A malicious controller can exhaust its service. Concurrency, request-rate, and storage limits are needed before using this service for hostile multi-tenant workloads.

Review tooling. Run-specific controller audits preserve useful evidence but are difficult to maintain as a general validator. Grading, replay, and heuristic review flags should remain distinct. Admission already requires evidence-backed resolution of regex flags rather than automatically discarding flagged losses.

Validation and rollout

At the time of this audit, revision 0.4.0 still needed rebuilt images and fresh readiness evidence before research. No running Brogue job was changed.