Sil-Q evaluation

A coding agent plays Sil-Q through its 80×27 terminal. The protected service owns the engine, private randomness, and scoring. A win requires taking a Silmaril from Morgoth's crown and escaping with it.

The September 2026 study completed ten Astra-medium sessions, each with eight practice games and two scored games: 0 wins in 20 scored games. Across all 100 games, 99 ended in death and one practice game was retired. Both Astra-max attempts were interrupted before scored play and excluded; the controls were canceled. There is no scored max result or measured effect of practice.

Report and replays · Report source · Results · Methods · Scope amendment

Current harness

Revision 0.5.0 uses silq-session-v2. It adds durable request receipts, verified replay after service restart, persistent client workspaces, and experiment supervision. Completion and victory are separate metrics; authenticated final scoring freezes outcomes.

Engineering checks passed 246 tests (two optional plotting tests skipped), native sanitizer and replay checks, and real container restart and scoring checks with zero model requests. Fresh model pilots, direct public play, and resource calibration are still needed before new research. Historical results retain their frozen implementation and materials.

Development

Use Python 3.11 or later, uv, and Docker with Compose. From this directory:

uv sync --frozen
uv run python scripts/bootstrap.py
uv run pytest -q -rs
uv run python scripts/native_qa.py --output /tmp/silq-native-qa.json
uv run python scripts/qa_inspect.py --output .private/qa-new --report /tmp/silq-inspect-qa.json
uv run python scripts/qa_recovery.py --output .private/qa-recovery-new

These checks make no model requests. Bootstrap builds the development image tags. To preserve existing tags, build separate Docker service and client targets and pass --service-image and --client-image to the smoke tests. Output directories and run names are single-use. CI runs the engineering checks.

Experiments

Copy experiments/development.template.json and set the name, images, limits, and bundle count. Its default limits have not been calibrated for this revision. Preparation freezes source, images, schedules, and commands without model calls:

cp experiments/development.template.json /tmp/silq-development.json
uv run python scripts/experiments.py prepare /tmp/silq-development.json --output .private/experiment-new
uv run python scripts/experiments.py status .private/experiment-new/experiment.json

To launch development model sessions, resume supervision, and review a closed run:

uv run python scripts/experiments.py run .private/experiment-new/experiment.json --workers 1
uv run python scripts/experiments.py resume .private/experiment-new/experiment.json --workers 1
uv run python scripts/audit_run.py .private/run-name

Workers survive supervisor exit. Locks and saved launch intents prevent duplicate Inspect launches. Failures stop queued launches; ambiguous exits require review. resume resumes supervision, not an interrupted Inspect conversation. Client notes, code, and request history persist in each run's workspace/. Only the service mounts state/. Keep both run directories and experiment manifests private.

Research dispatch requires fresh development, direct-play, resource, and production evidence, plus a readiness record whose experiment_sha256 binds the prepared plan. Max dispatch remains blocked until the new format has a reviewed medium-result gate. The older cohort.py and scope rules describe the closed study.

The audit wrapper checks request identity, native replay, tool history, and public observations. It reads report findings as well as exit codes. Truncated views and boundary flags need review; they do not automatically invalidate a loss. Admission requires qualitative review and evidence hashes.

Requests and recovery

The game guide documents the CLI. GET /state reads the current public frame and GET /overall reads the schedule. Mutation bodies contain exactly these fields:

Each new mutation uses a canonical UUID. Retrying the same endpoint and body returns its stored status and response, including after restart. Reusing an ID with different content fails. The CLI saves requests before sending them; silq retry retrieves the last request's receipt.

Recovery replays the original random stream and compares every committed native frame, including private state. Changed code, data, configuration, or receipts block recovery. Only an unterminated final journal line can be removed; its bytes are retained. Historical v1 sessions require their frozen service images.

An interrupted batch returns its executed prefix. A pending key is replayed on the same stream and its output accepted at recovery time; remaining keys are not sent. Read the returned state before continuing. A start interrupted before trial creation returns an error and can be issued deliberately with a new ID.

Deadlines remain absolute across downtime. An ending observed before the deadline retains its result after restart. Infrastructure failures remain invalid. One service can own a session at a time; HTTP connections and idle waits are bounded.

Results and historical records

analysis/ contains results, reviews, engineering records, screenshots, and the original guide versions. site/ contains the report, figures, and public replays. .private/ retains logs, journals, source snapshots, schedules, and image archives. Run-specific validators live in scripts/historical/; report and replay builders live in scripts/reporting/. See the script index.

Development history, selection rationale, study resource limits, and the September 9 audit describe earlier stages. Native errata explain errors in the materials used by the agents.

To rebuild the report and check its replay viewer:

uv run --frozen --with markdown python -m scripts.reporting.build_site
uv run --frozen --with playwright python scripts/reporting/qa_site.py

uv run --frozen --with matplotlib python -m scripts.reporting.plot_max_case renders the reviewed max case studies, including exclusions.