Methods

This document describes the v3 study completed in September 2026. Ten medium sessions were admitted; both main max attempts were interrupted and excluded. The scope amendment records the reduced follow-up, and the current README describes the later harness revisions. Those revisions were not used in the measured sessions.

Study design

Each independent session has eight practice games followed by two scored games. Conversation, notes, and controller code persist within a session. Every game starts a fresh native process with fresh data and cache directories; saves and native lore do not carry over. New sessions share no agent history.

A scored win requires escape from Angband at the surface while carrying at least one Silmaril. We report both scored outcomes and their maximum, with the session as the unit of uncertainty. Depth and other progress measures are diagnostic. Reaching 1000 feet, dying with a jewel, escaping empty-handed, quitting, and retiring earn zero. Every scheduled game must end, including after a scored win.

The medium study consists of ten sessions. The predeclared saturation threshold is a scored victory in nine of ten sessions. All ten must finish and pass review before applying it. Only a valid result below that threshold permits max runs. The original follow-up paired medium and max initial random streams and included fresh max controls with two scored games and no practice. Such controls would estimate the combined effect of practice, extra inference, and a persistent workspace, not an isolated learning mechanism. Prior knowledge of the public game is uncontrolled.

After medium completed, the user reduced the follow-up to the two main max sessions already launched. The fresh controls were canceled. One main max attempt ended after API errors; the other was stopped during its eighth practice game at the user's wrap-up request. Both remain in the attempted-session count and are excluded from capability estimates. No scored max result, comparison between reasoning levels, or practice-effect estimate is available.

Resources and admission

Research allowances were frozen at 12 hours and 100,000 keys per game, with a conservative $1,200 spend limit per session. The client uses a 1,050,000-token configured context, a 900,000-token compaction threshold, and up to 128,000 output tokens per request. Its effective context window is 997,500 tokens. Compaction is part of the setup; it does not preserve every earlier detail in active context. Resource calibration gives the rationale and checks.

Infrastructure failures invalidate a session. Unstarted and unfinished games remain zero placeholders in the raw fixed-length score record, but incomplete, invalid, and resource-bound sessions do not enter admitted capability summaries. A limit that materially constrains play requires a written revision and a fresh cohort. We retain the affected evidence rather than replacing selected losses.

The first Astra cohort, v1, was excluded after client telemetry revealed a smaller context window than the API metadata specified. V2 was excluded for overlapping native song and depth fields at 24 rows. V3 readiness followed the corrected 27-row GPT-5.5 medium pilot, direct public play, and exact native replay.

Native engine and observations

The evaluator links pinned Sil-Q gameplay code to an 80×27 terminal backend. Observations contain text, color and shade runs, cursor visibility, and warning bells from native terminal callbacks. The agent can use inventory, equipment, look, map, help, character, ability, and knowledge menus. It receives no hidden map, entity table, random state, private object identifiers, or legal-action oracle. The interface uses the standard numerical keyset; graphical tiles, sound, and auxiliary terminal windows are omitted.

The engine computes combat, stealth, visibility, discovery, character development, level transitions, and escape. The adapter does not interpret the map or change item and ability powers. The supplied manual is historical 1.5.0 text. Where it differs, the guide gives precedence to the pinned game and its descriptions; known errata still limit how the results can be interpreted.

Song lines occupy zero-based rows 21 and 22. At 24 rows, the bottom-anchored depth indicator also occupied row 22. A song redraw could erase it or replace it with the second song. Native fixtures reproduced the collision and verified that 27 rows preserve both songs and both depth fields. Exact replay alone did not catch this layout problem. Geometry evidence.

The adapter fixes the native calendar at September 7, 2026, avoiding holiday gifts. It replaces the predictably seeded native random generator with a protected 256-bit-key HMAC stream, retaining the bounded-integer rejection rule. Gameplay uses a counter domain; quick/cosmetic randomness uses a separate domain and preserves explicit reseeding. This changes the random implementation, so results are not comparable to an unmodified upstream seeded leaderboard.

Screen reads do not call the engine. Tests verify that the exercised native information menus do not consume gameplay randomness. Character creation and later actions can consume different draws: matched initial streams do not promise identical dungeons when agents choose different builds or actions.

Isolation and input

The coding agent runs in a client container with a writable workspace. The engine and grader run in a separate service container on an internal Docker network, with no exposed host ports, dropped capabilities, no-new-privileges, and a read-only root filesystem. Only the service mounts protected state. The client contains no engine, source data, schedule, test oracle, or grader.

The native build excludes debug and spoiler code and disables wizard entry, configuration editing, macros, colors, preferences, character and screen exports, save/load, and character-name-based preference loading. A checked upstream patch also fixes an out-of-bounds read for invalid birth-menu letters.

The public API accepts 1–64 ASCII keys per action. Each key receives a native response before the next is sent. Action requests name the current trial and frame; malformed or stale actions are rejected before native input. Start and finish use separate lifecycle endpoints. There is no reset, rewind, custom seed, score submission, tutorial, or replacement-trial endpoint.

HTTP batch responses include every intermediate screen. The CLI saves those screens locally and prints the final one with the saved range. silq history reads earlier frames without advancing the game. This avoids clipping large batch responses at the coding tool's output limit. Agent-owned archives and final answers are never scoring evidence.

Scoring and replay

The protected grader counts loose Silmarils and Morgoth crown variants from native inventory and compares the total with the native quest counter. Victory also requires positive-quantity possession and the native surface escape transition. Sil-Q sets the death flag on escape; requiring it to be false would reject real wins. Wizard, no-score, tutorial, or nonstandard-option state invalidates the record. Names, inscriptions, and reported victories do not affect scoring.

The service fsyncs key intents, observations, endings, and rejected requests into a hash-chained journal. The auditor reconstructs schedules and scores from that journal. Replays use the run's frozen service image and compare every native record. Before dispatch, the launcher retains immutable image tags, verified compressed Docker archives, source hashes, guide and manual contents, private schedules, and model/effort identities. Replay containers have no network and mount only the temporary data needed for playback.

These controls address score forgery, hidden-state reads through the API, save scumming, wizard commands, and dungeon replacement. They do not eliminate all possible bugs in the native C engine, container runtime, or host. Sanitizer and hidden-state test coverage is recorded in the development ledger. Current request and recovery behavior is documented in the README.

Review

Automated checks verify journal integrity, exact replay, model and reasoning effort, resource use, errors, and source identity. Admission also requires a review of visible assistant messages, available reasoning summaries, complete tool calls and results, controller revisions, native screens and actions, and terminal outcomes. Inspect model events recover history removed from the final context by compaction. Encrypted reasoning is not interpreted or published. Reviews and direct play were performed by the development agent; there has been no independent human expert validation.

Review distinguishes controller mistakes from environment defects. Invalid-key bells, shell errors, stale requests, clipped output, and time spent planning are examined in context. Material environment defects require a fix, new development checks, and fresh research schedules.

Security regex matches are review flags. Each needs evidence that the attempted operation was blocked or benign; an unresolved finding or successful bypass prevents admission. A blocked attempt does not erase an otherwise valid loss.