Two Claudes in a bare shell

June 12, 2026 · ~9 min read · interactive · Draft — still collecting data

On dreamed trajectories, verification habits, long-range recall, and what eighty turns reveal that two can't.

Take away a coding model's tools. No file-edit tool, no search tool, no plan mode — just a sandboxed repo and one rule: reply with exactly one bash command per turn, and you'll see its real output next turn. Every modern agent harness is built to hide this bare loop under padding. Strip the padding and you see something usually invisible: what the model itself believes doing software work looks like. I've been running frontier models through exactly this setup as part of a benchmark I'm building (more on that project another day), on real bug-fix tasks from SWE-bench Verified, each graded in a clean container against hidden tests. This post is about the most surprising head-to-head so far: Claude Fable 5 vs Claude Opus 4.8 — two models from the same lab, one generation apart, that behave like different species in the same cage.

The one mental model for everything below: when a trained habit meets an environment that doesn't support it, the habit wins. Each model arrived with a picture of how its work loop is supposed to go, and the bare shell either fit that picture or it didn't. The four charts are four views of that collision.

The Setup: One Command Per Turn

Five frontier models — Claude Fable 5, Claude Opus 4.8, GPT-5.5, GPT-5, GPT-4.1 — each ran four sympy bug-fix tasks from SWE-bench Verified, solo, in a network-isolated container, with a turn cap of 80 and a cost cap per run. A patch counts as solved only if hidden fail-to-pass tests pass in a fresh clean-room container. For context I'll occasionally bring in six other models I've run through the same harness (DeepSeek V3 & V4, Kimi K2, GLM-4.6, Qwen3-Coder). Small print up front: this is four tasks, one repository — a probe, not a leaderboard. The patterns below are sharp, but the error bars are honest-sized.

Two Turns vs. Eighty

Here's the head-to-head. Toggle the metric — the shape of the story doesn't change. Fable solved 2/4; Opus solved 1/4. But the how is the finding: on three of four tasks Opus ended the entire job in 2–5 turns and declared it complete. Fable, on the same tasks, worked 7, 9, 18, and 80 turns. Hover any bar for the details.

Figure 1 — Head-to-head on four tasks. Same lab, same harness, same tasks. Opus's 2-turn bars are not efficiency — three of them produced empty or wrong patches.

A 2-turn run that ends in "task complete" sounds efficient. It wasn't: on tasks #13757, #23534 and #23950 Opus's final patch changed zero lines (or failed the hidden tests), yet the model signed off confidently. Before reading those transcripts, one more measurement pins down exactly what was missing from Opus’s two turns.

Fable Verifies; Opus Never Ran a Test

The cleanest single behavioral discriminator I have between the two Claudes: did the model actually execute the test suite before declaring the task done? Fable ran real tests in 3 of 4 tasks (and in both of its solved runs, verified before claiming success). Opus executed zero test commands across all four tasks — every "all tests pass" in its transcripts refers to a test run it imagined.

Figure 2 — Real test executions per task. Verified-before-done: Fable 2/4 runs, GPT-5.5 4/4, Opus 0/4. GPT-5.5 shown as the cohort's most consistent verifier.

This is the difference that reads most like a trained value rather than a capability gap. Running pytest in a bare shell is not hard — V3-class models manage it. Fable's habit ("verify, then claim") and Opus's habit ("conclude from reasoning") are policies, and on this evidence they were shaped by different incentives.

102 Commands That Never Ran

On task #13757, Opus's first reply was 65,862 characters long and contained 102 bash code blocks — a complete imagined work session: commands, fabricated outputs formatted as if the system had returned them, fabricated test results, and a victory lap. Condensed, it reads like this (red marks text the model invented — none of these outputs ever existed):

The harness, of course, executed only the first real command and fed back the actual output. Opus looked at it, remarked that the result was "leftover from a previous turn," concluded its (imaginary) work was already finished, and submitted. The whole session lasted two turns and produced an empty patch. The same pattern repeated on two other tasks — a 17k-token dream on #23534, another on #23950. I call this a confabulated completion: the model simulates the loop it expected instead of using the loop it's in, then trusts the simulation.

Dreaming Is Generational — Except Opus

So is this just a Claude quirk? No — and the cross-model picture is what makes it worth a chart. Line up all eleven models I’ve put through this harness and confabulated completions follow training vintage almost perfectly: DeepSeek V3, a 2024-era model, invents its way through nearly every run; mid-cycle models slip now and then; today’s frontier models essentially never do it. There is exactly one exception: Opus 4.8.

Figure 3 — Fraction of runs where the model trusted an imagined session and called the job done. Bars sorted high to low. Opus 4.8 lands in territory belonging to models one or two generations older, while Fable 5 — same lab — sits at zero. Hover any bar for exact counts.

Look at where the two Claudes landed: Fable 5 never confabulated (0 of 4); Opus 4.8 did it on three of four tasks. These are sibling models from one lab, built on largely shared infrastructure — yet on this behavior they don’t differ by degree, they split absolutely. Whatever was trained differently between them surfaces here, in this one habit. Keep that in mind for section 8: two rival explanations fit this chart, and my data can’t fully tell them apart yet.

Recall Needs Time on Task

One thing I measure is long-range recall: in its visible prose, does the model correctly reference a specific artifact — a file path, a function, an error string — that it last saw more than ten turns earlier, without re-reading it? It's a working-memory probe: re-reading is memory by action; recall is memory from context. Each dot below is one run, plotted by how long the model stayed in the task vs. how often its references reached back beyond ten turns.

Figure 4 — Long-range recall rate vs. trajectory length. Each dot is one run; recall is only measurable on runs longer than 10 turns (shaded zone = unmeasurable). Opus 4.8 has no dot on this chart: it never stayed past 5 turns. Toggle to re-read rate — the same runs jump to 40–90%; click a model name to follow one model’s dots.

Three reads. First, the structural one: Opus is absent. Not because it can't recall — because confabulated completions end its runs before recall is even measurable. Behavioral metrics interlock: the dreaming habit censors the memory measurement. Second, recall rates cluster in a humble 3–12% band for almost everyone — frontier models mostly re-read instead of recalling (Fable re-read files in 60% of its long-task turns; cheap and reliable). Third, the high-recall outliers in the reference cohort are reasoning-heavy models on very long runs, which is a story for the longer write-up.

Dreamer, Worker, Pragmatist, Grinder

The two hardest tasks (#13757, #23950 — solved by nobody in this frontier cohort) are where models reveal their failure styles. Toggle between them. Watch how differently five models spend the same 80-turn budget when the task is genuinely beyond them.

Figure 5 — How five models spend an 80-turn budget on an unsolvable-for-them task. Bar length = turns used; color = how the run ended. Hover for cost and detail.

Four archetypes, remarkably stable across both tasks:

• The dreamer (Opus 4.8) — 2 turns, imagined success, supreme confidence. Cheap in turns, expensive in trust.
• The worker (Fable 5) — engages for real: 18 turns into #13757 until the cost cap, all 80 turns on #23950. Notably, when it stalls it says so — its #13757 run ends with an explicit "I am stuck" rather than a fake success. Honest failure is a behavior, and Fable has it.
• The pragmatist (GPT-5.5) — gives a real attempt (10–23 turns, tests included), concludes it can't crack it, stops voluntarily, reports failure. The cheapest honest worker in the cohort by an order of magnitude.
• The grinder (GPT-4.1, GPT-5) — burns the entire budget heads-down. GPT-4.1 spent 80 turns and ~150 commands on #13757 without ever stepping back. Persistence without progress evaluation. (Interestingly, GPT-4.1 also dream-quits instantly on 2 of 4 tasks — it's bimodal: grind or dream, nothing between.)

The Cage or the Animal?

Now the part I'd want a reviewer to push on, so let me push first. There are two live explanations for Opus's dreaming, and they have very different implications:

1. It's a property of the model's training. Something in how Opus 4.8 was optimized made "produce a plausible-looking complete session" rank above "interact with the environment and verify."
2. It's a property of my cage. Claude models are trained to act through structured tool calls — a rich, multi-tool format. My harness denies that format and demands bare fenced-bash turns. Maybe Opus, deprived of its native interface, falls back to simulating one — and the dreaming would vanish on a tool-call-native harness.

Three facts complicate the pure-cage story: Opus did solve the easy task cleanly in 5 well-formed turns, so it can operate this interface when the task is shallow; Fable 5 — trained by the same lab, presumably on the same tool formats — shows zero confabulation in the same cage; and models with no Anthropic-style tool training (GPT-4.1, V3) dream too. But the honest position is that with the current data I cannot fully separate "Opus's habits" from "Opus's habits when denied its tools" — the falsification experiment (same tasks, native tool-call harness) is queued, and I'll report it either way. Until then, every Opus number above should be read as a statement about Opus-in-a-bare-shell, not Opus.

Putting It Together

The headline isn't that one Claude beat another on four sympy bugs — n is far too small for leaderboards, and the cage caveat is real. The headline is that two same-lab models diverge categorically on behaviors you'd expect to be values, not capabilities: whether to verify before claiming, whether to keep engaging when the environment fights back, whether to admit being stuck. Those are exactly the behaviors that training incentives shape — which is the larger project this data comes from: reading the fingerprints that training leaves on behavior. More on that, with the full cohort and methodology, in a future post.

Data: 11 models × 4–12 runs each on SWE-bench Verified tasks, all under an identical minimal harness, graded against hidden tests in fresh containers. Charts show curated aggregates; tasks are public SWE-bench instances. Numbers are small-sample and one-repository — treat patterns, not decimals, as the signal.