Caveman vs Headroom: Which Tokens Actually Get Saved?
56 isolated Opus runs, real benchmark tasks, every token classified — which Claude Code token saver actually works, where — and the verdict a harness bug almost stole.
Verdict#
A tie at the top: caveman and the stack both cut cost per solved task −30%; headroom −18%. Quality is identical across all four — 10/14 each. (An earlier verdict had caveman “breaking a task” — that was our harness, not the tool; § why tasks failed.)
The matchup#
Caveman makes Claude answer in fragments; headroom compresses what Claude reads. One figure each.
* a shorter transcript also shrinks the replayed context (−33%) — a second-order effect, not compression.
Where it sits: an ordinary plugin — a SessionStart hook injects the ruleset into context, and the model itself changes register.
- answer text −47% (1.1k → 590 tok/run): explanations and summaries become fragments
- reasoning −42% (3.8k → 2.2k): the thinking between tool calls shortens with the register
- tool calls −37% (953 → 605): fewer, tighter calls — the code written is untouched
- knock-on: cache-read −33% (462k → 308k): a shorter transcript replays less history each turn
Where it sits: a localhost proxy via ANTHROPIC_BASE_URL. Nothing inside Claude Code changes — so behavior doesn't either.
- what it compresses: the old tool results re-sent every turn — file contents, pytest/log dumps, JSON — each block typed and routed (JSON structurally crushed · code trimmed AST-aware · prose/logs via a small model)
- cache-read −45% (462k → 254k tok/run): the history is compressed once, and the saving repeats every turn after
- what it never touches: everything Claude generates — reasoning, answers, tool calls all ≈ unchanged; same fixes, same turn count
Method#
| model | conditions | tasks | trials | total runs | wall | errors |
|---|---|---|---|---|---|---|
| claude-opus-4-8 | 4 | 7 | 2 | 56 | 2h16m | 0 |
| task | source | what the agent must do |
|---|---|---|
| C1 | SWE-bench Verified | pylint — fix a crash when the container CPU quota is fractional |
| C2 | SWE-bench Verified | pytest — fix a regression in dynamically-added xfail markers |
| C3 | SWE-bench Verified | sphinx — fix literalinclude prepend/dedent whitespace handling |
| C4 | writing task | contributor onboarding doc for the pylint codebase |
| O1 | SpreadsheetBench | match items across columns, return positions |
| O2 | SpreadsheetBench | assign unique code numbers by formula |
| O3 | SpreadsheetBench | cumulative balance across 2.7 MB workbooks |
Where the tokens go#
replayed context vs output ≈ 80:1. berry tick = all generated output; caveman's −42% cut is invisible at this scale — the point.
What a solved task costs#
Divide by solved tasks and the podium reorders.
caveman and the stack tie at −30%. an earlier version of this chart had caveman third — a stale-bytecode harness bug had failed its correct sphinx fixes (§ why tasks failed).
| condition | solved | est $/run | est $/solved | wall s/run |
|---|---|---|---|---|
| baseline | 10/14 | 0.528 | 0.740 | 96 |
| caveman | 10/14 | 0.367 | 0.514 −30% | 60 |
| headroom | 10/14 | 0.431 | 0.604 −18% | 148 |
| both | 10/14 | 0.369 | 0.516 −30% | 98 |
Per task#
Cost vs baseline, per task (pass rate). C = coding, O = spreadsheets.
| task | baseline | caveman | headroom | both |
|---|---|---|---|---|
| C1 · pylint bugfix | ref (2/2) | −40% (2/2) | −57% (2/2) | −57% (2/2) |
| C2 · pytest bugfix | ref (2/2) | +5% (2/2) | −41% (2/2) | +20% (2/2) |
| C3 · sphinx bugfix | ref (2/2) | −68% (2/2) | −10% (2/2) | −42% (2/2) |
| C4 · onboarding doc | ref (2/2) | −15% (2/2) | −13% (2/2) | −21% (2/2) |
| O1 · spreadsheet | ref (0/2) | −24% (0/2) | −29% (0/2) | −48% (0/2) |
| O2 · spreadsheet | ref (2/2) | −12% (2/2) | −35% (2/2) | −29% (2/2) |
| O3 · spreadsheet 2.7MB | ref (0/2) | −13% (0/2) | +21% (0/2) | −47% (0/2) |
no condition separates on quality. O1/O3 are four-way ties (§ why tasks failed); C3's −68% is caveman's biggest legitimate saving. C4, pure writing: −15%.
Why tasks failed#
| task | who failed | why | classification |
|---|---|---|---|
| C3 · sphinx | caveman 0/2 → overturned |
The gate executed stale bytecode: Apple's shared pyc cache invalidates by (whole-second mtime, file size); caveman's fix was a same-size line reorder that landed in the same second as the pre-fix compile. Its patch was byte-identical to baseline's passing fix. Re-gated on fresh bytecode: 2/2. | harness artifact — caught by forensics, gate fixed, verdict reversed |
| O1 · spreadsheet | all four (0/2 each) |
2 of 3 workbooks perfect. Ground truth wants a blank when a lookup has no match (IFERROR(…,"")); every condition wrote Excel-style #N/A instead. |
benchmark semantics — four-way tie, no distortion |
| O3 · 2.7 MB | all four (0/2 each) |
2 of 3 correct. Ground truth blanks the running balance on rows with no activity (IF(A21=0,"",…)); every condition kept computing the cumulative sum. |
benchmark semantics — four-way tie, no distortion |
After the appeal: zero failures separate the conditions — 10/14 everywhere. The scariest finding reversed on forensics: caveman's speed didn't break the sphinx task, it outran the bytecode cache's one-second mtime resolution and broke our gate. Its −42% reasoning cut shows no measured quality cost in this battery — though N=2 says treat that as absence of evidence, not proof of safety.
Caveats#
- N=2 — big deltas robust; single-cell effects are flags, not convictions (C3 proved it: the flag was our bug).
- One model, one battery — each tool's ceiling moves with task mix.
- Native SWE-bench envs (no Docker; official test patches at gate time) — token measurement, not official scores.
- Three harness bugs caught and fixed (per-block stream events; a test-path false-flag; a stale-bytecode gate). C2 re-gated 0/8 → 8/8; C3 caveman 0/2 → 2/2.
Reproduce#
python3 fetch_tasks.py # pin benchmark instances python3 gym.py smoke # auth + activation checks python3 gym.py run --phase full # 4 conditions × 7 tasks × 2 trials python3 analyze.py full # summary + results.json
isolation per run: fresh workspace · --setting-sources "" · pinned tools · scrubbed env · session-scoped plugin/proxy injection · no session persistence
References#
- 1caveman — JuliusBrussee/caveman, vendored at ec83e5b
- 2headroom — headroomlabs-ai/headroom, PyPI headroom-ai 0.34.0 (proxy mode, defaults)
- 3SWE-bench Verified — princeton-nlp, instances pylint-6903 · pytest-7490 · sphinx-10323, official FAIL_TO_PASS/PASS_TO_PASS gates
- 4SpreadsheetBench — RUCKBReasoning, instances 59055 · 13894 · 55392, cell-level comparator port
- 5runner, analyzer, raw event logs, and per-run gates: the skill-gym repo