Token-Saving Skills Review: Caveman, Ponytail, rtk and Headroom on Opus 5
Four Claude Code token savers against the model's own effort dial, on ten SWE-bench tasks hard enough to fail. 276 isolated runs, Opus 5 at extra-high effort.
Verdict#
each dot = 30 runs (10 tasks × 3). whiskers = 95% intervals on the pass rate, clustered by task — they overlap: pass-rate gaps between tools are not significant; cost gaps are (Fig. 2). hover any mark.
task-paired; whiskers = 95% bootstrap intervals over tasks. baseline = $2.52/run at API prices.
caveman halves reasoning (26.3k → 13.2k) and turns (47 → 34) — a deeper cut than the dial's high, keeping 4 more solves. rtk and headroom leave output alone and raise context and turns: Opus 5 works 99% in Bash and already trims its own output, so there was little left to compress (headroom's own ledger: 2.1% removed).
↓ = fewer solves than baseline on that task. darker = more solves of 3.
filled = solved. headroom went 6/6 where the arms disagreed, so both were re-run 7 more times, rule fixed in advance (PLAN.md §5b). The streak did not hold: luck. Every baseline miss traced to an ordinary slip — a dropped -> retann; an ambiguous attrs convention.
Method#
v1 tied at 10/14 everywhere because its tasks were ones every agent solves. v2 picks tasks by measured difficulty, then keeps only those that can move.
| arm | cuts | injected per run as |
|---|---|---|
| caveman | answer prose | plugin (--plugin-dir) |
| ponytail | generated code | plugin; SessionStart hook, ~1.4k tokens of rules |
| rtk | shell output | PreToolUse hook: pytest → rtk pytest |
| headroom | the whole request | localhost proxy via ANTHROPIC_BASE_URL |
| effort-high / low | reasoning | --effort high / low — the free yardstick |
all arms: claude-opus-5 at --effort xhigh, six pinned tools, no MCP, offline shell, fresh workspace, 3 trials per task. Graded gate: official FAIL_TO_PASS / PASS_TO_PASS, environments verified against the gold patch.
Leaks closed before the first real run#
| leak | symptom | fix |
|---|---|---|
| account connectors | claude.ai MCP connectors loaded into the nested agent | --strict-mcp-config + init-event check |
| future commits | git log --all contained the real fix | fetch only the base commit's ancestry |
| the released fix | agent ran pip download of the fixed release | offline shell + lookup audit |
| harness in reach | workspaces beside the held-out tests | opaque paths outside the tree |
| shared temp dir | gold-identical patch scored 0/10 under parallel runs | private TMPDIR per run and gate |
v1 archive — August 2026, Opus 4.8, 56 runs (caveman vs headroom on easy tasks)
v1 · The matchup#
Caveman makes Claude answer in fragments; headroom compresses what Claude reads. One figure each.
* a shorter transcript also shrinks the replayed context (−33%) — a second-order effect, not compression.
Where it sits: an ordinary plugin — a SessionStart hook injects the ruleset into context, and the model itself changes register.
- answer text −47% (1.1k → 590 tok/run): explanations and summaries become fragments
- reasoning −42% (3.8k → 2.2k): the thinking between tool calls shortens with the register
- tool calls −37% (953 → 605): fewer, tighter calls — the code written is untouched
- knock-on: cache-read −33% (462k → 308k): a shorter transcript replays less history each turn
Where it sits: a localhost proxy via ANTHROPIC_BASE_URL. Nothing inside Claude Code changes — so behavior doesn't either.
- what it compresses: the old tool results re-sent every turn — file contents, pytest/log dumps, JSON — each block typed and routed (JSON structurally crushed · code trimmed AST-aware · prose/logs via a small model)
- cache-read −45% (462k → 254k tok/run): the history is compressed once, and the saving repeats every turn after
- what it never touches: everything Claude generates — reasoning, answers, tool calls all ≈ unchanged; same fixes, same turn count
v1 · Method#
| model | conditions | tasks | trials | total runs | wall | errors |
|---|---|---|---|---|---|---|
| claude-opus-4-8 | 4 | 7 | 2 | 56 | 2h16m | 0 |
| task | source | what the agent must do |
|---|---|---|
| C1 | SWE-bench Verified | pylint — fix a crash when the container CPU quota is fractional |
| C2 | SWE-bench Verified | pytest — fix a regression in dynamically-added xfail markers |
| C3 | SWE-bench Verified | sphinx — fix literalinclude prepend/dedent whitespace handling |
| C4 | writing task | contributor onboarding doc for the pylint codebase |
| O1 | SpreadsheetBench | match items across columns, return positions |
| O2 | SpreadsheetBench | assign unique code numbers by formula |
| O3 | SpreadsheetBench | cumulative balance across 2.7 MB workbooks |
v1 · Where the tokens go#
replayed context vs output ≈ 80:1. berry tick = all generated output; caveman's −42% cut is invisible at this scale — the point.
v1 · What a solved task costs#
Divide by solved tasks and the podium reorders.
caveman and the stack tie at −30%. an earlier version of this chart had caveman third — a stale-bytecode harness bug had failed its correct sphinx fixes (§ why tasks failed).
| condition | solved | est $/run | est $/solved | wall s/run |
|---|---|---|---|---|
| baseline | 10/14 | 0.528 | 0.740 | 96 |
| caveman | 10/14 | 0.367 | 0.514 −30% | 60 |
| headroom | 10/14 | 0.431 | 0.604 −18% | 148 |
| both | 10/14 | 0.369 | 0.516 −30% | 98 |
v1 · Per task#
Cost vs baseline, per task (pass rate). C = coding, O = spreadsheets.
| task | baseline | caveman | headroom | both |
|---|---|---|---|---|
| C1 · pylint bugfix | ref (2/2) | −40% (2/2) | −57% (2/2) | −57% (2/2) |
| C2 · pytest bugfix | ref (2/2) | +5% (2/2) | −41% (2/2) | +20% (2/2) |
| C3 · sphinx bugfix | ref (2/2) | −68% (2/2) | −10% (2/2) | −42% (2/2) |
| C4 · onboarding doc | ref (2/2) | −15% (2/2) | −13% (2/2) | −21% (2/2) |
| O1 · spreadsheet | ref (0/2) | −24% (0/2) | −29% (0/2) | −48% (0/2) |
| O2 · spreadsheet | ref (2/2) | −12% (2/2) | −35% (2/2) | −29% (2/2) |
| O3 · spreadsheet 2.7MB | ref (0/2) | −13% (0/2) | +21% (0/2) | −47% (0/2) |
no condition separates on quality — and no task here could have (§ method). O1/O3 are four-way ties (§ why tasks failed); C3's −68% is caveman's biggest legitimate saving. C4, pure writing: −15%.
v1 · Why tasks failed#
| task | who failed | why | classification |
|---|---|---|---|
| C3 · sphinx | caveman 0/2 → overturned |
The gate executed stale bytecode: Apple's shared pyc cache invalidates by (whole-second mtime, file size); caveman's fix was a same-size line reorder that landed in the same second as the pre-fix compile. Its patch was byte-identical to baseline's passing fix. Re-gated on fresh bytecode: 2/2. | harness artifact — caught by forensics, gate fixed, verdict reversed |
| O1 · spreadsheet | all four (0/2 each) |
2 of 3 workbooks perfect. Ground truth wants a blank when a lookup has no match (IFERROR(…,"")); every condition wrote Excel-style #N/A instead. |
benchmark semantics — four-way tie, no distortion |
| O3 · 2.7 MB | all four (0/2 each) |
2 of 3 correct. Ground truth blanks the running balance on rows with no activity (IF(A21=0,"",…)); every condition kept computing the cumulative sum. |
benchmark semantics — four-way tie, no distortion |
After the appeal: zero failures separate the conditions — 10/14 everywhere. The scariest finding reversed on forensics: caveman's speed didn't break the sphinx task, it outran the bytecode cache's one-second mtime resolution and broke our gate. Its −42% reasoning cut shows no measured quality cost in this battery — and this battery could not have shown one: every task sat at the ceiling or the floor (§ method). That is the question v2 exists to answer.
Caveats#
- 30 runs per arm sees ~20-point pass-rate drops, not 5. Cost differences are solid; pass-rate differences between tools are leads.
- 7 of 10 tasks are ones the baseline always solves: v2 mostly tests whether a tool breaks something.
- Dollar figures are the CLI's estimate at API prices; the runs billed a subscription.
- rtk 0.49.0 and headroom 0.37.0 at defaults, on a Bash-first agent. Log- or JSON-heavy work is a different regime.
- Native environments, not the official Docker images: not SWE-bench scores.
Reproduce#
python3 fetch_tasks.py # pin benchmark instances python3 gym.py smoke # auth + activation checks python3 gym.py run --phase full # 4 conditions × 7 tasks × 2 trials python3 analyze.py full # summary + results.json # v2 python3 fetch_tasks.py swebench --gold # pin instances (+ answer keys, for verification only) python3 bin/verify_swebench.py <instance> # red at base, green with the gold patch, in this env rm -rf .cache/gold # no answer key on disk while agents run python3 gym.py run --phase calib --jobs 4 # baseline only python3 analyze_v2.py calib --freeze # calibration table + frozen task list python3 gym.py run --phase v2 --jobs 4 # tool arms on the frozen tasks python3 analyze_v2.py report
isolation per run: fresh workspace · --setting-sources "" · pinned tools · scrubbed env · session-scoped plugin/proxy injection · no session persistence · v2 adds: no MCP · history cut at the base commit · offline shell · workspace outside the harness · private temp dir · lookup audit
References#
- 1caveman — JuliusBrussee/caveman, vendored at ec83e5b
- 2headroom — headroomlabs-ai/headroom, PyPI headroom-ai 0.34.0 (proxy mode, defaults)
- 3SWE-bench Verified — princeton-nlp, instances pylint-6903 · pytest-7490 · sphinx-10323, official FAIL_TO_PASS/PASS_TO_PASS gates
- 4SpreadsheetBench — RUCKBReasoning, instances 59055 · 13894 · 55392, cell-level comparator port
- 5runner, analyzer, raw event logs, and per-run gates: the skill-gym repo
- 6ponytail — DietrichGebert/ponytail, vendored at v4.10.0 (1d95ff7); rtk — rtk-ai/rtk v0.49.0, release binary pinned by SHA-256; headroom-ai 0.37.0 for v2
- 7per-task solve rates — swe-bench/experiments, 10 Verified submissions (Nov 2025 – Sep 2026)
- 8prior art — JetBrains AI blog, July 2026: caveman, rtk and ponytail on SkillsBench (rtk +7.6% cost; ponytail −10.3%); no quality difference detected, and not powered to prove equivalence
- 9v2 design, amendments and selection rule: PLAN.md in the repo