The controlled test
One coding job, five runs of every setup: fix four real bugs in sqlite-utils, add a feature, and answer a teammate's question. Claude Sonnet 5.5 at high effort, graded by hidden tests. Every setup solved all five tasks with no regressions, so cost is the only difference.
A two-sentence CLAUDE.md asking for short replies and the least code that works saved almost as much as ponytail, with no add-on at all.
| Setup | Median cost | Against plain | Real or noise? |
|---|---|---|---|
| Plain Claude Code | $0.71 | – | |
| ponytail | $0.62 | −13% | Real |
| Two-sentence CLAUDE.mdShort replies, least code, no add-on | $0.64 | −10% | Real |
| rtk | $0.69 | −3% | Noise |
| caveman | $0.82 | +15% | Noise |
| context-mode | $0.91 | +27% | Real |
Bloons-style tower defense
One prompt, an empty folder and nobody answering questions. Every build is playable. On Opus, ponytail wrote 37% less game code than plain Claude Code and finished half an hour sooner.
Claude Opus 5.5
-
Plain
94 min, $19.16
223 KB of game code
-
ponytail
−11%60 min, $17.05
140 KB of game code
Claude Sonnet 5.5
-
Plain
32 min, $4.32
128 KB of game code
-
ponytail
52 min, $13.67*
114 KB of game code
*This run hit a usage limit and resumed hours later, which reloads the whole session. Runs that paused cost about three times as much, so this pair's costs don't compare.
Squishy gummy bears
One prompt, one shot: gummy bears you can grab, stretch and throw, in three.js. Ponytail's Opus build cost less than half as much and is about half the size. Drag a bear to see whether anything went missing.
Claude Opus 5.5
-
Plain
35 min, $7.78
68 KB, one file
-
ponytail
−54%16 min, $3.61
35 KB, one file
Claude Sonnet 5.5
-
Plain
21 min, $3.88*
60 KB, one file
-
ponytail
23 min, $2.57
45 KB, one file
*The plain run hit a usage limit and resumed, which raises its cost, so ponytail's lead here is smaller than it looks.
A landing page
The long brief from One landing page, ten ways: a waitlist page for Fernway, a made-up HR startup, with no design skills installed. Ponytail made the cheapest page with both models, and shipped fewer images than plain both times. Hover a page to scroll down it, or click to open it.Tap a page to open it.
Claude Opus 5.5
-
Plain
10 min, $2.50
487 KB, 11 images
-
ponytail
−42%6 min, $1.44
247 KB, 7 images
Claude Sonnet 5.5
-
Plain
9 min, $1.26
242 KB, 13 images
-
ponytail
−55%4 min, $0.57
36 KB, 3 images
The rest of the results
Every add-on also ran the builds above, plus two jobs with exact answers. No add-on changed accuracy on either, so they only moved the bill.
- caveman cost more than plain on four of five Opus jobs. Its replies were terse, but its code and tool calls weren't, and its instructions add about 3,800 tokens to every session.
- rtk and context-mode came in cheaper on some jobs and dearer on others. In the controlled test, rtk's saving was noise and context-mode cost 27% more.
- Data analysis on NFL and Federal Reserve data: every setup got all 12 answers right. Ponytail was cheapest with Opus ($0.65 against plain's $0.72) and tied rtk with Sonnet ($0.22 against $0.31).
- An interactive dashboard: every build got all 24 numbers right, and ponytail was cheapest with both models (Opus $2.10 against $3.44, Sonnet $1.40 against $1.64), even though both of its runs paused for a usage limit. The dashboards bundle market data that can't be republished, so they aren't posted.