Binary pass@1 is a four-way tie at 7/13, so the signal moves to the frontier-hard tier: reasoning effort compounds for Opus and collapses for Fable. Ranked by hard-tier progress (mean fraction of hidden tests passed on the five hardest tasks).
| # | Model | pass@1 | Hard tier | Cost | Refusals |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.8 highHard-tier leader | 7/13 | 0.42 | $12.77 | 0 |
| 2 | Claude Opus 4.8 low | 7/13 | 0.26 | $6.95 | 0 |
| 3 | Claude Fable 5 low | 7/13 | 0.17 | $22.35 | 2 |
| 4 | Claude Fable 5 high | 7/13 | 0.00 | $26.27 | 1 |
All four configurations score 14/15 but each misses a different task; at equal accuracy, cost spans 4×. Ranked by cost.
| # | Model | Score | Cost | Time | Tokens | $/solved |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.8 highValue leader | 14/15 | $5.00 | 27 min | 0.71 M | $0.36 |
| 2 | Claude Fable 5 low | 14/15 | $5.99 | 27 min | 0.40 M | $0.43 |
| 3 | Claude Fable 5 medium | 14/15 | $11.10 | 44 min | 0.64 M | $0.79 |
| 4 | GPT-5.5 high | 14/15 | $20.43 | 71 min | 2.97 M | $1.46 |
Three models, three vendors, all score 8/10 and fail the same two tasks; resource use spans an order of magnitude. Ranked by cost per solved task.
| # | Model | Score | Cost | Time | Tokens | $/solved |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 lowValue leader | 8/10 | $2.27 | 14 min | 0.14 M | $0.28 |
| 2 | Claude Sonnet 5 high | 8/10 | $8.41 | 59 min | 1.75 M | $1.05 |
| 3 | GLM-5.2 high | 8/10 | $15.81 | 124 min | 10.82 M | $1.98 |
A reasoning-effort sweep: Sonnet 5 scales cleanly with effort while Opus 4.8 stays flat. Ranked by pass@1, then cost per run. Cost is per run (three repeats per cell).
| # | Model | pass@1 | $/run |
|---|---|---|---|
| 1 | Claude Sonnet 5 highAccuracy leader | 100% | $0.13 |
| 2 | Claude Sonnet 5 medium | 99.4% | $0.11 |
| 3 | Claude Opus 4.8 medium | 98.1% | $0.13 |
| 4 | Claude Sonnet 5 low | 97.4% | $0.07 |
| 5 | Claude Opus 4.8 low | 97.4% | $0.11 |
| 6 | Claude Opus 4.8 high | 97.4% | $0.14 |
The pilot: both models solve every real bug at 100%; the gap is entirely cost, concentrated in a 76k-line repository. Ranked by cost.
| # | Model | Score | Cost | $/task | 76k-LOC task |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.8 highValue leader | 4/4 | $5.89 | $1.47 | $2.08 |
| 2 | Claude Sonnet 5 high | 4/4 | $19.88 | $4.97 | $15.15 |
Every number here comes from the open-source harness, computed from full run traces — nothing is hand-entered. Regenerate any row with vulcanbench leaderboard, or reproduce a whole report from scratch: a two-model run costs about ten dollars. The grading underneath is deterministic hidden tests, spelled out in the methodology.
