Merged open-source pull requests, graded by deterministic hidden tests.
The harness, live — an actual v1-micro run on Claude Sonnet 5 (low): 26 sandboxed tasks, graded by hidden tests, 25 pass at $0.82 total.
Most coding benchmarks saturate: when every frontier model scores 98 to 100 percent, the benchmark is measuring its own ceiling, not the models. VulcanBench is built to stay discriminating by sourcing tasks from real engineering work that current models cannot all solve, and by treating economics as a first-class result.
Every task is a real pull request merged into a production open-source project (flask, sqlglot, aiohttp, urllib3, poetry, redis, and more), each merged after the models' training cutoffs so the fix is genuinely novel and cannot be recalled from memory. A model receives the repository sliced at the pull request's base commit plus a terse issue describing the symptom, exactly as an engineering team would hand off a ticket. It is graded by the project's own hidden tests: no LLM judge, no rubric, just pass or fail, run in a network-isolated sandbox.
Each task's pass or fail is decided by hidden tests validated to fail on the base commit and pass on the merged fix. No model judges another model.
Every pull request was merged after the evaluated models' training cutoffs, so no model has seen the solution. Memorization cannot help.
Cost, wall-clock time, agent steps, and tokens are recorded alongside accuracy. When correctness converges, efficiency is the signal.
Fig. 1 — When correctness converges, economics is the signal. Costs include each provider's prompt-cache discount. Read the full report →
Each measured run gets a dedicated page: the model card, the findings, the per-task results, and the downloadable report. Four published so far.
Browse the reports →Every task, hidden test, gold patch, per-task Docker image, and the full grading harness. Reproduce any result; a two-model run costs about ten dollars.
View on GitHub →