console-audited · separate keys · reproducible
Every agent says it's fast and cheap. The bill decides.
Real bugs, reported the way humans report them. Each agent runs on its own Anthropic API key, graded by hidden acceptance tests — then its self-reported cost is audited against the provider's actual bill. Both numbers get published.
the audit · 19 bugs, both agents · 2 rounds · anthropic consoleconsole-verified
empryosaid $8.19billed $8.21✓ matched — both audits
pisaid $7.67billed $10.77✗ missed 29% of its bill
fixes 15/19 vs 13/19billed cost −24%wall clock −38%
round 02 · vs pi · real world · 2026-07-16
Five real bugs from hono, zod and ky
No more fixtures. Real merged fix PRs, the original issue text verbatim, git history scrubbed so the answer is unreachable, every fix past the models' training cutoffs. Graded by the fix PR's own regression tests.
protocol tracereal-world
›"RPC headers not merging when function" — hono #5089, verbatim
·hidden acceptance · the fix PR's own regression tests, dropped in after the run
✓7/10 vs 6/10 fixed · billed $7.08 vs $9.19 · wall −32%
findings
post-cutoff fixeshistory-scrubbed reposhaiku sweep 3/5 vs 2/5only empryo cracked zod first tryky bug unsolved 0/4crash-invisible spend caught
- billed cost
- −23%
- steps
- −28%
- wall clock
- −32%
Don't trust our numbers. Run them.
Two API keys and one command reproduce any round — harness, fixtures, hidden checks and raw results are public. Task validation is free.
bun harness-real/run-real.ts --tiers haiku,opus