Tank Duel Benchmark

14 copies of one three.js game, built from the same spec by 7 build setups, two runs each, to choose how graph-loop should build. Every run went through the loop's stages: build, test suite, an independent review, and up to two repairs. Then 14 hidden checks, which no builder saw, graded the game through its own controls.

By Babak Bandpey

Each setup, averaged over its two runs

What each run cost

First build Repairs

Play them

Each game opens as it was built. Drive with W, A, S, D or the arrow keys, and fire with Space.

What the runs showed

Every run passed all 14 hidden checks, and the reviewer accepted none: each round it found a new geometry edge case, such as a shell grazing a tank or a tank firing while pressed against a wall. So the checks could not separate the setups; cost and time did.
Opus alone (E) was the cheapest: $1.71 a run on average, with first builds of 7 and 12 minutes. Sonnet alone (A, graph-loop today) took 19 and 24 minutes to build and cost $2.08 a run.
Handing work to Sonnet workers cost more, not less. Opus directing Sonnet (B, C, F) cost $2.30 to $2.89 a run, against $1.71 for Opus alone, for the same score.
Told to split the work and start as many Haiku workers as it chose (G), Opus started 3 and 4, mostly to write tests, and wrote the game logic itself. Haiku's share of the cost was 3 to 5 cents; the runs cost $1.91 and $2.28, close to Sonnet alone and more than Opus alone. In B, C and F, Opus never picked Haiku, and Sonnet with a Haiku worker (D) never handed anything over.
G changed two things at once, the worker model and the wording of the instruction, so its result cannot say which of them mattered. It also ran after the others, on a quieter machine, so its 18 and 22 minutes are not comparable with the rest.
The runs wrote 386 to 621 lines of game code and 234 to 439 lines of tests. The two G runs, whose tests Haiku wrote, have among the most tests: 439 and 399 lines.
Continuing the same session for a repair paid for itself: two repairs cost $0.13 to $0.66 that way, and $1.06 to $1.28 from a fresh session given a brief (F).
After the build, the review was the slowest stage: 5 to 9 minutes a run over its three rounds.
A resumed session reports its running total, so adding up each call's figure counts earlier calls again. The numbers here charge each call only what it added, as graph-loop's own budget does.

How it was measured

The setups: A, Sonnet 5.5 builds alone (graph-loop today). B, Opus 5.5 directs Sonnet and Haiku workers, and Sonnet repairs. C, the same, with Opus repairing. D, Sonnet builds with a Haiku worker. E, Opus builds alone. F, as C, but each repair starts a fresh Opus session from a short brief instead of continuing the old one. G, Opus splits the work and starts as many Haiku workers as it decides, several at once where steps are independent, and repairs in the same session.
Builds ran at high effort, repairs at medium, workers at xhigh. The reviewer was the same for every run: gpt-6.1-sol at medium.
Each run had a $10 Claude budget and at most two repairs. A run that the reviewer still refused after its last repair would still become a pull request in graph-loop, with the findings attached.
The first twelve runs (A to F) ran at the same time on one machine, and G's two ran later, so the test suite's times include that load.
One new game is a different job from graph-loop's usual small edits to existing code, so the result is one input to graph-loop's defaults, not the whole answer.