{"id":"2026-10-24-59a37162","title":"Gleam fleet vs Elixir swarm, bench/hard, haiku-4-5, 1x","createdAt":1792878309014,"summary":{"plan":"hard-17x1-haiku45","baseline":"swarm-desk Elixir native loop, evals/baseline-hard-final.json: haiku45-full 45/51 = 0.882 hidden pass, $0.153 per seat","seats":17,"graded":17,"passed":14,"visible_passed":17,"scope_denials":0,"usd":2.8505,"wall_ms":172210,"errors":[],"per_seat":[{"seat":"h-allocate-remainder","shape":"haiku45-native","goal":"allocate-remainder","outcome":"done","turns":15,"tests_run":5,"scope_denials":0,"usd":0.1195,"ms":38230,"visible_pass":true,"graded_pass":false},{"seat":"h-business-days","shape":"haiku45-native","goal":"business-days","outcome":"done","turns":20,"tests_run":6,"scope_denials":0,"usd":0.1753,"ms":49889,"visible_pass":true,"graded_pass":true},{"seat":"h-cart-merge","shape":"haiku45-native","goal":"cart-merge","outcome":"done","turns":12,"tests_run":2,"scope_denials":0,"usd":0.0851,"ms":29954,"visible_pass":true,"graded_pass":true},{"seat":"h-catalog-search","shape":"haiku45-native","goal":"catalog-search","outcome":"done","turns":15,"tests_run":5,"scope_denials":0,"usd":0.1364,"ms":45687,"visible_pass":true,"graded_pass":false},{"seat":"h-csv-parse","shape":"haiku45-native","goal":"csv-parse","outcome":"done","turns":18,"tests_run":4,"scope_denials":0,"usd":0.163,"ms":48421,"visible_pass":true,"graded_pass":true},{"seat":"h-discount-cap","shape":"haiku45-native","goal":"discount-cap","outcome":"done","turns":14,"tests_run":4,"scope_denials":0,"usd":0.1055,"ms":36756,"visible_pass":true,"graded_pass":true},{"seat":"h-format-negative-code","shape":"haiku45-native","goal":"format-negative-code","outcome":"done","turns":19,"tests_run":4,"scope_denials":0,"usd":0.1807,"ms":46771,"visible_pass":true,"graded_pass":true},{"seat":"h-installments","shape":"haiku45-native","goal":"installments","outcome":"done","turns":14,"tests_run":2,"scope_denials":0,"usd":0.089,"ms":33935,"visible_pass":true,"graded_pass":true},{"seat":"h-invoice-credit","shape":"haiku45-native","goal":"invoice-credit","outcome":"done","turns":11,"tests_run":2,"scope_denials":0,"usd":0.0786,"ms":28349,"visible_pass":true,"graded_pass":true},{"seat":"h-invoice-rounding-audit","shape":"haiku45-native","goal":"invoice-rounding-audit","outcome":"done","turns":28,"tests_run":7,"scope_denials":0,"usd":0.3902,"ms":87974,"visible_pass":true,"graded_pass":true},{"seat":"h-money-words","shape":"haiku45-native","goal":"money-words","outcome":"done","turns":17,"tests_run":4,"scope_denials":0,"usd":0.1768,"ms":64176,"visible_pass":true,"graded_pass":true},{"seat":"h-parse-strict","shape":"haiku45-native","goal":"parse-strict","outcome":"done","turns":18,"tests_run":7,"scope_denials":0,"usd":0.1921,"ms":60441,"visible_pass":true,"graded_pass":true},{"seat":"h-report-currency-mix","shape":"haiku45-native","goal":"report-currency-mix","outcome":"done","turns":14,"tests_run":4,"scope_denials":0,"usd":0.1244,"ms":41941,"visible_pass":true,"graded_pass":true},{"seat":"h-shipping-weight","shape":"haiku45-native","goal":"shipping-weight","outcome":"done","turns":8,"tests_run":1,"scope_denials":0,"usd":0.0471,"ms":22754,"visible_pass":true,"graded_pass":true},{"seat":"h-sku-glob","shape":"haiku45-native","goal":"sku-glob","outcome":"out_of_turns","turns":30,"tests_run":9,"scope_denials":0,"usd":0.5592,"ms":171701,"visible_pass":true,"graded_pass":false},{"seat":"h-tax-cent-bug","shape":"haiku45-native","goal":"tax-cent-bug","outcome":"done","turns":19,"tests_run":5,"scope_denials":0,"usd":0.1841,"ms":46708,"visible_pass":true,"graded_pass":true},{"seat":"h-tax-inclusive","shape":"haiku45-native","goal":"tax-inclusive","outcome":"done","turns":7,"tests_run":1,"scope_denials":0,"usd":0.0435,"ms":23590,"visible_pass":true,"graded_pass":true}],"agrees_with_run_json":true},"markdown":"# Gleam fleet vs Elixir swarm on bench/hard\n\n**Verdict: proven.** The Gleam native harness passes 14/17 hidden tests (0.824). The Elixir swarm baseline passes 45/51 (0.882). The threshold is 0.732.\n\n| | Hidden pass | $ per task | Runs |\n|---|---|---|---|\n| gleam-agents, haiku45-native | 14/17 = 0.824 | 0.1677 | 1 |\n| swarm-desk Elixir, haiku45-full | 45/51 = 0.882 | 0.1531 | 3 |\n\n## Setup\n\n- 17 seats, one per bench/hard task. All start at once, one BEAM process each.\n- Same model (claude-haiku-4-5), same seed, same hidden tests as the baseline.\n- Each seat may write only `src/` and `test/`. Hidden tests are copied in after the agent stops.\n- The score is recomputed from per-seat receipts and must agree with the run summary.\n\n## Misses\n\n| Seat | Outcome | Turns |\n|---|---|---|\n| h-allocate-remainder | done | 15 |\n| h-catalog-search | done | 15 |\n| h-sku-glob | out_of_turns | 30 |\n\nTwo of the three misses are tasks the baseline also missed (allocate-remainder, catalog-search). sku-glob ran out of its 30 turns.\n\n## Not verified\n\n- One run of 17 is a parity smoke check, not a ranking. The band is wide.\n- Only the native harness ran. Other harnesses are not tested.\n- The cf-build-bench gate was tested on one task (kv-crud, 1/1), not in this run.\n"}