Gleam fleet vs Elixir swarm, bench/hard, haiku-4-5, 1x

2026-10-24T21:45:09.014Z

Gleam fleet vs Elixir swarm on bench/hard

Verdict: proven. The Gleam native harness passes 14/17 hidden tests (0.824). The Elixir swarm baseline passes 45/51 (0.882). The threshold is 0.732.

	Hidden pass	$ per task	Runs
gleam-agents, haiku45-native	14/17 = 0.824	0.1677	1
swarm-desk Elixir, haiku45-full	45/51 = 0.882	0.1531	3

Setup

17 seats, one per bench/hard task. All start at once, one BEAM process each.
Same model (claude-haiku-4-5), same seed, same hidden tests as the baseline.
Each seat may write only src/ and test/. Hidden tests are copied in after the agent stops.
The score is recomputed from per-seat receipts and must agree with the run summary.

Misses

Seat	Outcome	Turns
h-allocate-remainder	done	15
h-catalog-search	done	15
h-sku-glob	out_of_turns	30

Two of the three misses are tasks the baseline also missed (allocate-remainder, catalog-search). sku-glob ran out of its 30 turns.

Not verified

One run of 17 is a parity smoke check, not a ranking. The band is wide.
Only the native harness ran. Other harnesses are not tested.
The cf-build-bench gate was tested on one task (kv-crud, 1/1), not in this run.

Every seat

Seat	Shape	Outcome	Turns	$	Hidden
h-allocate-remainder	haiku45-native	done	15	0.1195	fail
h-business-days	haiku45-native	done	20	0.1753	pass
h-cart-merge	haiku45-native	done	12	0.0851	pass
h-catalog-search	haiku45-native	done	15	0.1364	fail
h-csv-parse	haiku45-native	done	18	0.163	pass
h-discount-cap	haiku45-native	done	14	0.1055	pass
h-format-negative-code	haiku45-native	done	19	0.1807	pass
h-installments	haiku45-native	done	14	0.089	pass
h-invoice-credit	haiku45-native	done	11	0.0786	pass
h-invoice-rounding-audit	haiku45-native	done	28	0.3902	pass
h-money-words	haiku45-native	done	17	0.1768	pass
h-parse-strict	haiku45-native	done	18	0.1921	pass
h-report-currency-mix	haiku45-native	done	14	0.1244	pass
h-shipping-weight	haiku45-native	done	8	0.0471	pass
h-sku-glob	haiku45-native	out_of_turns	30	0.5592	fail
h-tax-cent-bug	haiku45-native	done	19	0.1841	pass
h-tax-inclusive	haiku45-native	done	7	0.0435	pass

Recomputed score.json

{
  "plan": "hard-17x1-haiku45",
  "baseline": "swarm-desk Elixir native loop, evals/baseline-hard-final.json: haiku45-full 45/51 = 0.882 hidden pass, $0.153 per seat",
  "seats": 17,
  "graded": 17,
  "passed": 14,
  "visible_passed": 17,
  "scope_denials": 0,
  "usd": 2.8505,
  "wall_ms": 172210,
  "errors": [],
  "per_seat": [
    {
      "seat": "h-allocate-remainder",
      "shape": "haiku45-native",
      "goal": "allocate-remainder",
      "outcome": "done",
      "turns": 15,
      "tests_run": 5,
      "scope_denials": 0,
      "usd": 0.1195,
      "ms": 38230,
      "visible_pass": true,
      "graded_pass": false
    },
    {
      "seat": "h-business-days",
      "shape": "haiku45-native",
      "goal": "business-days",
      "outcome": "done",
      "turns": 20,
      "tests_run": 6,
      "scope_denials": 0,
      "usd": 0.1753,
      "ms": 49889,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-cart-merge",
      "shape": "haiku45-native",
      "goal": "cart-merge",
      "outcome": "done",
      "turns": 12,
      "tests_run": 2,
      "scope_denials": 0,
      "usd": 0.0851,
      "ms": 29954,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-catalog-search",
      "shape": "haiku45-native",
      "goal": "catalog-search",
      "outcome": "done",
      "turns": 15,
      "tests_run": 5,
      "scope_denials": 0,
      "usd": 0.1364,
      "ms": 45687,
      "visible_pass": true,
      "graded_pass": false
    },
    {
      "seat": "h-csv-parse",
      "shape": "haiku45-native",
      "goal": "csv-parse",
      "outcome": "done",
      "turns": 18,
      "tests_run": 4,
      "scope_denials": 0,
      "usd": 0.163,
      "ms": 48421,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-discount-cap",
      "shape": "haiku45-native",
      "goal": "discount-cap",
      "outcome": "done",
      "turns": 14,
      "tests_run": 4,
      "scope_denials": 0,
      "usd": 0.1055,
      "ms": 36756,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-format-negative-code",
      "shape": "haiku45-native",
      "goal": "format-negative-code",
      "outcome": "done",
      "turns": 19,
      "tests_run": 4,
      "scope_denials": 0,
      "usd": 0.1807,
      "ms": 46771,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-installments",
      "shape": "haiku45-native",
      "goal": "installments",
      "outcome": "done",
      "turns": 14,
      "tests_run": 2,
      "scope_denials": 0,
      "usd": 0.089,
      "ms": 33935,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-invoice-credit",
      "shape": "haiku45-native",
      "goal": "invoice-credit",
      "outcome": "done",
      "turns": 11,
      "tests_run": 2,
      "scope_denials": 0,
      "usd": 0.0786,
      "ms": 28349,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-invoice-rounding-audit",
      "shape": "haiku45-native",
      "goal": "invoice-rounding-audit",
      "outcome": "done",
      "turns": 28,
      "tests_run": 7,
      "scope_denials": 0,
      "usd": 0.3902,
      "ms": 87974,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-money-words",
      "shape": "haiku45-native",
      "goal": "money-words",
      "outcome": "done",
      "turns": 17,
      "tests_run": 4,
      "scope_denials": 0,
      "usd": 0.1768,
      "ms": 64176,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-parse-strict",
      "shape": "haiku45-native",
      "goal": "parse-strict",
      "outcome": "done",
      "turns": 18,
      "tests_run": 7,
      "scope_denials": 0,
      "usd": 0.1921,
      "ms": 60441,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-report-currency-mix",
      "shape": "haiku45-native",
      "goal": "report-currency-mix",
      "outcome": "done",
      "turns": 14,
      "tests_run": 4,
      "scope_denials": 0,
      "usd": 0.1244,
      "ms": 41941,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-shipping-weight",
      "shape": "haiku45-native",
      "goal": "shipping-weight",
      "outcome": "done",
      "turns": 8,
      "tests_run": 1,
      "scope_denials": 0,
      "usd": 0.0471,
      "ms": 22754,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-sku-glob",
      "shape": "haiku45-native",
      "goal": "sku-glob",
      "outcome": "out_of_turns",
      "turns": 30,
      "tests_run": 9,
      "scope_denials": 0,
      "usd": 0.5592,
      "ms": 171701,
      "visible_pass": true,
      "graded_pass": false
    },
    {
      "seat": "h-tax-cent-bug",
      "shape": "haiku45-native",
      "goal": "tax-cent-bug",
      "outcome": "done",
      "turns": 19,
      "tests_run": 5,
      "scope_denials": 0,
      "usd": 0.1841,
      "ms": 46708,
      "visible_pass": true,
      "graded_pass": true
    },
    {
      "seat": "h-tax-inclusive",
      "shape": "haiku45-native",
      "goal": "tax-inclusive",
      "outcome": "done",
      "turns": 7,
      "tests_run": 1,
      "scope_denials": 0,
      "usd": 0.0435,
      "ms": 23590,
      "visible_pass": true,
      "graded_pass": true
    }
  ],
  "agrees_with_run_json": true
}

JSON