# gleam-agents

> Run N coding agents with their own goals, shapes and file scopes, graded by real benchmarks. Gleam on the BEAM. This site is docs and published reports only.

- [gleam-agents](https://gleam-agents.coey.dev/docs.md)
- [Catalog: agent shapes](https://gleam-agents.coey.dev/docs/catalog.md)
- [Plans: N seats, N goals](https://gleam-agents.coey.dev/docs/plans.md)
- [Scope: what an agent may touch](https://gleam-agents.coey.dev/docs/scope.md)
- [Graders](https://gleam-agents.coey.dev/docs/graders.md)
- [Pi extension](https://gleam-agents.coey.dev/docs/pi.md)
- [Roadmap](https://gleam-agents.coey.dev/docs/roadmap.md)
- [Reports](https://gleam-agents.coey.dev/api/reports): JSON list of published eval reports

## Full docs

# gleam-agents

Run N coding agents at once. Each agent has its own goal, its own shape (model, harness, tools, budget) and its own file scope. Real benchmarks grade the result. The core is Gleam on the BEAM: one process per agent, so a crash or a runaway loop stays in one place.

## The problem

An eval of coding agents needs three things that most harnesses mix together:

1. **Who the agent is.** Model, tools, budget, harness.
2. **What it may touch.** Files it can write, and nothing else.
3. **Who grades it.** A grader the agent can not see or change.

When these are tangled, you can not tell whether a score came from the model, the harness or a leaky grader.

## The promise

One plan file names N seats. Each seat is shape × goal × scope × grader. `fleet` runs every seat at once, writes one receipt per seat, and a separate script recomputes the score from those receipts.

## How it works

```text
plan.json ──► fleet (Gleam, one BEAM process per seat)
                │  copy seed ► workspace
                │  agent loop: model ◄► tools (list, read, write, edit, grep, run_tests, done)
                │      write/edit pass through the scope guard
                │  visible tests ► grader command (hidden tests, cf-build-bench gate, …)
                ▼
          items/<seat>.json ──► score.mjs (recompute) ──► report ──► /reports
```

- **Shapes** live in `catalog/*.json`. See [Catalog](/docs/catalog).
- **Plans** live in `evals/*/plan.json`. See [Plans](/docs/plans).
- **Scope** is a list of writable paths. Reads may go anywhere inside the workspace. Writes outside the list are refused and counted. See [Scope](/docs/scope).
- **Graders** are shell commands. The agent never sees them. See [Graders](/docs/graders).

## Quick start

```sh
git clone https://github.com/acoyfellow/gleam-agents && cd gleam-agents
export ANTHROPIC_API_KEY=…
scripts/smoke.sh evals/smoke/plan.json
```

To publish a report here, install the Pi extension and sign in. See [Pi extension](/docs/pi).

## What this site is

This site is **docs and published reports only**. It runs no agents and calls no models. You sign in with your Cloudflare account so a report shows who published it. Nothing here bills your account.

## What it does not do

- It is not a sandbox. A seat runs shell commands (tests, graders) on the machine that runs `fleet`. Run it where you accept that.
- One smoke run is an anecdote. The reports say how many runs they cover.
- Only the `native` harness exists today. Other harnesses are planned in [Roadmap](/docs/roadmap).

# Catalog: agent shapes

A shape is one JSON file in `catalog/`. It says who the agent is. It does not say what the agent works on.

```json
{
  "name": "haiku45-native",
  "harness": "native",
  "model": "claude-haiku-4-5",
  "price_per_m": { "in": 1, "out": 5 },
  "max_turns": 30,
  "max_tokens": 16000,
  "tools": ["list_files", "read_file", "write_file", "edit_file", "grep", "run_tests", "done"]
}
```

| Field | Meaning |
|---|---|
| `name` | The id a plan uses |
| `harness` | The loop that drives the model. Today only `native` |
| `model` | Model id sent to the Anthropic Messages API |
| `price_per_m` | List price per million tokens, for the cost column |
| `max_turns` | The run stops with `out_of_turns` after this many model calls |
| `max_tokens` | Output cap per call. Hitting it with no tool call ends the run as `ran_out_of_tokens` |
| `tools` | The tools this shape may call. A tool not on the list is refused |

## Tools

| Tool | What it does | Scope |
|---|---|---|
| `list_files` | Every file in the workspace | read |
| `read_file` | One file | read, inside the workspace |
| `write_file` | Create or replace one file | write list only |
| `edit_file` | Replace one exact, unique snippet | write list only |
| `grep` | Extended regex over the workspace | read |
| `run_tests` | The seat's `test_command` | runs in the workspace |
| `done` | Finish with a summary | — |

## Model endpoint

`fleet` calls `https://api.anthropic.com/v1/messages` with `ANTHROPIC_API_KEY`. Set `LLM_URL` to use a compatible gateway instead, and `LLM_TOKEN_FILE` for a gateway token file.

# Plans: N seats, N goals

A plan is one JSON file. Each seat is one agent.

```json
{
  "name": "smoke-1x",
  "baseline": "what this run is compared with",
  "seats": [
    {
      "id": "s1-sonnet-catalog",
      "shape": "sonnet5-native",
      "goal": "catalog-search",
      "prompt_file": "prompts/catalog-search.md",
      "seed": "../../bench/hard/seed",
      "writable": ["src/catalog.ts", "test/catalog.test.ts"],
      "test_command": "node --test 2>&1 | tail -30",
      "grade_command": "…hidden tests…"
    }
  ]
}
```

| Field | Meaning |
|---|---|
| `shape` | A file name in `catalog/` |
| `goal` | A label for the report |
| `prompt_file` | The task text, relative to the plan |
| `seed` | A folder copied into the seat's workspace, relative to the plan |
| `writable` | Exact files, or folders ending in `/` |
| `test_command` | What `run_tests` runs. The agent sees its output |
| `grade_command` | Runs after the agent stops. Exit 0 is a pass. The agent never sees it |

Every seat starts at the same moment. Seats share nothing but the machine.

## Run

```sh
scripts/smoke.sh evals/smoke/plan.json proof/my-run
```

Output:

```text
proof/my-run/run.json              the run summary fleet wrote
proof/my-run/items/<seat>.json     one receipt per seat
proof/my-run/items/<seat>.grade.txt  grader output
proof/my-run/score.json            recomputed from items/, must agree with run.json
```

`score.mjs` exits 1 if its numbers do not match `run.json`. The agent's own "done" is never the score.

# Scope: what an agent may touch

Each seat gets a workspace (a copy of its seed) and a write list.

- **Reads** may name any path inside the workspace.
- **Writes** (`write_file`, `edit_file`) must match the write list. An exact path matches itself. A path ending in `/` matches everything under it.
- A path that starts with `/`, contains `..`, or is empty is refused for reads and writes.
- Each refusal is returned to the model as an error and counted in the receipt as `scope_denials`.

The rule lives in one file: `src/fleet/scope.gleam`. Its unit tests are in `test/fleet_test.gleam`.

## What scope does not cover

Scope guards the agent's file tools. It does not sandbox `test_command` or `grade_command`, which run as shell commands with your user's rights. A seat's test command is written by you, not by the agent.

## Proof

`evals/smoke/plan-scope.json` tells an agent to write `src/tax.ts`, `../escape.txt` and `notes/ok.txt` with a write list of `notes/`. The grader passes only if the first two were refused and the third landed. Receipt: `proof/scope-1/`.

# Graders

A grader is any shell command. It runs in the seat's workspace after the agent stops. Exit 0 means pass.

## bench/hard hidden tests

`bench/hard` is 17 TypeScript tasks on a small checkout codebase. Each task has hidden tests in `bench/hard/accept/`. A reference solution passes every hidden test, and the seed fails every one. The grader copies the hidden test in, runs it, and removes it:

```sh
mkdir -p test/accept && cp "$FLEET_ROOT"/bench/hard/accept/<task>.test.ts "$FLEET_ROOT"/bench/hard/accept/fixtures.ts test/accept/ \
  && node --test --test-reporter=dot test/accept/<task>.test.ts; e=$?; rm -rf test/accept; exit $e
```

## cf-build-bench gate

[cf-build-bench](https://github.com/cloudflare/cf-build-bench) asks a model to build a Cloudflare Worker from an empty project. Its gate checks four things: it bundles, it typechecks, hidden tests pass in the Workers runtime, and it uses only allowed bindings. `fleet` hands the seat's `src/` to the gate:

```sh
cd "$CF_BUILD_BENCH" && npx tsx src/cli.ts gate --task kv-crud --candidate "$OLDPWD/src" --run-tests
```

The gate runs each command in a pinned Docker toolchain image. Build it once with `node scripts/build-test-toolchain.mjs` in cf-build-bench.

For this grader the seat's `test_command` is the same gate. The agent sees the gate's output, as it does in cf-build-bench's own loop, but never the hidden test source.

## Writing your own

Keep the grader out of the seed and out of the write list. Read only what the agent produced. Print why it failed: the output is saved as `items/<seat>.grade.txt`.

# Pi extension

The extension lets a Pi session run an eval and publish its report here.

## Install

```sh
pi install git:github.com/acoyfellow/gleam-agents
```

The package is the repo root. It points Pi at `pi/index.ts`.

## Sign in

In Pi, run `/gleam-agents login`. It prints a short code and a link. Open the link, sign in with Cloudflare, check the code, and approve. The extension saves a token in `~/.config/gleam-agents/token` (mode 600). The token lasts 30 days. Signing out on the site revokes every token.

`/gleam-agents whoami` shows who the token belongs to.

## Tools

| Tool | What it does |
|---|---|
| `gleam_agents_run` | Runs a plan with `scripts/smoke.sh` and returns the recomputed score |
| `gleam_agents_publish` | Uploads a run's `score.json` and a markdown report to `/reports` |
| `gleam_agents_reports` | Lists published reports |

Set `GLEAM_AGENTS_URL` to use another instance. Set `GLEAM_AGENTS_REPO` to the clone that `gleam_agents_run` uses (default: the installed package folder).

## What the extension sends

Only the report you publish: the recomputed `score.json` and its markdown. It sends no prompts, no code and no model keys.

# Roadmap

Each item is an experiment with a claim that can fail. Status lives in `status.json`.

| Item | Claim | Status |
|---|---|---|
| Native harness in Gleam | N seats, own goals, own scopes, graded by real benchmarks | proven by smoke receipts |
| cf-build-bench grader | The bench gate grades a seat's output unchanged | proven on `kv-crud` |
| Scope guard | Writes outside the list are refused and counted | proven by `scope-probe` |
| Other harnesses | pi, opencode and claude-code run as shapes with the same receipts | not started |
| MCP servers in shapes | A shape can list MCP servers its agent may call | not started |
| Edge workspaces | Each seat works on an Artifacts branch through ply, with no local folder | spike passed in swarm-desk Batch A (16/16 local) |
| Resume | A long seat survives a restart and reconnects by box id | not started |
| Mutant score in the judge | Weak visible tests predict wrong ships | 8/9 in swarm-desk Batch A |