ESCAPES

The six rules

Live at 1 July: the bug was already in the code at the snapshot and was reported after it, so a scan on 1 July could have caught it. These 235 are the bugs scored on this page.

By kind of failure

A bug can carry several classes; the table counts its main one. Classes are keyword guesses: a hand check of 20 corrected 4.

None of these products appears in any list breken develops, tunes or measures on. All 33 are now on the held-out lists our development guards read.

// one entry, shape only
{
  "id": "<owner>/<product>#<issue>",  "direction": "A",  "language": "Go",
  "issue": { "url": "…/issues/<n>", "createdAt": "<date>", "authorAssociation": "NONE",
             "userWords": "<what the user saw>", "signal": { "reactions": 0, "comments": 6 } },
  "fix": { "url": "…/pull/<n>", "mergedAt": "<date>", "howLinked": "closing reference" },
  "preFixSha": "<the fix's first parent>",  "fixSha": "<the fix commit>",
  "introducedAt": "<date>",
  "sites": [ { "path": "src/…/update.ts", "lines": [118, 131] } ],
  "tests": [ "src/…/update.test.ts" ],
  "snapshot": { "live": true },
  "class": ["broken-flow"],  "severity": "medium",  "executable": true
}

Every system runs under the same spend cap per product. breken reports only what it would show a customer; the coding agents are asked for at most 10 findings per product; static rules return every hit. Code-review bots see a diff, not a product, so pointing one at the change that introduced a bug is an aimed run: published apart, never on the leaderboard.

What a system is given

Two run modes

This page shows snapshot runs: one scan on 1 July, measured against what users reported in the months after.

Sent to Claude Code and Codex CLI as a fresh session: no memory, no project instructions, no tools beyond what the CLI ships. The product name and the commit are the only things filled in.

You are testing <owner/product> at the commit checked out in the current directory.
kind A: It is a product that people or programs use (a web app, an API or an MCP server).
kind B: It is an SDK, CLI or toolkit that people use to act on other things.

Find bugs that real users would run into in normal use: wrong results, broken flows,
lost data, wrong access, crashes, or (for a library or CLI) wrong output or failures
for valid use. You may read the code and docs, build it, run its tests and run it.
Report at most 10 bugs, best first. For each one, if you can, give a reproduction:
a shell command (and any new test file's full content) that FAILS on this commit
because of the bug and would pass once it is fixed.

Answer with JSON only:
{"findings":[{"id":"f1","title":"one sentence","failure":"what the user or caller sees go wrong",
  "steps":"how a user gets there",
  "reproduction":{"command":"...","files":{"path":"content"},"setup":"install command, if needed"},
  "files":["the source files at fault"]}]}

Finding contract

{
  "id": "f-012",
  "title": "Refund after a partial capture returns the full authorised amount",
  "failure": "The ledger refunds the authorised total, not the captured part",
  "steps": ["create an order", "capture part of it", "refund"],
  "reproduction": { "command": "npm test -- refunds.partial",
                    "files": { "test/refunds.partial.test.ts": "…" }, "setup": "npm ci" },
  "files": ["src/payments/refund.ts"],
  "location": { "path": "src/payments/refund.ts", "line": 88 }
}

A finding counts as a catch only if it names the same user-visible failure the user later reported. Three rungs decide it, in order. The higher rung wins.

Metrics

What is not a match

# build (read-only GitHub access; writes only to the sealed directory)
node bench/escapes/build.mjs --phase mine
node bench/escapes/build.mjs --phase verify
node bench/escapes/build.mjs --phase assemble --max-per-product 12 --min-per-product 5
node bench/escapes/freeze.mjs

# run a system
node bench/escapes/arms/run.mjs --arm breken    --label v1 --mode snapshot --cap-usd <usd>
node bench/escapes/arms/run.mjs --arm plain-llm --label v1 --mode snapshot --cap-usd <usd>
node bench/escapes/arms/run.mjs --arm static    --label v1 --mode snapshot

# match and score
node bench/escapes/match.mjs --label v1
node bench/escapes/score.mjs --label v1 --judge --model-env judge.env --cap-usd <usd>
node bench/escapes/score.mjs --label v1