Trajectory Lab: Scoring Structure, Not Length

I handed the same auth-refactor task to the same model twice. The first time I gave three agents a vague brief ("clean up the auth module") and walked away. Two of them wrote the same file, the lead reported a clean merge that didn't actually reconcile the two versions, and the run scored 31 out of 100. An F.

The second time I gave the same three agents a tight, bounded brief: one file each, explicit boundaries, a verification step. Same model, same task, same swarm topology. It scored 100. An A.

Last time, in Scaffolding Over Horsepower, I argued that the leverage in agentic coding is the structure around the loop, not the model or the agent count. That was an argument. The two scores above are what happened when I tried to turn the argument into a number: a system I've been building called Trajectory Lab. A claim you can't measure is just an opinion, and I wanted to know if mine survived contact with data.

It mostly did. But the part that matters most, the part that justifies the whole tool, isn't proven yet, and I'm not shipping until it is. This post is the build and the gap, in that order.

Score the shape, not the length

The whole design rests on one refusal: length is not a signal. A run isn't better because it took more steps or worse because it took fewer. That sounds obvious until you notice how much agent tooling quietly rewards verbosity. So every metric in Trajectory Lab is a ratio or an event count: something that doesn't move just because the run got longer.

It scores a trajectory: one agent run, an ordered set of steps by one or more agents toward a goal. Each step is classified into one of read | write | verify | plan | merge, and steps are attributed to the leaf actor that ran them: a sub-agent's writes belong to the sub-agent, not the orchestrator. From there, four metrics, each between 0 and 1, where W is the number of writes:

None of these counts steps. A 4-step run and a 40-step run with the same shape score the same. That's the point, and later it turns out to be the empirically correct choice, not just an aesthetic one.

Health is then 100 minus a set of named, attributable penalties, each one traceable to specific steps in the run, not a black-box number. The weights aren't guesses; I'll get to where they came from.

p_ground  = -round(14 * (1 - grounding))
p_valid   = -round(26 * (1 - validation))
p_focus   = -round(28 * (1 - focus))
p_collide = -round(min(collisions   * 15, 36))   // capped
p_merge   = -round(min(failedMerges * 14, 28))   // capped

health = clamp(100 + p_ground + p_valid + p_focus + p_collide + p_merge, 0, 100)
// grade: A >= 90 · B >= 75 · C >= 60 · D >= 45 · else F

And every penalty surfaces as a human-readable finding that points at the real steps that caused it. Severity isn't cosmetic: it decides what the CI gate will block on, so it's load-bearing.

One core, two compile targets

Here's the architecture decision I'm proudest of, because it heads off the one bug a scoring tool cannot survive: the dashboard and CI disagreeing on a score. If the web tool says an agent run is an A and the CI gate says it's a D, the whole product is worthless: nobody can trust either number. A second implementation that can drift is, for this product, the unforgivable bug.

So there is exactly one scorer: a pure Rust core (types, classifier, scoring engine, gate) with no I/O. It's compiled two ways: to WebAssembly for the browser dashboard, and to a native binary for the trajectory CLI that runs in CI. Same bytes of logic, two delivery vehicles. The browser and the build server are literally running the same compiled function.

An earlier TypeScript scorer still exists, but only as a conformance oracle: a parity check runs every trajectory through TS, native, and WASM and asserts all three agree (currently 502 of 502 test vectors identical) before the TS version is allowed to retire. The boundary types the React layer consumes are generated from the Rust structs, so the UI can't drift from the core either. The invariant is mechanical, not a matter of discipline.

The gate blocks on shape changes, not score swings

The obvious way to gate CI on a score is to block when the number drops. That's a trap. Agent runs are non-deterministic; the same task scores 88 one run and 84 the next for reasons that have nothing to do with quality. Gate on a wobbling number and you'll either block constantly or tune the threshold so loose it never fires.

So the gate is categorical, not numeric. It hard-blocks on exactly one thing: a new high-severity finding mode (a collision, or a hallucinated merge) that appears in the candidate run and was absent from every run in a baseline window of recent runs on the base branch. A genuinely new structural failure, not a few points of noise. The numeric health is warn-only by default; blocking on a health regression is an opt-in stricter rule.

The part where I can't ship it

Everything above is built and working. The web tools score, the CLI gates, the three implementations agree to the vector. And the npm package is marked private, because none of that matters if the score doesn't actually predict whether a run succeeds. A scorer that gives confident grades uncorrelated with reality is worse than no scorer: it launders a guess into a number. So shipping is gated on validation, and validation is where the honesty lives.

The first test I could run, I ran. There's a public dataset (nebius/SWE-agent-trajectories on HuggingFace) of real SWE-agent runs on SWE-bench, each labeled with whether its patch actually resolved the task. I adapted roughly 3,100 sampled runs into the trajectory schema and scored them with the same core, then measured each metric by AUC: the probability that a resolved run scores above an unresolved one, where 0.5 is a coin flip and means no signal.

These are spike numbers from a sampled subset, not a proof, but they were enough to act on. A train/test grid search re-weighted the health formula toward the signals that held (focus-led, validation close behind), lifting composite health AUC from about 0.55 to about 0.63, with the Rust and TS cores kept in lockstep the whole way. That's the decision behind the weights you saw earlier.

The natural fix is to find a multi-agent dataset. There isn't one. A deep multi-source search came up empty on the specific combination I need: multi-agent, and per-agent file-targeted tool calls, and a per-run success label, and public. The closest datasets are failure-only or store traces as opaque prose with no structured file targets. So the only path is to generate labeled multi-agent traces where I control the schema: a parallel-worker-with-merge swarm, the one topology that can actually produce a collision.

A cold-read review caught me conflating two different questions there, so I split it into two experiments. And the distinction is the most important thing in this whole post.

Experiment A asks only: does the emitted trajectory faithfully represent what the agents did, and do the scorer's findings match an independent ground truth: the actual git diffs and whether the merge applied and built? No outcome label needed. This one is built. On a faithful corpus, collision, hallucinated-merge, and blind-write detection all hit precision and recall of 1.0. And critically, I ran a teeth check: a deliberately dishonest corpus where agents lie about what they touched (one reports a clean, partitioned set of writes while secretly editing a shared file, another claims a write it never made) collapses collision fidelity to zero. A fidelity metric that can't fail proves nothing; this one can fail, which is what makes its passing meaningful.

Experiment B asks the question that actually matters: does the presence of a collision predict a worse outcome, beyond just being a proxy for "the task was decomposed badly in the first place"? That needs a control arm (partitioned versus overlapping decompositions) and real statistical power. It is not done. Until it holds, the gate's headline feature is a plausible hypothesis with a working detector, not a validated predictor.

What holds, and what doesn't

What holds: length is measurably the wrong thing to reward. On real data it's not just unhelpful but mildly inverted, and the structural signals (churn, validation) carry what little predictive power length seemed to have. Structure is scorable in a way that points at specific steps. And because there's one core compiled two ways, the grade on the dashboard and the verdict in CI can't drift apart. That invariant is mechanical and done.

What doesn't hold yet: the two failure modes the gate is built around (collision and hallucinated merge) are unproven as predictors, because no public data could test them and the experiment that can is only half-run. So the package stays private. The last post's discipline was to not add a building block until you can name the failure it prevents. The same rule applies one level up: don't ship a tool that grades agent runs until you can show the grade predicts the failure it claims to catch. I can name the failure. I can detect it faithfully. I can't yet prove the detection predicts anything. And that gap, not the build, is the real status of this project.