Measuring LLM features before trusting them

Context

Much of my 2026 went into LLM features that write to real systems: an agent that builds workflow graphs from a chat request, AI steps inside those workflows that decide which CRM records a run should touch, and an editing agent on Frame. Each one had the same problem on day one. It looked great on the five requests I tried by hand, and I had no idea how it did on the fiftieth.

This page is about the evaluation work behind those features: what I measured, how, and what the measurements caught that hand-testing never would have. The story of the workflow builder itself, including the call to delete it, is in I built an AI workflow builder — then deleted it.

The constraint

Three things made “just write some tests” not enough.

• The output is a structure, not prose. A workflow graph is right or wrong in specific places: the trigger, the node types, the fields each node sets, the edges between them. A judge model asked “is this a good workflow?” will happily say yes to a graph that writes to the wrong field.
• The model rephrases everything. Node ids, labels, message wording and filter encodings vary from run to run on the same request. A scorer that compares strings fails correct output constantly, and then nobody trusts the scorer.
• The agent talks back. A good builder asks a clarifying question when a request is ambiguous. A single-turn test can’t tell a sensible question from a stall.

And every eval run is a set of real, billable model calls, so the suite had to stay cheap enough to run on every prompt or tool change.

Options I considered

Hand-checked examples, LLM judges, or deterministic scorers. Hand checks don’t scale and drift with whoever is checking. An LLM judge is fast to write and good at grading prose, but it is a second model with its own failure modes, and it hides why something failed. Deterministic scorers take longer to write and only work where the output has structure — which, for a workflow graph, is almost everywhere. I chose deterministic scorers for everything structural and kept an LLM judge for the one place the output really is prose.

A third-party harness or the team’s own framework. I started with a Braintrust remote eval, because it was the fastest way to get a baseline. Once the builder’s evals needed seeded fixture data, multi-turn conversations and per-dimension results the team could browse, I moved them into the in-house eval framework the team already ran other agents through, and extended it.

Replay old traffic or write cases. Replaying real conversations means real customer data in an eval set. Written cases against a seeded fixture tenant don’t, and they can be designed to cover gaps on purpose.

The call

Make the agent drivable, and watch what it does

The first change wasn’t a scorer. It was an eval-only endpoint that drives one builder turn synchronously, skipping the queue and the real-time channel the product used, and a lightweight observer that records which backend lookups the agent made along the way: entity search, the org’s Salesforce schema, workflow run history. In production no observer is attached, so it costs nothing.

That observer turned out to matter as much as the output. “Did it check the Salesforce schema before writing a Salesforce update?” is a question about behavior, and the final graph can’t answer it.

Deterministic scorers that tolerate rephrasing

The core scorer grades each result on separate dimensions, and stores a pass/fail and a reason for each one, so a failure says where it failed:

• Status. Did the turn end where it should: a proposed workflow, a clarifying question, or an applied change?
• Node types. A multiset compare: order doesn’t matter, counts do.
• Workflow structure. Compares the proposed graph against the expected one with deliberate tolerance:
  • node and edge ids can be renamed freely, and the graph is compared by topology;
  • cosmetic fields (icon, label, description, position) are ignored;
  • prose fields like a Slack message or an AI filter prompt are presence-only: they must exist, but the wording is the model’s;
  • recipients compare by type, so a variable that resolves to a real Slack id at runtime still passes;
  • everything load-bearing (the fields a node actually acts on, and which node feeds which) is strict.
• Lookups. Did the agent make the backend calls the case expects?

The tolerance list is the real design work. Too strict and the suite fails correct output; too loose and it passes broken graphs. Every rule on that list came from a specific false failure or a specific miss.

score(case, result):
  status       = exact(result.status, case.expectedStatus)
  node_types   = multiset(result.nodes.types, case.expected.types)
  structure    = compareGraph(result.graph, case.expected,
                   ignore=[ids, icon, label, description, position],
                   presenceOnly=[message, aiFilterPrompt, agentId],
                   strict=[loadBearingFields, edgeTopology])
  lookups      = multiset(observer.callbacks, case.expectedCallbacks)
  pass         = all dimensions pass

A simulated user for clarifying questions

Cases can declare how ambiguous they are, with hints about what the user really meant. When the builder asks a question on one of those cases, a separate model plays the user, answering only from the hints, and the conversation continues until the builder proposes a workflow or hits a four-turn cap. Unambiguous cases run exactly as before.

That one change turned “the agent asked a question” from an automatic failure into something graded: did it ask a good question, take the answer, and get to the right workflow? And a simulator failure is recorded as a failed result, not a crashed run, so one flaky turn can’t take the whole suite down.

Grade the effect, not the claim

When the builder learned to edit existing agents instead of creating duplicates, the obvious test was to check its reply: did it say it updated the agent? That’s the wrong question. The scorer gained two checks: that the agent made the right decision (update, select, create, or ask first), and that the live agent was actually re-versioned afterwards. An agent saying it did something and an agent doing it are different results, and only one of them matters to a customer.

Coverage that follows the product

Cases are tagged by category and mode, and every run reports pass rates per category, so a change that helps new workflows and quietly breaks edits shows up as exactly that. I filled coverage gaps on purpose: edits to complex workflows, multi-turn disambiguation, error paths, preserving what the user didn’t ask to change, and cascading effects. Every new node type the builder learned (creating Salesforce records, sending and drafting email) shipped with eval cases in the same stretch, and a new AI step that picks CRM records got its own eval type with its own deterministic scorer: the right record ids, and never more than the limit.

The one LLM judge in the system grades the agent’s natural-language answers about workflow run history, against a reference answer. It only runs on cases that have one, so it can’t dilute the structural score on build cases.

Compare runs, not numbers

A single eval run is a sample, not a measurement: the same code on the same cases can score differently twice in a row. So the question is never “what’s the score?” but “what changed between these two runs?” I built a side-by-side compare view that aligns two runs case by case, classifies each row as regressed, improved, unchanged, errored or only in one run, diffs the dimensions, and shows per-category deltas. The diff logic is plain, unit-tested functions, so the view can’t disagree with itself.

Beyond the builder

Shadow-testing the replacement for an LLM

Some of our workflows used an LLM filter to decide whether a meeting was worth acting on. A deterministic signal built from attendance data could replace it: cheaper, faster and explainable. But “probably equivalent” isn’t a migration plan, so I built the evaluation first.

• Backtest what you can, shadow what you can’t. One condition could be scored against historical runs. The other depended on attendance evidence the system had only just started recording, so a historical replay would have answered “unknown” for almost every meeting. That one had to be computed forward, alongside the live LLM filter, on new traffic.
• Unknown isn’t a violation. The comparison is three-way. Only the case where the LLM passed a meeting and the new condition definitively rejected it counts against the migration. “Unknown” means the fallback did its job: it neither breaks the bar nor proves it holds, so it gets its own counter.
• Count witnesses, not just errors. A zero error rate over meetings the new condition never actually decided proves nothing. The gate requires a minimum number of definitive answers before it will clear a migration.
• Re-runs should only add evidence. A rolling window drops as much old traffic as it adds; an anchored window means every re-run covers strictly more. The report says which mode it ran in, and what would raise a short witness count.

Two corrections came out of the first production run, and both are in the tool now. One condition was being scored with a function that would never ship, so its number described a different predicate; I removed it from the tool and retracted it rather than relabeling it. And one class of meeting carried a stale flag that would have counted as a pass; those are excluded until the data is cleaned.

Comparing harnesses before building one

When a spec for Frame proposed running a sandboxed coding agent to edit HTML documents, I treated it as an eval question before it became an architecture. I ran the plain tool loop we already used and two sandboxed coding agents against the same set of editing requests on sample documents, with an independent render check on every candidate. The plain loop matched them on correctness and was much faster, without needing a VM per turn. The coding agents also taught me something a single run wouldn’t have: one wrote an accurate page and an inaccurate summary of it, twice. So the preview has to be the rendered result, not the agent’s description of it.

Live tests, kept out of CI

For the multi-provider LLM layer, deterministic tests drive the whole chain (model config, provider dispatch, the logged provider) on mocks in CI. Separate live tests, behind a build tag and never in CI, make real calls to each provider: a text turn and a full tool round-trip. Mocks can’t tell you a provider’s streaming format changed; a short manual run can.

Red-team evals on this site

The “Ask about Kevin” assistant on this site has its own eval suite, and it’s public. Each case is deliberately simple and deterministic: a prompt, and patterns the answer must or must not contain. Categories cover prompt injection, role-play, salary fishing, off-topic requests, defamation bait, ungrounded claims, multi-turn escalation, and the job-fit report’s own adversarial inputs. One check fails any answer that mentions a term from a private denylist that never enters the model’s prompt. A mock upstream runs the plumbing for free; real-model runs refuse to target anything but a local server unless told otherwise.

What broke

The auditor hallucinated, and the evals caught it. The builder had a small, cheap model auditing its proposals for conditions the user asked for but the graph lacked. One eval case kept failing with an extra record-lookup step nobody asked for. The cause: the auditor drifted to the stock example in its own system prompt and reported it as a missing condition, and the repair loop dutifully told the builder to add it. A prompt fix alone wasn’t enough for a small model, so the code now drops any “missing condition” that shares no meaningful words with the user’s request, and records when it does so the drift is visible. A model checking a model needs checking too.

The first baseline was humbling, usefully. Every output was structurally valid, and far fewer were right. The model had learned the shape of a workflow before it learned the meaning. Without the eval, the valid-looking graphs would have been the ones we shipped.

Scores moved with no code change. Running the same baseline twice gave different numbers. From then on, every change was compared against a fresh baseline run, never a remembered score.

A fix cost a regression. The first eval-driven prompt fix clearly improved the main scores and made one case skip a lookup it used to make. The eval made that a decision instead of a surprise.

Result

• Prompt and tool changes to the builder were gated on the eval suite, including a staged migration to native tool calling where every step had a before-and-after run.
• The scorers, the user simulator, fixture seeding, category breakdowns and run comparison became part of the team’s shared eval framework, and a second AI step used the same pattern for its own evals.
• An LLM filter got a measured migration path with an explicit bar, instead of a judgment call.
• An architecture question for Frame was answered with a comparison on a fixed test set, before anyone built a sandbox.
• The evals outlived the feature they were built for: when the in-app builder was retired, the framework stayed.

What I’d do differently

Build the observer and the compare view first. I added them early, but not first. Knowing what the agent did and what changed between runs is worth more than any single score, and every later piece leans on both.

Write the tolerance rules down as rules. The structural scorer’s tolerances were right, but they lived in code and in my head. A short document saying what is strict, what is presence-only and why would have made new cases faster to write and easier to review.

Budget for the judge’s evals. The hallucinating auditor was found by luck of coverage. Any model that grades or gates another model deserves a few cases of its own.

The views here are my own, not those of any employer, past or present.