I built an AI workflow builder — then deleted it

Context

Clearskies Workflows is the automation engine I spent most of 2026 building: a canvas where you wire a trigger (a call ends, a schedule fires, a CRM record changes) to steps that find records, run an agent, post to Slack, send email or write back to Salesforce. It’s powerful, which is another way of saying it has a lot of knobs. To build a good workflow you need to know which trigger fits, which node types exist, what variables each step exposes, and how your own Salesforce org is shaped.

So in March 2026 I started on the obvious feature: an AI Workflow Builder. You describe what you want in chat, in the app or from Slack, and an LLM agent generates the workflow graph or edits the one you have open. The first version streamed into an in-app chat panel over PubNub.

Between March and June it took about 84 PRs. Then I deleted it: 28.5k lines on the backend and another 1.8k on the frontend. This is about why building it carefully and then deleting it were both the right call, and how the first made the second obvious.

The constraint

A workflow graph isn’t prose. If an LLM writes a slightly-off paragraph, a person reads past it. If it writes a slightly-off workflow, the workflow fails to publish, or it publishes and does the wrong thing to someone’s CRM. “Looks plausible” was never the bar. The bar was: the graph is structurally valid, it does what the user asked (all of it), and it only uses fields that exist in their Salesforce org.

The second constraint was ownership. I was building the builder alongside the engine it targeted, so anything that let the builder drift from the product’s real code paths would cost me twice.

Options I considered

There were two layers of decisions, and they came at different times.

The first was how to make the agent reliable. I could keep growing the system prompt, which was already about 500 lines inline in Go. I could check the agent’s output after the fact and make it fix its own mistakes. Or I could start by measuring, so I’d know which of those was working. I did all three, and measuring came first.

The second was how the agent should act. The prototype used JSON pseudo-tools: the model wrote out a JSON blob describing what it wanted to do, and we parsed it. The alternative was native tool calling, with real schemas, the provider enforcing them, and the tool implementations being the same functions our MCP server already exposes to external clients.

Once that migration was done, a third question came up that I hadn’t planned for: should there be an in-app builder at all? Clearskies is sold as the context layer that Claude, ChatGPT and other assistants query over MCP. If the builder’s tools were the MCP server’s tools, our chat panel was just one more client, and a weaker one than the assistants our customers already use every day.

The call

Measure first

Before most of the reliability work, I set up a Braintrust remote eval: a small set of seed requests and deterministic scorers that are cheap to check and hard to argue with. Is the output schema-valid? Did the turn end in the right status (a proposed workflow, or a clarifying question when one was warranted)? Did it pick the right trigger and the right node types? Did it make the lookups it should have, like checking the org’s Salesforce schema before writing to it?

The first baseline was sobering in a useful way. Every output was schema-valid, but far fewer ended in the right status or used the right node types. The model had learned the shape of a workflow perfectly and the meaning much less reliably.

Then I ran the same baseline again, same code and same cases, and the scores moved. That’s the first thing evals teach you about LLMs: a single run is a sample, not a measurement. From then on every change was compared against a fresh baseline run, never a remembered number.

The first fix the eval pointed at was small. When a call-triggered workflow named a Slack channel the builder couldn’t resolve, the agent politely asked the user what to do and stayed in conversation instead of proposing anything. I changed the prompt so call-triggered workflows fall back to the account’s Slack channel (or the host’s DMs), say so in the reply, and propose.

The fix clearly improved status, trigger and node-type matching, and it cost one regression: on one case the agent now jumped straight to the fallback and skipped a lookup it used to make. The net was clearly worth it, but the eval is what made that a decision rather than a surprise. From then on, prompt and tool changes were gated on the eval, not on how a few hand-tried requests felt.

Then validate, repair and audit

Next I put guardrails around each proposal:

• Validate and self-repair. Every proposal runs through the same validation as publishing a workflow. If it fails, the errors go back to the model and it gets up to two more tries.
• A semantic auditor. A small, cheap model (gpt-5-nano) checks that every condition the user named made it into the graph. Validation catches graphs that are broken; the auditor catches graphs that are valid but incomplete.
• Org-schema injection. The agent gets the customer’s actual Salesforce schema, so it builds against real objects and fields instead of guessing at them.
• A testable prompt. In the same change as the auditor, the ~500-line inline prompt became a Go text/template with golden-fixture tests, so a prompt edit became a reviewable diff with tests.

The repair loop is simple enough to sketch:

proposal = agent.propose(request, orgSchema)
for attempt in 1..2:
    errors = publishValidate(proposal)
    if errors is empty: break
    proposal = agent.repair(proposal, errors)
audit(proposal, request)   # every condition the user named is present

Then move to the MCP server’s tools, in stages

The migration off pseudo-tools went in steps, each gated on the eval: first a registry of builder tools, then native tool calling, then OpenAI’s strict mode for the tool schemas. Last, I promoted the MCP server’s tool handlers to plain package-level functions so the builder called exactly the code that external MCP clients call. Doing it in stages meant every step had a before-and-after score, and a regression pointed at one change instead of four.

That last step is what changed my mind about the whole feature. Once the builder’s tools were literally the MCP server’s tools, the in-app builder was a chat UI plus a system prompt wrapped around a capability our customers could already reach from Claude or ChatGPT, assistants they already have open all day.

Then delete it

So the new-workflow flow stopped opening our chat and started pointing people to install the Clearskies plugin in Claude or ChatGPT. The plugin is public: scratchpad-ai/clearskies-plugin. Product and engineering made the call together, and the transition was seamless. Then I removed the in-app builder: −1.8k lines on the frontend, −28.5k on the backend.

What broke

Most of the reliability work exists because something specific went wrong, and the evals made it visible.

Valid isn’t correct. The baseline said it plainly: every output schema-valid, far fewer with the right outcome. Structural checks alone would have shipped a builder that produced clean, publishable graphs that often did the wrong thing.

Valid and correct-looking isn’t complete. A graph can pass validation and still quietly drop one of the user’s conditions. Nothing structural catches that, which is why the auditor’s only job is to check the user’s request against the graph.

The prompt didn’t scale. Five hundred lines of inline Go string meant every edit was a guess. Templating it with golden fixtures didn’t make it shorter, but it made changes reviewable.

The plumbing lied in ways mocks couldn’t see. Moving to native tool calling surfaced two bugs that had been hiding behind the pseudo-tools. Tool-call messages existed in our request types but were silently dropped when converted for the provider, so multi-turn tool use had never round-tripped. And our streaming code kept only the first chunk of a tool call’s arguments: fine for test mocks that send arguments in one piece, broken against real streaming, where they arrive in fragments.

Publish validation had a hole. Self-repair is only as good as the validator it repairs against. Generated graphs exposed nodes with no incoming edge slipping through publish validation, so I added connectivity checks there too, which tightened publishing for human-built workflows as well.

Going headless moved the risk, not removed it. Once outside LLMs were calling the MCP server directly, two things mattered more. First, errors: an LLM client needs an error that tells it what to do next, not just a status code, so I designed an error contract for LLM clients that carries guidance. Second, authority: testing coding agents against the server turned up gaps worth closing. Token scopes are now enforced server-side, so a workflow’s AI token can’t call write tools even when it acts for an admin, and MCP tokens are no longer accepted as app logins.

Result

• Building workflows with AI now happens in the customer’s own assistant, through the public Clearskies plugin and the same MCP server everything else uses.
• The in-app builder is gone: about 30k lines of code we no longer maintain.
• The eval work outlived the feature. It grew into an in-house framework with structural scorers, an LLM user-simulator for multi-turn clarifying questions, grading of the agent’s decisions, and a UI for comparing runs side by side.
• The MCP server is the single code path for building workflows, with server-enforced token scopes and an error contract written for LLM clients.

What I’d do differently

Ask “whose chat window?” on day one. I built the in-app chat first because that’s what an AI feature looked like in early 2026. The question that ended the feature, whether our UI is the best place for this, was answerable before I wrote the first PR.

Route tools through MCP from the start. The pseudo-tool phase taught me things, but the version where the builder and external clients share one code path is the one that held up. Starting there would have saved a migration.

Keep the evals, always. I wouldn’t change that part. The eval suite is the reason every step above had a number, and the reason deleting 28.5k lines felt like a decision instead of a gamble.

The views here are my own, not those of any employer, past or present.