Most Things You Could Build Aren't Worth Building

I recently talked myself out of building an AI agent.

Bug work was consuming roughly an eighth of our engineering capacity. Support escalations reached engineering, someone investigated, and a fix eventually shipped. An agent that could read the report, find the cause, write a patch, test it, and open a pull request sounded useful.

I started by testing the cheapest part of that idea: could it diagnose the bug?

Start with permissions

We serve regulated customers, including banks, healthcare organizations, and customers in the EU. The agent would have no production access, no raw customer data, and no ability to merge code.

It could use approved tickets, logs, safe reproductions, source code, and internal discussion. I connected Jira, Zendesk, and Slack through MCP, then built a root-cause analysis skill around that evidence.

The skill classified each issue as a defect, expected behavior, a product gap, or insufficient information. A diagnosis had to identify the relevant code path or behavior and cite its evidence. The agent updated the ticket; a second pass labeled candidates that might be suitable for a fix.

Test diagnosis first

I let engineers continue their normal process. When they shipped a fix, I compared it with the agent’s earlier diagnosis. A plausible explanation earned no credit unless it matched the cause addressed by the actual fix.

In the sample I reviewed, the agent was plausibly wrong about half the time.

It did well on problems with direct evidence. In one case, a PHP input-variable limit was dropping a form field; moving the payload into a JSON body addressed the cause. In another, an email template referred to a token missing from its substitution map.

The misses involved runtime behavior, data moving between components, bundling, or similar-looking systems with different behavior. Our enterprise platform has customer-specific feature combinations and customizations. A convincing reading of one code path often missed what happened when those pieces ran together.

Plausible is expensive

A confident wrong diagnosis costs an engineer time. They must follow it far enough to discover the error, then start again. Generating the explanation was cheap; reviewing it was not.

My estimate was that a fuller agent might solve 10–20% of these issues. Applied to a bug load of roughly 12% of engineering capacity, that offered about 1–2% of total capacity before review and maintenance. For a small team, those costs could consume the savings.

Better runtime evidence might improve the result. But collecting it would require starting the application, seeding data, reproducing failures, and maintaining that infrastructure. Someone would also have to own the prompts, connectors, model changes, and review process.

Count the work around the agent

Before adding code generation, I would need rules for eligible bugs, required tests, acceptable evidence, and when the agent must stop. I would need a reliable way to reproduce the problem and prove that a patch fixed it without introducing another one.

Those requirements are part of the product’s cost. A demo that ends at an open pull request leaves most of them unpaid.

Several bugs in the sample were regressions. Better prevention and regression tests might return more engineering time than an agent built to repair them later.

Ship the smaller version

The useful version was smaller: triage, evidence gathering, a proposed diagnosis, and candidate labels. Stop before generating code.

I can measure diagnosis accuracy by bug type, the precision of the candidate labels, and whether engineers find the output useful enough to save time. If those results justify it, a narrow class of bugs may eventually earn a path to automated fixes.

Make the agent prove itself at the cheapest useful layer. Diagnose before fixing. Compare against shipped fixes. Price review as work.

When a workflow does justify tool use, my MCP evaluation case study covers the next test: whether an agent selects the right operation and respects the boundaries around it.