Stop Pointing Agents at the Wrong Work
An engineering leader I spoke with yesterday has caps on how much his team can spend running coding agents. Some of his engineers hit the month's allotment by day three.
The easy explanations are that the cap is too low, or that the heaviest users are the ones getting the most done. The spread inside a single team argues against both. Same budget, same tools, and one engineer ships a week of work while another has a large bill and a branch nobody wants to review.
What separates them is which problems they pointed the agent at.
An agent needs a check it can run without you
Agents work in a loop. Propose a change, run something, look at the result, adjust. That is different from autocomplete or a chat window, where you see every step. Here you hand over a task and the agent takes several turns before you see any of it, and the value comes from that loop running many times with you not in it.
That only happens when the problem has a check the agent can execute on its own. A failing test. A compiler. A reproduction case. A script whose output you can compare against a known good file. The agent tries, the check fails, it tries again, and the spend is buying convergence.
When there is no runnable check, the loop is open at one end. The agent produces something that looks finished, stops, and hands the entire verification job to you. The work did not get cheaper. It moved from your keyboard to your review queue, and you still sign off on what ships.
A bug with a reproduction is close to ideal agent work. “Clean up the reporting module” is close to the worst. Both take about a minute to describe. The economics diverge the moment the run starts.
The run is decided before it starts
Three things worth settling before you kick one off:
- Can the agent tell whether it is getting closer? Say the check out loud first. If the honest answer is that you will read the diff and decide, then you are the check, and the work can only move as fast as you can read.
- Is the decision already made? Agents execute a decision well and make one badly. If you are still choosing between two approaches, the session turns into arbitrating confident-sounding options instead of finishing anything.
- Does the briefing cost more than the work? Agents run on the context you hand them. Twenty minutes explaining the constraints around a ten minute change is a bad trade, and it is the one engineers misjudge most often.
|
|
Where can AI save you time?
My friends at Big Creek Growth put together a quick survey to spot the repetitive work you can hand off to automation.
|
|
Where the money goes without anything coming back
Certain shapes of work consume tokens steadily and return branches you throw away:
- Exploratory work in a codebase you do not know yet. You cannot specify the target and you cannot grade the result, so each iteration is a guess reviewed by a guess.
- Requirements that only get clear once you see something wrong. That is a legitimate way to work, and it means you are the check, running at human speed.
- Changes with no test surface. Config, infrastructure, access rules, data migrations. The failure shows up later and somewhere else.
- Work where “this looks right” is the acceptance criteria. Naming, structure, readability calls. The agent can produce twenty versions and none of them settle the question.
Whether a task suits AI at all is a separate question worth answering first. Even once you have, the shapes above still fail, because the problem never gives the agent anything to push against.
Two corrections is the signal to restart
The rest of the allotment usually goes to correction turns.
The agent takes a wrong turn. You explain what went wrong and let it try again. It gets closer and misses in a new way. You correct again. By the fourth exchange you have spent more than the original attempt, and the work is sitting on an assumption from the first response that nobody has revisited.
Two corrections without real movement usually means the specification was wrong, not the attempt. Discard the branch and restate the problem from the beginning. It feels wasteful because you are throwing away output you paid for. It is cheaper than the fifth correction, which carries the same flawed premise forward and costs more than the fourth, since context grows with every turn.
Caps make the tradeoff visible. They do not create it. A team without caps pays the same bill in review hours, and that one does not show up on a dashboard where anyone can see it.
The number worth knowing is not how fast people burn through the allotment. It is how many tickets on your board arrive with a check that something other than you could run. On most teams that number is lower than it should be, and it is a number you can raise.