Give an agent a bug report and it will produce a diagnosis. The diagnosis will be specific, plausible, well-written, and delivered with complete confidence whether or not it is right.
This is not a flaw you can prompt away with "be careful". It is what happens when a system that pattern-matches on text is handed a text description of a problem. The description looks like a hundred bugs it has seen; it produces the most likely cause for that shape of description.
Sometimes that is correct. When it is not, you have spent an hour implementing a fix for a bug you do not have.
The rule#
No diagnosis until there is a failing test or a reproduced failure you watched run.
Not "I believe this reproduces it." An actual command, actual output, actual failure, in front of you.
How to ask for it#
The wording matters less than the sequencing, but this shape works:
There is a bug: [symptom, with the exact input and the exact wrong output].
Do not propose a cause yet. First, write the smallest test that reproduces this and run it. Show me the failing output. If it does not fail, tell me it does not fail and stop — do not adjust the test until it fails.
That last clause is important and easy to leave out. Without it, a test that unexpectedly passes gets quietly modified until it fails, which produces a reproduction of a different bug.
Why this works#
Three separate things happen when you force reproduction first.
It converts a text problem into a system problem. The agent now has to interact with the actual code, the actual data, the actual failure — not the description of them.
It surfaces wrong assumptions immediately. If the agent's mental model of the code is wrong, the test will not reproduce, and you find that out in one cheap turn instead of after a patch.
It leaves you with a regression test. You needed one anyway. Getting it before the fix rather than after means it definitely tests the right thing, because you watched it fail for the right reason.
The follow-up that catches the rest#
Once you have a reproduction, ask for the cause — and ask for the evidence separately from the conclusion:
Now find the cause. For each step of your reasoning, quote the line of code or the log output that supports it. If a step is an inference rather than something you read, say so.
Requiring citations does not make a model honest. What it does is make unsupported steps visible to you, because a step with no quotable evidence has nothing under it. You will see the gap.
When the reproduction is expensive#
Some bugs do not reproduce in a unit test — race conditions, production-only data, a third-party API behaving badly.
The rule still holds; the bar just moves. Acceptable substitutes:
- A log line from the actual failure, quoted exactly
- A stack trace with the real line numbers
- A query run against a copy of the real data, with output
- An instrumented build that prints the state at the moment of failure
What is not acceptable is a theory with nothing under it. If you genuinely cannot get evidence, that is worth knowing explicitly — it means the next step is instrumentation, not a patch.
The failure mode this prevents#
The expensive version of this goes:
- Agent proposes cause A.
- You accept it, agent implements the fix.
- Symptom persists.
- Agent proposes cause B, confidently, and notes that A was "also worth fixing".
- Repeat.
Each round adds a change to the codebase that addressed nothing. After four rounds you have a diff full of unrelated defensive code and the original bug.
Reproduction first turns that loop into: reproduce, fix, watch the test go green, done.
Next: permissions — what to let an agent run without asking.