I’ve spent, pretty much, most of the past six months letting AI write my tests. Not experimenting with it, but actually relying on it, day in, day out, while building GTMS (a test management CLI whose own test suite is wholly AI-written). The following five techniques have survived six months of daily contact with reality.
They’re all cheap. None of them needs a bigger model. You can use every one of them today, with whatever AI tooling you’re using. The story of why I work this way comes after, but the recommendations come first so that you can put them into practice now.
1. Write down what should happen before the AI writes the code, and hand it over as untouchable
Don’t let the model derive expected behaviour from the code it’s testing. The danger isn’t the AI seeing the code. The danger is the expectations coming from the code. If the expectations come from the code then the tests can only ever agree with whatever was built. Write the intent down before the code exists, so the agent has to derive the acceptance tests from the requirement (a PRD, a story, a ticket). Even three bullet points of “what correct looks like” changes the game. But the wording of the hand-over matters.
Say this explicitly when you ask AI to write tests based on your acceptance criteria.
These criteria are immutable. Do not edit or reinterpret them to make a failing test pass.
One warning: whoever writes the requirement is really the one designing the tests. The AI will implement your intent with impressive fidelity. And that fidelity cuts both ways. Your good decisions propagate, and so do your blind spots. The AI won’t fill the gaps you left. Its job is to implement what you wrote, not to question it. Review the acceptance criteria or test case document with the same seriousness as the code, because everything in it propagates.
In short, whoever reviews the ACs or test case specs is doing the highest-leverage review in the pipeline. This is worth a human’s time precisely because the AI will treat what’s written there as beyond question. And that’s the whole point. We want something immutable early in the process, to keep every agent further down the pipeline on track.
2. Add one sentence of permission to every test-writing prompt
I paste this, verbatim, into every prompt where I’m asking an agent to write tests:
If anything prevents you from writing a test that verifies the requirement as stated (for example, the requirement appears ambiguous, untestable, or inconsistent with what the product actually does), do not silently adapt the test to fit. State the problem in a comment block at the top of the test file, and still write the strongest test you can for the requirement as written.
What I’ve found is that the failure mode isn’t that models want to paper over problems. It’s that the slightest indication that they need to “write a passing test” gives them no legitimate way to report a problem when they find one. Give them that way out and, in my experience, they take it. What you get back instead of a quietly twisted test is a written diagnosis of the actual problem. Plus the best ‘honest’ test case you can hope for.
That’s pretty valuable, for one sentence of text.
3. Never say “make the tests pass”
“Fix the failing tests” can be misinterpreted as telling the model that passing tests is the goal. A correct test running against a buggy product should fail. You might know that but does your AI agent know that? Make sure it does. When a test run fails, I ask for a verdict instead:
For each failing test, decide: is the test wrong (revise it, and record why), or is the product wrong (leave it failing and tell me).
The correctly failing test leads you to a bug report. The failing test the model “fixes” is a bug buried.
It’s obvious to you and me. Is it obvious to your agent?
4. Prove your tests can fail: break your product once, on purpose
A suite that has only ever passed has proven nothing. Take ten minutes and try this:
‘Revert one recent bug fix on a branch (your git history is a free supply of realistic bugs), run the AI-written suite against it, and see whether anything fails.’
It’s worth doing this because reviewing tests is tedious (and it doesn’t always work). I once had an AI generate a suite of acceptance tests that were genuinely good, with careful stream handling, exact assertions, and no lazy matchers. Two of the seven were broken from birth. They hardcoded an identifier the product rejects by design, so each test aborted during setup and its real assertions never executed.
I might have found this by being more diligent with my reviews. But let’s face it, none of us are 100% diligent with our reviews.
Getting an LLM to break a working product can be an easier way to “test your tests”.
5. On failure paths, make the AI assert the exit code, not just the message
Negative tests are just as important but sometimes harder to write. One bug I’ve dealt with printed a perfect, detailed error message… while exiting 0. Every assertion on the text of the error passed happily. Only an exit-code assertion could tell working from broken apart.
Your scripts, your CI, and increasingly your other AI agents never read your error prose; they branch on codes and structured output. So this goes in the prompt too:
Every test of a failure case must assert the exit code (or status field), not only the message.
It’s one line, and it targets exactly the class of bug that’s invisible to a human at the terminal and fatal to automation.
The hypothesis: separate test case specs, written up front
Behind these five techniques sits one idea, an assumption I’d baked into my way of working long before I thought to prove it. That assumption was:
Writing separate, human-readable test case specifications up front is beneficial when you’re using AI coding agents.
When AI writes your automated tests, there are two forks you can take:
Fork (b) is the default. Most teams already write one-line acceptance criteria, and the agent goes straight from requirement to automation. Fork (a) adds a deliberate middle step: a detailed test case document, written before the code, reviewed by a human (sometimes), and handed over as immutable.
Does fork (a)’s extra artefact earn its keep? A silent assumption on my part wasn’t good enough, so I built an evaluation system to find out:
Real bugs planted back into a real product, and a measurement of which fork’s tests actually notice.
I’ll come back to the actual evaluation system at the end of this post. And the next blog post will cover the evaluation in full.
First, why did I come to this assumption that separate test case specs would be best?
Why I believed it
The reason for my hypothesis is far older than AI. Think about what a test case actually was when a good tester wrote one. It was never just a restatement of the requirement. It was an information collection point. Into it went the requirement, yes, but also the tester’s experience, their history of finding defects in the product, even data from support tickets they’ve dealt with. Half of those inputs aren’t in the repo. From the perspective of an AI agent generating test cases, no amount of reading the requirement and the code gets anywhere near this level of information.
Yet AI now makes it super easy to generate great-looking test case specifications from just the requirements and, where it’s available, the code.
And therein lies the problem with letting the agent author its own test cases and acceptance criteria. An agent that creates a feature and then writes the test from the code it just wrote isn’t testing anything. It’s describing itself. The principle I keep coming back to:
An executable specification (e.g. test automation script) that can be edited by the thing it’s meant to constrain isn’t a good test case specification. It’s a mirror.
I should be clear here. A year ago I would see AI agents weakening tests to get a pass. More recently, with better models (or maybe even my way of working), I haven’t actually caught an AI weakening an assertion to get a pass.
What I still see though is the slope that leads there. A genuinely useful independent review that calmly proposed narrowing a failing test’s assertion as one of two “equivalent” fixes. Something that would have made the test pass by deleting the very claim it existed to check. In this particular scenario the right fix was completing the test’s setup instead. Both options looked equally sensible in the review comment. That’s the slope. Techniques 1 to 3, at the start, exist because of it.
The test that convinced me
The strongest lesson didn’t come from an AI at all. It came from things going wrong!
One acceptance test in the GTMS product suite existed to catch one specific product failure. If a file path contains spaces, commands built from that path must still treat it as a single path. Keep that in mind. This test had one job! Deal with paths that have spaces.
At some point, that test started failing on Windows. Nothing to do with paths or spaces. It was an environment problem in the test’s own setup. The problem was real. The diagnosis was reasonable. The fix was reasonable too.
But the fix was applied to the test itself. And while it cured the environment problem, it also stripped out the paths-with-spaces check. The test now ran cleanly everywhere. It just no longer tested the one thing it existed to test.
A few weeks later I shipped the product with the exact product bug that test was written to catch. The test never made a sound. Finding and fixing the bug cost me a full day.
Nothing malicious happened. Nobody was careless. That’s what makes it the important story. Coverage doesn’t get deleted by sabotage, it gets deleted by reasonableness: a sensible person (or a sensible agent) fixing the visible problem and quietly dropping the invisible one.
And the only reason I ever found out? The test’s stated objective, a sentence sitting outside the script, still said exactly what the test existed to prove. The script no longer proved it. Nothing inside the script, and nothing in weeks of passing runs, would ever have told me that. The mismatch between the “Test Case Spec” and the “Test Script” was the whole discovery.
A “Test Case Spec” sits as an intent document outside the requirements and the code. It states what the test exists to prove. This is where my experience had led me to fall back to the traditional “Test Case Spec” in the first place.
The immutable “Test Case Spec” can act as the alarm that tells you when your “Test Script” no longer matches the original test’s intent.
Where the edges are
A slight caveat first though. This way of working, with “intent-driven test cases”, provides additional integrity, not insight. They make it much harder for a test to quietly stop doing something we need it to do.
However, if your test case (intent document) has a blind spot, everything downstream inherits it faithfully. The “old school” test case specification is where a lot of the thinking happens and details need to be captured. It’s worth having and it’s also worth a good review.
I’m starting to think that we should be resurrecting and raising the importance of the “old school” test case specification in the age of AI. And it’s more than just capturing the test case intent up front.
As I mentioned earlier, the traditional test case spec is where the good tester’s knowledge and experience gets captured. I’m now thinking that there are some interesting tool integrations worth exploring that will help mimic the good tester’s way of working. Additional data feeds into the test case specifications. Defect history and support tickets flowing into test design. Data flowing the way it always flowed through a good tester’s head. But that’s a future post.
And because you shouldn’t take my word for it
My way of working with AI coding/testing is having independent, standalone test case specifications. It seems to be working well for me. “Seems to” implies it’s no more than a vague hypothesis. I’m human, and as a human it’s always nice to be right. Which is why I want my hypothesis and my way of working to be the right way. Everyone coding with AI agents should have standalone, independent test case specifications. Right?
Well, possibly not!
I figured it would be better to start proving my hypothesis with some real-life empirical data. So I built an experiment designed to prove or refute my own way of working. I took real bugs GTMS used to have, planted them back into the product one at a time, and measured (properly with pre-registered bugs, scored claim by claim) which AI-written tests actually noticed. Evaluations run with test case specs, without them, with the AI reading the code, and completely blind.
Some of what I believed survived. Some of it didn’t. The second part of this blog post covers what the evidence actually demonstrated. For now though, this blog post focuses on the idea and reasoning behind my thinking. It also states my recommendations that can be applied practically now.
The recommendations I set out at the start of this post came from this evaluation’s empirical data. In part 2 I’ll explain why I stand by them.