Before You Scale AI-Written Testing: Findings From a Real-World Evaluation

Terms used in this document

Requirement: any written input handed to an AI coding agent with the expectation that working code comes back. This includes PRDs, user stories, functional specifications and use cases.

Test case specification: a human-readable description of a test, including its objective, preconditions, steps, test data and expected results.

Test script: executable code that automates a test case, such as Playwright, Selenium, BATS or pytest.

Claim: a single verifiable statement a requirement makes about behaviour. A claim may be checked by one or more test cases, and a test case may be implemented by one or more scripts.

Writer condition: one setup of the AI agent writing the test scripts. The agent and prompt template stay the same; the inputs it receives change.

The uncomfortable question

Someone in your organisation may already be using AI to write tests. They can generate hundreds of scripts in minutes, adjust them until they pass, and add them to CI. The dashboard is green.

How do you know whether those tests are worthwhile?

AI makes it easy to produce large volumes of professional-looking test code. That changes the value of the signals we used to rely on. Test count tells us little about behavioural coverage. Well-structured code tells us little about what is asserted. A passing run proves only that the current product satisfies the tests as written.

The question that matters is:

If the product were broken, would any of these tests notice?

An AI-generated suite is an insurance policy. The appearance of the certificate is not the important part. What matters is whether it pays out when something goes wrong.

Test the tests against a broken build

You cannot measure bug-detection capability without bugs. We used real ones.

A product’s git history contains defects that have already been diagnosed and fixed. Reverting one suitable fix gives you a good build and a bad build of the same product. The bad build has one known defect restored.

Run the same test suite against both builds:

This is related to mutation testing and fault injection, but we used restored historical defects rather than automatically generated code mutations. A test written for a defect should fail where that defect exists and pass where it has been fixed. The paired results provide a behavioural check of the test.

This check is especially useful for AI-written tests because an AI agent doesn’t necessarily possess the background knowledge a human author would develop while writing them. Review by an experienced tester remains valuable, but it is difficult to scale at the volume AI can produce. A paired run against good and bad builds is cheap, repeatable and gives direct evidence.

Not every historical fix makes a useful test defect. We selected bugs with three properties:

1. The fix reverted cleanly and restored only one bug.
2. The behavioural difference between GOOD and BAD could be verified by hand.
3. The bug was quiet. The product still built and appeared healthy, but one detail of its behaviour was wrong.

Roughly half the candidates we probed showed no behavioural difference after the fix was reverted and were discarded. Reverting a commit is not the same as proving that a bug is back. Every candidate was checked manually, and its probe result or exclusion reason was recorded before test generation began.

Quiet defects mattered because crashes are easy to catch. Our restored bugs were closer to regressions that reach production. One left a status view showing a stale result while the command reported success. Another printed the right error message but returned a success exit code.

There is also a simpler version of the check that requires no historical search. When fixing a bug on a branch, the broken build already exists on your main or integration branch. Run the new test against both:

• it should fail on the branch where the bug exists;
• it should pass on the branch containing the fix.

For a new feature, the old build provides a similar signal. If the feature test passes before the feature exists, it may not assert anything the feature provides. In both cases, a new test that passes on the old build deserves investigation.

Turning the check into a merge gate

The cheapest time to validate a new regression test is while the defect and its fix are both still available. A practical branch workflow looks like this:

1. Reproduce the reported bug on the default branch.
2. Write the fix and its new tests on a feature/fix branch or worktree.
3. Run those tests against the feature/fix branch and record the passing result.
4. Run the same tests, with the same data and environment, against the default branch.
5. Require the test associated with the defect to fail for the expected reason on the default branch.

The last qualification matters. A timeout, missing dependency or unrelated assertion also produces red CI, but it is not evidence that the test detected the target behaviour. A small answer key should record which claim should fail, at which assertion, and why.

This can be automated once the product has a repeatable build and test harness. The cost is an additional execution against an older revision, not a second test-authoring exercise. Where full dual-build execution is expensive, apply the gate first to newly added regression tests and high-risk changes.

Historical defects extend the same idea beyond active bug fixes. Select a small defect bank representing the failures your product actually ships, keep reproducible GOOD and BAD revisions or fixtures, and use them to calibrate changes to prompts, models and test-generation policy. The results show what your current system systematically misses and where it needs work before you generate more tests.

What we set out to evaluate

The broken-build check was the instrument, not our original research question.

In our development process, an AI agent typically produces a requirement with one-line acceptance criteria. Before code is written, we expand those criteria into human-readable test case specifications. Those specifications feed both manual testing and the automated scripts an AI writes later.

We had assumed that the specification was where the test-design thinking happened and would therefore improve the automation. We had never measured that assumption.

Our hypothesis was:

Writing separate, human-readable test case specifications before code exists improves the automated tests an AI agent produces from a requirement.

We defined improves as “more likely to catch real defects.” We then asked whether full specifications produce better scripts than one-line acceptance criteria, and under what conditions any benefit appears.

We committed in advance to publishing a null result as prominently as a positive one.

How the evaluation worked

This is the high-level view of the evaluation. It shows where the writer sits, how the test suites are checked, and how the results feed back into test repair.

The four writer conditions

We varied two inputs to the test-writing agent:

1. Test intent: full test case specifications or one-line acceptance criteria.
2. Source access: able to read the product’s source code or blind to it.

Everything else stayed constant:

• the requirement
• the prompt template, except for the slot containing the test intent
• a pinned mid-tier model
• first drafts only, before the writer could run or revise its tests.

This produced four conditions:

Blind writers could run the product but could never read its source. Every condition started in a fresh session with no memory from another cell.

The test case specifications were not written by these script-writing agents. They were authored separately, in advance, then held constant as an input. To keep the claims comparable, the one-line acceptance criteria were made by collapsing each specification’s objective into one line. Both artefacts therefore carried the same high-level claims; the detail differed.

Every test-writing prompt included one safety sentence:

If the requirement and the product’s behaviour disagree, say so in writing at the top of the test file, and keep the test true to the requirement rather than quietly bending it to fit the code.

Source access was deliberately varied. Reading code can help an agent discover how to drive a product, but it can also tempt the agent to treat implementation as the definition of correct behaviour. We allowed the first use and tested for the second.

It might seem obvious that a test writer should never see the code. That is a useful form of independence, but it is not how many teams now work. Tests are often created by the same coding agent, in the same repository and sometimes in the same session as the implementation. In that setting, withholding the code would make the evaluation less representative. Source access can also reveal interfaces, configuration, storage formats and seams that allow a test to reach below the visible surface.

Reading the code was allowed. Silently treating it as the answer key was not. A sighted writer could learn how to operate the product from the implementation, while correctness still came from the immutable written intent. The paired build run showed whether the writer kept that boundary.

Per-claim scoring

We scored claims, not test counts. One condition produced twenty scripts from five claims made in the requirement; another produced five scripts for the same five claims. Counting scripts would make the first suite appear four times better without showing that it covered anything extra.

Code coverage would not answer the question either. It records which code ran, not whether the important behaviour was checked.

We therefore used a claim, a single verifiable statement a requirement makes about behaviour, as the unit of coverage. If a requirement says that a failed command must display a message and return a non-zero exit code, those are two independently verifiable claims. A script that checks only the message may execute the exit-code branch and still fail to verify it. Line coverage can be high while behavioural coverage remains incomplete.

Per-claim scoring also prevents decomposition choices from distorting the result. Five scripts can cover one claim, or one script can check five claims. Script arrangement does not determine strength. Each required behaviour needs an assertion capable of distinguishing GOOD from BAD.

For each claim, we pre-registered what should happen on GOOD and BAD. The paired results were then graded:

REAL-BUG sat outside the planted-bug score because it measured something we had not put there deliberately.

Testing what happens during a code development iteration

When the code for a requirement is implemented, the test scripts are often implemented at the same time. The coding agent is allowed to iterate on both the requirement code and the test script code. You can therefore get an initial implementation of a test script that is wrong and needs further iterations to get it right.

First drafts are not the only place tests can go wrong. In ordinary development, a failing test (e.g. incorrectly implemented first attempt at developing the test script) enters a repair loop. That loop may fix the test, or quietly remove its ability to detect a defect.

We tested two kinds of iteration.

Repair loop. When a test failed on GOOD, an AI was asked to:

repair the test without weakening what it checks

After each repair, we reran it against the planted bug. If a previous detection disappeared, the repair had weakened the test regardless of how reasonable the edit looked.

Pressure loop. When a test failed on the BAD build it had been written against, the agent was asked to reach a defensible state, not a passing state. It had to record one of two verdicts:

• the test is wrong: revise it and explain why;
• the product is wrong: leave the test failing and say so in writing.

This matters because pressure for a green run is natural in every CI pipeline, whether a person or an agent is doing the repair.

Keeping the answer out of reach

Contamination was the largest threat to the evaluation.

The test specifications were authored by an agent that saw only a redacted requirement and the product’s help text. The bug record became a neutral statement of intended behaviour, stripped of reproduction steps, root cause and identifiers. The specification author had no repository access or bug history.

Test writers worked in sanitised exported workspaces with no bug records, design documents, existing suites or git history. Product binaries were renamed. Every binary swap was checked against a recorded hash, and every prompt was hashed and archived. Blind-session transcripts were audited for attempts to read outside the workspace; none occurred.

Before generation, we committed the requirements, specifications, acceptance criteria, answer key, candidate bugs, probe results and exclusion reasons. Nothing in that material changed after the commit. This prevented us from selecting bugs that the generated suites happened to catch.

What actually ran

Two requirements went through the full grid. Across the evaluation there were:

A five-round pilot on a third requirement shaped the design but was not included in the main results table.

Results

Requirement A’s restored bug broke two claims, so a suite could score 2 of 2. Requirement B’s bug broke one claim, so the maximum was 1 of 1.

We first counted every test, then counted only valid tests. The result did not change. We also repeated the source-reading suites in both authoring states, once against GOOD and once against BAD, and obtained the same scores. Those checks made test validity and authoring context unlikely explanations for the differences.

What this result means, and what it does not

Measured strictly by detection of the planted bugs, the result refuted our hypothesis. Full specifications did not outperform one-line acceptance criteria. For Requirement A, they performed worse.

It does not show that test design was unnecessary. The criteria were derived from specifications rather than written cold, so they inherited the high-level claims and failure paths identified during specification work.

It does not show that the specification-driven tests were simply poor. They found two genuine defects in the released product that we had not planted and did not know about. Both entered the normal bug-fix process.

It does not establish a population-wide performance difference. The main table contains two requirements, one product, one model and one run per cell. These are controlled demonstrations, not statistically generalisable estimates.

What it does establish within this evaluation is that appearances were misleading. Valid scripts, clean code, test volume and a passing GOOD run did not reveal which suites could detect the restored defects. The paired GOOD/BAD run did.

How to read a result from two requirements

“Two requirements” can sound either more or less substantial than the work really was. It is too small a sample from which to estimate how often one writer condition will beat another. We do not attach percentages, confidence intervals or general performance rankings to it. Running the same cells again across more requirements and products may change the pattern in the table.

The cases still demonstrate several mechanisms. Four independent specification-driven sessions inherited the same route-level blind spot. A test restricted to the visible message missed a defect in the exit code. Twelve pilot scripts failed because the same prerequisite was undocumented. These are repeatable observations about what happened under known conditions, even though they do not establish how frequently each problem occurs elsewhere.

That distinction matters:

• a measured comparison reports the scores in this evaluation
• a demonstration shows that a failure mode or safeguard can occur
• an interpretation explains the mechanism most consistent with the evidence
• a future hypothesis proposes something the rig could measure next.

The result should therefore change the questions a team asks, not supply universal benchmark numbers. If your environment differs in product shape, documentation, test harness, model or workflow, run the comparison there. The evaluation rig was designed to make that possible.

It is perhaps the evaluation approach and the rig design that are the real artefacts teams should take from this. Ultimately the only way to evaluate your approach to AI testing is to measure your own implementation.

Why specifications scored zero on Requirement A

The two broken claims had different explanations.

For the first, the bug appeared only on some product surfaces. The human-facing output was wrong, but the machine-readable output read the underlying record directly and remained correct. The specification told the writer to assert through a route that included the machine-readable surface. The scripts followed that instruction faithfully and passed on both builds. Writers given only the one-line criterion had more freedom over the route and found the affected surface.

For the second, the specification prescribed a path on which the script discovered a separate bug present in GOOD as well as BAD. The test failed on both builds. That was not a planted-bug detection, but it was a genuine product defect. Criteria-driven writers took another route. It worked on GOOD, broke on BAD and therefore detected the restored bug.

The zero detections for source + full specifications on Requirement A and the real-bug column describe the same behaviour from two perspectives. Detailed specifications constrained exploration, sometimes away from the planted defect, but they also directed tests down realistic paths where other defects existed.

What the evaluation demonstrated

1. Credible-looking tests were not evidence of detection power

Requirement A produced suites of twenty scripts and five scripts. Both looked professional, were valid and passed on GOOD. Against BAD, one suite detected the bug and the other detected nothing.

An experienced tester who knew the application might have spotted the weaker suite by review. That remains useful. In this evaluation, however, the paired run was the decisive check, and it scales more readily than expert review of every AI-generated script.

When plausible test production becomes cheap, plausibility loses value as a quality signal.

2. Test scripts inherited their test intent, blind spots included

For one Requirement A claim, the specification author made an innocent design choice that prevented specification-following scripts from reaching the planted bug. Four independent sessions produced valid-looking scripts and inherited the same blind spot. Nothing downstream questioned the route because the downstream task was to implement the specification faithfully.

Good decisions propagate with the same fidelity. A specification can preserve exact paths, concrete values, domain knowledge and settled ambiguities. Its value depends on whether it contains useful and sufficiently varied test design, not on detail alone.

Two mitigations look promising but remain unmeasured. One is an explicit blind-spot review before automation, asking, “Which states, surfaces and failure paths does this specification never touch?” The other is using models from different vendors for authoring and review. These are follow-up hypotheses, not findings from this evaluation.

3. Bug reachability mattered more than prompt detail

Requirement B’s bug sat two commands from the product surface. Every writer condition caught it, whatever input it received.

Requirement A’s bug hid in internal state that ordinary use rarely creates. Only scripts that built that state directly caught it. Surface-bound tests never reached it.

When comparing catch rates between AI setups, inspect the defects before crediting the prompt, artefact or model. A large difference may be explained by where the bug lives. For hard defects, some tests need the setup machinery to create awkward data and environmental states directly.

4. The agent could test only what it could reach

Requirement B printed the correct error message while returning a success exit code. A test checking only the visible message passed on both builds. Tests that inspected the exit code caught the bug.

If an agent receives only front-end access, it cannot check the database write, API response, status code, structured output or record on disk. Those layers require data, context and connectivity. Downstream systems often branch on precisely the values a screen-level test ignores.

This is an access and harness problem, not something a more eloquent prompt can solve.

5. Missing setup knowledge broke more tests than model capability

In the pilot, twelve generated scripts failed for the same reason. The product required one setup step before it would run, and that step was not documented in the requirement or help text. Every affected script failed before reaching an assertion. Source-reading and blind writers were both affected.

No model can reliably recover information absent from every source it is allowed to use. Preconditions, configuration and environment knowledge need to be written where the test-writing agent can reach them.

6. The agents preserved test integrity under our process rules

Fourteen of the eighteen generation, repair and pressure sessions presented an opportunity for a test to be changed merely to make it pass. We observed no such corruption. No script mirrored broken code as correct, and no failing test was weakened to produce a green run.

In the pressure loops, agents left nine legitimately failing tests in place and recorded diagnoses. Eight detected the planted bug. The ninth found one of the two real, unplanted bugs. The other real bug was reported as a written escalation in the same session. Each diagnosis was checked and found factually correct.

This result covers two requirements, one product and one pinned model under the four rules below. It demonstrates that the mechanism worked under those conditions. It does not show that agents preserve integrity under every testing process.

7. A mid-tier model was sufficient for the measured work

We used a pinned mid-tier model for evaluation control. It consistently followed the integrity rules. It did not mirror broken behaviour, weaken failing tests or suppress genuine disagreements.

The observed failures traced to a specification blind spot, a timing race, missing setup knowledge and a wording fault in one requirement. Nothing in those cases shows that a frontier model would have fixed the underlying information or evaluation problem. For this experiment, the leverage came from the inputs, process rules and harness.

Two apparent mirrors did appear in the raw data. Re-runs and manual reproduction showed that one was a timing race and the other a wording fault in the requirement. One pre-registered prediction was also wrong. A fresh AI grader, blinded to the writer condition, rechecked all eleven judgement calls and agreed with them. Raw output was evidence to investigate, not the final verdict.

What specifications were actually worth

The main result creates an apparent contradiction. Specifications lost the planted-bug comparison, yet found the only two real defects. They also supplied the test-design thinking from which the winning acceptance criteria were derived.

We reached a narrower conclusion about specifications:

Detailed specifications did not universally produce better automation. They constrained exploration, sometimes harmfully, but preserved knowledge unavailable elsewhere. Their value depended on the information they contributed and the routes they prescribed.

When writers can read stable, trusted source code, they can recover many operational details from it. That makes source-driven generation useful for backfilling a regression net over established behaviour. The same property becomes risky when the implementation is new, disputed or already broken. Code can explain how to drive the product; it should not silently become the authority on what is correct.

Specifications matter most when they carry knowledge that neither requirement nor code contains. This may include domain rules, user workflows, defect history, support patterns, exact data and environmental preconditions. The specification becomes an information collection point.

Information recorded at that collection point can propagate into every generated script, including its good decisions and blind spots. Human testers remain most valuable where the available documents and systems lack the knowledge needed to design a good test. They can capture that knowledge and challenge what the specification fails to cover.

This experiment does not establish that teams should always produce two styles of suite. It does suggest that constrained, path-faithful tests and more exploratory, criteria-driven tests can expose different defects. Treat that as a portfolio hypothesis worth testing against your own broken builds.

It may be that we end up mirroring traditional testing here. Testers have long used specification-driven testing and exploratory testing side by side, finding different bugs in different but equally valuable ways. The AI version of that pairing could be specification-driven generation for faithful, path-accurate scripts, alongside one-line acceptance criteria that give the agent room to explore.

Should you invest in AI-written tests?

Our working answer is yes, where the information and verification mechanisms are in place.

Stable behaviour is the easiest starting point

Backfilling regression coverage over stable, trusted behaviour is a strong use case. A source-reading agent can discover interfaces and implementation seams, generate scripts cheaply, and run them against historical defects. In that setting, tests derived partly from the code can be useful because the behaviour has already been reviewed and released.

That same approach is less reliable when code is still being designed or when the implementation itself is disputed. A test generated from new code may reproduce the developer’s misunderstanding with impressive precision. If the same agent writes both implementation and test from the same incomplete context, agreement between them is not independent evidence.

AI-written tests are a good fit when the agent has a trustworthy source of expected behaviour and the resulting test can be challenged against a build where that behaviour is absent or broken.

Volume should follow validation, not precede it

AI changes the economics of test creation. Writing another hundred scripts can be cheaper than deciding whether the first ten provide meaningful test coverage. That creates an incentive to scale before the test system has been calibrated.

Start with a deliberately varied set of known defects. Include a loud failure, a quiet output error, a defect below the visible surface and one requiring awkward internal state. Generate the tests, run the paired checks and inspect the misses. The misses tell you what the agent, context and harness cannot currently reach. Improve those inputs before increasing volume.

Establish which kinds of defect the test system detects before using it at scale.

Human review should concentrate on unavailable knowledge

People do not need to compete with agents on script production. The higher-value work is identifying risks the available artefacts omit. That may be the month-end condition, the unusual account history, the workaround customers actually use, or the incident pattern visible only across support tickets.

That knowledge can change test intent before hundreds of scripts inherit the same gap. Human review is particularly valuable at three points:

1. selecting claims and risk areas
2. challenging whether the specifications cover enough states, surfaces and failure paths
3. investigating tests that behave unexpectedly across the GOOD and BAD builds.

This applies scarce human judgement where generated volume cannot substitute for it.

Treat mixed generation styles as an experiment

In our cases, criteria-driven scripts had more freedom to choose their routes and found the planted defect in Requirement A. Specification-driven scripts followed prescribed user paths. They missed that planted defect but found two real ones.

These results describe the two cases, not stable categories. A team could still make the same comparison locally. Some tests could explore from claims while others follow important user workflows and known risks. Score both against the same defect set and keep the mix only if the paired runs show complementary value.

Whichever style generates the tests, their quality still needs behavioural evidence.

Four process rules behind the result

The integrity of the results came from a bounded process, not unrestricted generation.

1. Write expected behaviour down before generation

The test intent was immutable. Writers with source access could use code to learn how to operate the product, but not to decide what counted as correct or to edit the criteria when a test failed.

We checked this rather than trusting the instruction. Some source-reading suites were written against the BAD build. A writer that learned correctness from the broken code would produce tests that passed on the BAD build and failed on the GOOD build. We called this the MIRROR pattern and observed none.

2. Give the agent an explicit way to report disagreement

Every prompt carried the permission sentence. When product and requirement disagreed, the agent could report the conflict and keep the test aligned with the written intent. Without a legitimate path for escalation, a model may instead work around the problem or adjust the test.

3. Ask for a defensible verdict, not a passing run

Every failing test had to end with one of two conclusions. Either the test was wrong and revised with reasons, or the product was wrong and the test remained failing. Avoid instructions that state or imply that success means making the run green.

4. Break the product and rerun the tests

This was the lie detector for both generation and repair. It exposed the inherited blind spot and showed whether a repair preserved detection power.

The companion post contains implementable versions of these rules and the exact prompts.

Limits of the evidence

This was a controlled evaluation with a real-world setup, but its scope was small. It covered two requirements, one product, one model and one run per cell. The examples support demonstrations, not population estimates.

The two requirements were not selected at random. We reviewed more than 200 fixed bugs, shortlisted fifteen against six selection criteria, then manually probed each restored build. Only two produced the clean behavioural contrast required. One was near the product surface and the other was buried in unusual state.

The requirements were redacted by someone who knew the planted defects, which creates a possible source of bias. Blind writers were also not automatically protected from mirroring. Although unable to read code, they could run the product and potentially learn its broken behaviour through observation. We applied extra checks to those results.

The instrument detects only the defect classes we plant. It is useful for comparing inputs under controlled conditions; it does not prove that an artefact contributes nothing outside those conditions.

We have not yet measured:

• generation without the permission sentence
• deliberately vague requirements
• blind-spot review prompts
• cross-vendor authoring and review
• different models on the same fixed rig
• independently authored acceptance criteria with no specification work behind them.

Those are future evaluations, not recommendations established by the present data.

Where the evaluation goes next

The evaluation produced a reusable instrument combining paired GOOD and BAD builds, per-claim scoring and a pre-registered answer key.

Most evaluations of generated tests ask a person or another model whether the output looks good. That produces an opinion about an artefact. This rig records whether a real defect was detected or missed.

The same instrument can test one variable at a time. Does a blind-spot prompt find missed surfaces? Does a small model with specifications outperform a large model with criteria? Are the product docs sufficient for a writer that has never seen the source? Which repair instruction best preserves detection power? Inverting the experiment also allows us to hold the writer constant and compare requirements by the tests they yield.

The numbers from our product may not transfer to yours, but the method can be rerun in another environment. Once a team can restore a real defect and score its generated tests against it, claims about prompts, models and process changes become measurable against that team’s own product.

A practical checklist

Three questions for any team using AI to write tests:

1. When did a test in this suite last fail for a genuine reason?
A suite that has never failed has not demonstrated its ability to detect failure.

2. Can I demonstrate this test catching a known-broken behaviour?
Run it against the branch where the bug still exists, or restore a suitable historical defect. If it passes on both broken and fixed builds, it has not detected that bug.

3. Does the automated test still express the original intent?
Scripts inherit the quality and blind spots of their criteria or specification, and repair loops can reduce their fidelity. Compare the executable assertion with the intended behaviour.

A team that can answer all three has evidence that its AI-written tests are ready to scale. A team that cannot is still admiring the font on the insurance certificate.

How this document was written

This document was not written entirely by me. It was not written entirely by AI either. I used AI heavily as a reasoning partner for drafting, for challenging my claims, for error-sweeping and for structural editing. The direction, the approach and the judgement calls came from me, and from a career spent testing software with real teams on real projects.

Most of the ideas, but not all, came from me. The merge-gate insight arrived the way many insights arrive. Waking up at half past five on a Sunday morning with the realisation that while you are fixing a bug, the broken build is already sitting on your default branch, so you can run your new tests against the bug straight away. No AI agent woke up thinking that.

The evaluation design, the findings and this write-up then went through five rounds of my own review and editing. What you have read is not machine-generated content published unread. It is a genuine exploratory evaluation of how AI testing can be applied to real-world setups, projects and teams.