AI makes each of us more productive on our own. The trouble usually starts when your whole team starts using it.
Everyone prompts it differently. Everyone keeps their own notes. Everyone uses their own workflow.
Before long you have tests nobody fully trusts and know-how nobody shares. At that point you find that AI is slowing your team down, not speeding it up.
I’ve been working on a way of testing with AI that keeps a team working consistently. It has three parts:
- harness the work
- guide the AI
- assure the result
This post gives you an overview of all three and how they work together.
The kits that put this into practice are coming soon.
Two problems it has to solve
The test passes when it should fail
Ask an agent to fix a failing test and it is very good at making it pass. Sometimes that means you actually get a fixed test. Sometimes it means weakening an assertion, dropping a step or changing an expected value.
The run is green. The test no longer checks what you agreed, and nothing in the result tells you.
That’s not testing. That’s papering over the cracks!
The guides go stale
Every agent session starts with no memory. Your agent reads your guides. Guides that tell an agent how you write tests, which helpers to use, how to run things, what’s currently broken.
Those guides fill up fast, and a lot of what goes in is written by AI. Then a helper gets renamed, a fixed wait gets removed, a workaround outlives its bug.
Nobody goes back to tidy up. The guide is still pointing at the past. Every agent using your guides is living in the past too.
So every new agent session misses the mark.
The Harness: defining the workflow
You’ll have seen the plan, implement, verify approach many recommend for agentic development. The Harness is the testing version of that loop.
- Test case = Plan. What the test must prove, written from the story before any script exists.
- Automate = Implement. AI writes the script from the approved test case.
- Execute = Verify. Run it, and check the result against the plan.
Between the first two steps sits a person. Nothing is automated until someone approves the test case. And that test case is the plan the agent needs to follow in order to automate it.
Every test case gets a unique ID from a deterministic tool, never from the agent. That can be a light script that stamps a template with an ID. Or you can plug in GTMS for the full lifecycle.
Guide: the know-how every step reads
The Guide holds how you test on this project. It’s how you write tests, reuse steps, create test data, work with environments, run and diagnose tests. You’ll probably find a list of known issues useful too.
It also learns. When a test fails, a diagnosis finds out why, with evidence, and the lesson goes back into the guides.
One rule keeps it honest. A guide says only what is true now. When something stops being true, it leaves the guide. A past event worth keeping moves to a history file; the rest is deleted, and git still has it.
Assure: checks it, independently
Assure answers both problems. Its checks run in a fresh session, separate from whoever did the work. An independent review.
check-intent compares the test script with its approved test case. Same steps, same values, same expected results? Does this automated test do what the test case says it should? And it asks this even when the test is passing.
check-guides checks every claim in the guides against the repository as it is now: still true, no longer true, or no evidence. A person confirms each fix before it’s written.
How the three fit together
Three kits, seven commands. Each command does one job, then stops.
And the flow, end to end:
When a test fails, it goes back a step. Fix the script and run it again, or fix the test case and approve it again. It is never patched to go green.
The rules that always hold
- Nothing is automated until a person approves the test case.
- Never fix a test just to make it pass (no weakened assertion, longer timeout and so on).
- Each command does one job, then stops.
- Checks run in a fresh session (checks are independent)
- Test case IDs come from a tool, never from the agent (deterministic processes as the bedrock)
What’s next
This grew out of the Testing Playbook, which becomes the Guide. The three kits are coming soon: one repo, one folder each.
Nobody has fully worked this out yet. This is the shape that has held up for me so far. Next, I’ll go deeper into those two problems and how the checks catch them.