Can this prompt-driven test automation system scale with complex applications using lessons learnt loops?
I wondered if AI could fail at automating a complex test case, learn from that failure, and succeed on the second try. Here’s my attempt at building a process with a feedback loop that acheives that.
Can this prompt-driven test automation system scale with complex applications using lessons learnt loops? I know from experience that AG Grid scenarios are really difficult to automate – drag-and-drop, row grouping, complex UI interactions. They’re automation nightmares. So what happens when AI comes up against these sorts of challenges?
What I’m looking at is scenarios where you encounter a complex application that defeats standard automation approaches. Where you can get Claude Code to complete some deep research and work out solutions. Solutions you can then build into a learning loop. And then, maybe creating a growing knowledge base of lessons that gets better over time, making each subsequent test easier to automate.
Let’s see if we can get this to work.
I picked AG Grid specifically because I know it’s going to fail with the standard, simple, approaches used with test automation. The drag-and-drop implementation in AGgrid is non-standard, the components are complex, and it’s exactly the kind of application that makes automation engineers question their career choices.
Here’s how the Lessons Learnt Loop works:
The Core Philosophy:
Traditional test automation often fails when encountering complex UI patterns – and then you’re stuck. You either spend hours debugging selector strategies, or you give up and mark it as “manual only.” The Lessons Learnt Loop treats this as a learning problem instead. Your first execution attempt gathers data about what didn’t work. An ultra-thinking phase analyzes why it failed and documents solutions. The second attempt uses these lessons to succeed where the first attempt failed.
This shift in thinking has interesting implications. Just like a student learns from mistakes, our automation system can build knowledge about specific testing challenges. The AG Grid drag-and-drop that defeats standard automation becomes a documented, solved problem. The lessons learnt document becomes organizational knowledge that can be shared across teams. The result: automation that gets smarter with each challenge it encounters.
The Three Key Components:
Failure Detection and Analysis
Not all failures are created equal. An “element not found” error is different from “drag started but drop didn’t register.” This approach captures the nuances of complex failures – particularly important with frameworks like AG Grid that implement custom event handling. The failure analysis isn’t just logging errors; it’s understanding the underlying cause. This deep understanding is what enables the ultra-thinking phase to generate meaningful solutions.
Ultra-Thinking Research Phase
The ultra-thinking capability is like having a senior automation engineer research the problem for you. It doesn’t just try random alternatives – it systematically investigates the framework’s architecture, understands the implementation choices, and proposes solutions based on that understanding. For AG Grid, this meant discovering the custom event system and proposing API-based alternatives. This isn’t trial and error; it’s informed problem-solving.
Knowledge Integration
The lessons learnt document isn’t just a one-time fix – it becomes part of the automation’s knowledge base. Future test cases can reference these lessons, avoiding the same failures. Over time, you build a comprehensive understanding of your application’s quirks and complexities. This is organizational learning embedded in your automation system.
The Four Commands Enhanced:
Building on the Test Automation Compiler from Experiment #4, I’m using the same four commands but with lessons learnt integration:
/discover – Discovery execution (now accepts lessons learnt documents)/learn – Extract automation patterns (incorporates lessons)/generate – Create YAML from learnings (uses lesson-based strategies)/validate – Validation execution (confirms lesson effectiveness)I took an AG Grid test case through the complete learning loop process – expecting failure, researching solutions, and applying lessons.
What I asked:
/discover TestManagement\test-cases\TC-002-aggrid-column-management-and-row-grouping.md
What happened:
Claude Code attempted to execute the test case using standard Chrome DevTools MCP approaches. The test involved dragging column headers into the row grouping panel – a seemingly simple interaction that’s actually one of AG Grid’s most complex features.
The discovery phase captured:
“It’s not behaving in the way our AI agent was expecting. It’s a non-standard implementation of the drag and drop components… It can drag, but it can’t actually drop.”
This wasn’t a surprise – I specifically chose AG Grid because I knew it would fail. AG Grid uses a custom drag-and-drop implementation that doesn’t respond to standard browser events. The drag appears to work visually, but the drop zone doesn’t accept the element. This is exactly the kind of complex scenario that makes automation engineers either write custom JavaScript handlers or give up entirely.
The key insight here is that the failure was informative. We learned exactly what doesn’t work and why – setting up the next phase perfectly.
What I asked:
please can you examine why this drag-and-drop for the row grouping didn't work,
ultrathink and come up with solutions. Please write these solutions to the
file aggrid-lessons-learnt.md
What happened:
Claude Code spent five minutes in deep research mode. This wasn’t just error analysis – it was comprehensive investigation into AG Grid’s architecture and implementation choices.
The ultra-thinking phase discovered:
The research went deeper, analyzing AG Grid’s documentation and finding:
applyColumnState() methodThe output was a comprehensive lessons learnt document with:
This research phase is transformative. Instead of blindly trying different selectors or timing strategies (the traditional debugging approach), we’re building understanding of the underlying system. The lessons learnt document isn’t just a workaround – it’s a knowledge artifact that captures expert-level understanding of AG Grid automation.
What I asked:
/discover TestManagement\test-cases\TC-002-aggrid-column-management-and-row-grouping.md
aggrid-lessons-learnt.md
What happened:
I completely cleared the context – this was a fresh start with no memory of the previous failure. The only difference was the inclusion of the lessons learnt document.
This second discovery run was radically different:
gridApi.applyColumnState() instead of drag-and-dropnode.setExpanded(true)“It’s been through all of the test steps. It says that it matches exactly what was expected in the test case.”
The transformation was impressive. What failed completely in the first attempt now executing flawlessly. The drag-and-drop interaction that would have taken complex custom code was replaced with a simple API call. The visual result was identical – the Country column appeared in the row groups panel, the data reorganized into groups, and the Belgium group expanded to show Isabella Kingston’s data.
But here’s the crucial point: the test case didn’t change. The markdown still described dragging and dropping. The AI learned to interpret “drag Country to row groups” as “apply row grouping by Country” and implement it the most reliable way possible.
That’s pretty impressive!
What I asked:
/learn TestManagement\discoveries\TC-002-discovery-log.json aggrid-lessons-learnt.md
Then generate the YAML specification:
/generate TestManagement\learnings\TC-002-learnings.json
TestManagement\test-cases\TC-002-aggrid-column-management-and-row-grouping.md
aggrid-lessons-learnt.md
Finally, validate the compiled automation:
/validate TestManagement\specs\TC-002-test-spec.yaml
What happened:
The complete 4-phase workflow (Discover → Learn → Generate → Validate) now incorporated the lessons learnt at every stage:
Learning Extraction Enhanced:
The /learn command didn’t just extract patterns from the discovery log – it cross-referenced them with the lessons learnt document. It understood that certain UI interactions should be replaced with API calls. It learned that timing for AG Grid animations needs special handling. It identified which selectors were reliable and which were fragile. The output wasn’t just a pattern list – it was an intelligent synthesis of discovered behavior and documented solutions.
YAML Generation Transformed:
The /generate command produced a radically different YAML specification than it would have without the lessons. Instead of drag-and-drop instructions, it contained API calls. Instead of complex event sequences, it had simple state changes. The YAML included:
Validation Results:
The validation phase confirmed that our learning loop worked perfectly:
✅ Validation Complete - TC-002 PASSED
Outcome: ✅ PASS (13/13 steps successful)
📊 Key Metrics
| Metric | Result |
|-------------------------|-------------------------------|
| Total Steps | 13 (Setup: 3, Test Steps: 10) |
| Successful Steps | 13 |
| Failed Steps | 0 |
| Assertions Passed | 4/4 |
| Deterministic Execution | ✅ YES |
| Spec Followed Exactly | ✅ YES |
| Fallbacks Used | ❌ NONE |
| Improvisation | ❌ NONE |
The validation was twice as fast as discovery because it executed deterministically without exploration. Every step passed on the first attempt. No fallbacks were needed. The API approach was not just a workaround – it was demonstrably superior to UI automation.
The Run Twice Pattern Validated:
Both the discovery run (with lessons) and the validation run produced identical functional outcomes:
This proves the YAML specification successfully captured ALL automation knowledge, including the lessons learnt. The compilation from markdown to YAML was complete and correct.
After watching this 4-phase workflow (Discover → Learn → Generate → Validate) complete, some clear patterns emerged:
Works well for:
Gets messy with:
Surprises:
⚡ Quick Verdict:
🟢 GREEN: “I’m going to start using this in production” – for catching up on technical debt
The Good:
The Concerns:
Would I use this?
Absolutely. I’m already planning to scale this up for production use. The ability to fail, learn, and succeed systematically is transformative for complex test automation. This isn’t just a clever experiment – it’s a practical solution to real problems I face daily.
Here’s where I’d use it immediately:
When would I NOT use it?
There are clear boundaries to this approach:
What I’m still curious about and want to test further…
The Lessons Learnt Loop isn’t just error recovery – it’s systematic knowledge building for test automation.
Traditional automation fails and stays failed. This approach fails, learns why, documents solutions, and succeeds. The difference is profound:
The pattern demonstrated here – fail → research → document → succeed – mirrors how human experts develop. We’re essentially teaching our automation system to become an expert through experience. The ultra-thinking phase acts like a senior engineer researching a problem. The lessons learnt document captures that expertise permanently. The second attempt applies that expertise successfully.
This approach transforms test automation from a brittle, high-maintenance burden into a learning system that gets smarter over time. Every challenge makes it stronger. Every failure makes it wiser.
This experiment proves that test automation with AI can literally learn from its own failures. The Lessons Learnt Loop isn’t just a workaround for complex scenarios – it’s a fundamental shift in how we approach automation challenges.
What started as an expected failure with AG Grid’s drag-and-drop became a demonstration of systematic problem-solving. The automation failed, researched why, documented solutions, and succeeded on the second attempt. The real success was turning an “impossible to automate” scenario into a solved problem.
The implications extend beyond this single test case. Every organization has applications with quirky behaviors that defeat standard automation. Every team has that list of “manual only” test cases they’ve given up on. The Lessons Learnt Loop offers a path forward: systematic learning that turns automation failures into documented solutions.
What makes this approach particularly powerful is its alignment with how organizations actually work. Senior engineers naturally build mental models of application quirks. The Lessons Learnt Loop captures that knowledge explicitly, making it shareable, reusable, and permanent. When that senior engineer leaves, their automation knowledge stays.
The four-phase workflow (Discover → Learn → Generate → Validate) now has an enhancement: front load the earlier phases with lessons learnt. This creates a positive feedback loop where automation gets progressively smarter. Each failure contributes to future success. Each lesson learned benefits all subsequent tests.
I’m moving this from experiment to production. The approach is mature enough, the results are compelling enough, and the need is definitely there. Those AG Grid tests that have been “manual only” for years? They’re about to become automated.
If you’re interested in trying this, these are the exact prompts I used:
Want to try this yourself?
Really was simple to get setup once you understand the pattern: Fail → Research → Learn → Succeed. I’ll get this packaged up and released on GitHub very soon. Then you can try it and let me know what happens when you try it – I’m especially curious about what frameworks defeat your first attempt and whether ultra-thinking helps you too.
