AI Experiment #1: Recording Videos and Converting Them to Test Cases

Can you actually record yourself testing and get test case documentation back automatically?

I wondered if this works… so I tried it.

The Question

I hate writing test case documentation, especially after I’ve already done the testing manually. So I wondered… what if I just record a video of myself testing and let Claude Code write the test cases? Could this actually work?

I’m kind of thinking of two potential scenarios if this works. First, when you have something to test but nobody has bothered to write a decent spec – you could record yourself exploring the application and generate test cases from that. Second, maybe you have a wireframe of the application (before it’s built) and you can record yourself stepping through the wireframe to create your test cases before development even starts.

Anyway, no point speculating about how to use this if it doesn’t work. So will it work?

What I’m Using

• Screen recording – Windows Snipping Tool for capturing test execution
• ffmpeg – To extract frames and process video
• Claude Code – AI coding assistant with vision capabilities
• Financial dashboard demo app – Test application for the experiment

I had ffmpeg already installed (it comes with a Playwright install). The approach would be to use ffmpeg to convert video into a sequence of screenshots that Claude Code could then process and convert into a structured test case in markdown format.

The Setup

Here’s how I got this working:

Installation:
ffmpeg was already on my system from Playwright. If you need it, you can get it from ffmpeg.org.

Configuration:
Created a test case template and some principles for writing good test cases (both in markdown). I used Claude.ai to generate these with these prompts:

• Please create me a Test case template with standard fields (ID, title, preconditions, test steps, expected results)
• Please create a Test case creation principles document with rules for writing good test cases

These documents provide context for Claude Code so it knows what structure to follow.

Test Case Template:
Standard template with placeholders for ID, title, description, preconditions, test steps, expected results, and actual results.

Starting Point:
Fresh Claude Code project initialized with /init, the template files, and a 43-second screen recording of me adding an investment account to the financial dashboard application.

The Experiment

Try #1: Can Claude Process Video Directly?

I asked Claude Code: “You are an expert in software testing. My goal is to create a process where I can record a video of me using an application and have you turn it into a structured test case in markdown format. Are you able to process and understand video?”

What happened:
Claude said yes, it could process video. So I asked it to take my test-add-account.mp4 file and create a test case.

It came back with an error reading the file.

Reaction:
As expected – Claude can’t directly process video files. But this was the learning moment that led to the actual solution.

Try #2: Using ffmpeg to Extract Frames

I prompted Claude to use ffmpeg to convert the video into screenshots that it could then analyze.

What happened:
Claude immediately understood the approach and started generating the ffmpeg command. It suggested using 1 frame per second, but I thought that wouldn’t be enough detail.

Adjustment:
I asked it to use 2 frames per second instead: “Please continue with this approach but use fps of 2.”

Result:
It created a temporary frames directory and extracted 84 PNG images from my 43-second video (roughly 2 per second as requested).

Try #3: Generate Test Case from Screenshots

Claude then read and analyzed all 84 screenshots, extracting test steps and expected results from the sequence of images.

What happened:
It created a complete test case file: TC-001-add-investment-account.md

The test case included:

• Test case ID and title
• Full description of what’s being tested
• The actual values I’d used in the test (account type: “Investment”, name: “Main Investment”, and the exact description text I typed)
• All the navigation steps
• Expected results for each step
• Screenshots attached as evidence

It actually caught specific details I’d entered – “Main Investment” as the account name, the exact description text I typed, even the modal form behavior and validation messages.

That’s pretty impressive!

Patterns I Noticed

After running this experiment, some patterns emerged:

Works well for:

• Short, focused recordings (under 2 minutes)
• Clear, deliberate actions (not clicking too fast)
• Happy path flows where you’re showing the intended behavior
• When you narrate what you’re doing (helps provide context)
• Recording exploratory testing sessions you’re doing anyway
• Creating test cases for applications with poor or missing specifications

Gets messy with:

• Long exploratory sessions (with no clear objective)
• Multiple test scenarios in one video (boundaries get confused)
• Fast clicking and navigation (harder to capture what’s happening)
• Expected vs actual results – it sees what happened, not necessarily what SHOULD happen

Surprises:

• It caught UI elements I didn’t explicitly mention or focus on
• Better at documenting steps than creating assertions/validations
• Quality of test case depends heavily on recording quality and pace
• Narration in video helps A LOT with context and understanding
• The 2 frames per second rate worked well – enough detail without overwhelming
• For production use, you’d create the frames, generate the test case, then delete the temporary screenshots

The Honest Take

⚡ Quick Verdict:
🟢 Yes, but not for everything

Would I use this?
Absolutely. Not as a replacement for all test documentation, but definitely for specific scenarios.

For what?

• Quick documentation of manual tests I’m already doing anyway
• Bug reproductions where I want both video evidence and structured test steps
• Onboarding new team members – record once, get sharable test cases
• When I’m too lazy to write it up properly (let’s be honest)
• Applications where specs are missing or poor
• Exploratory testing sessions where I want to capture what I discovered

When would I NOT?

• Complex business logic verification where precise assertions matter
• Anything needing precise validation points

Still needs refinement for longer test runs. A lot of potential once perfected. You’d want to add some validation loops to ensure the test cases being generated are accurate and complete.

Still Curious About

What I’m still curious about and want to test further…

• Can it spot differences between expected and actual results if I point them out in the video?
• Would audio narration improve the accuracy significantly?
• Can it generate Given-When-Then format specifically for BDD test cases?
• Will it work with something like AGgrid or other complex component libraries? (That’ll be interesting!)
• What about testing mobile apps using screen recordings from devices?
• How would it handle wireframe recordings to generate test cases before the app is built?

The Prompts I Actually Used

If you’re interested in trying this, these are the exact prompts I used:

In Claude.ai (for setup):

Please can you create me a test case template for software testing in markdown

Please can you create me a list of principles and rules that should be followed when creating good test cases for software testing

In Claude Code:

please review files in this project

You are an expert in software testing. My goal is to create a process where I can record a video of me using an application and have you turn it into a structured test case in Markdown format. Are you able to process and understand video?

please take the video test-add-account.mp4 and create a test case

[Request interrupted by user for tool use] please continue with this approach but use fps of 2

please could you take this process and the steps followed and create a claude code slash command named create-test-from-video

Resources

• ffmpeg: https://www.ffmpeg.org/
• Demo Application: Vercel Demo Application 
• Claude Code: https://claude.com/claude-code
• Claude Code Cookbook (Vision best practices): https://github.com/anthropics/claude-cookbooks/blob/main/multimodal/best_practices_for_vision.ipynb 

Want to try this yourself?

Really was simple to get setup once you have ffmpeg installed. Let me know what happens when you try it – I’m especially curious about whether audio narration improves the results and how it handles more complex applications.