TLDR; Record and Playback is a notoriously bad way to automate. But what if we record, then have AI write code from the recording? Answer: it is a little better.
I performed an experiment to try tand demonstrate how to replicate the AI-powered test automation features commonly found in commercial SaaS testing tools using open source solutions. The process involved recording user interactions in an application with Chrome DevTools Recorder, exporting the recording as JSON, and then using an AI tool to convert the JSON into JUnit tests with WebDriver and page objects.
Overview Video
In this video I:
- record some web actions using Chrome Dev Tools recorder
- fail to play them back because record and playback is notoriously bad
- feed the recorded JSON file into OpenCode with a prompt
- check the results of the AI process writing Automated Execution using WebDriver and Java
We do get working automated coverage out the back. I would need to edit it a little, and if I wanted to do this longer term I would need to amend the prompt to meet my coding preferences.
process
I used the triangle application on Test Pages:
Test Pages has a deliberately hard to automate side bar. The easy automation is through the top level menu.
But I recorded the Triangle application by:
- clicking on the side menu bar to open the Triangle application
- and the instructions
- reading the instructions
- clicking back on the triangle application
- interacting with the app to generate invalid and valid triangles
When I tried to playback the recording in the Chrome Dev Tools, it failed.
Most of us experience failure with record and playback tooling.
AI tooling
I have OpenCode, configured to use Chrome Dev Tools MCP, and working with Kat Code Pro Free in OpenRouter, but could just as easily have used any of the free coding LLMS on Open Code Zen or with Ollama locally.
- https://opencode.ai/
- https://github.com/ChromeDevTools/chrome-devtools-mcp/
- https://openrouter.ai/
- https://openrouter.ai/kwaipilot/kat-coder-pro:free
I used a basic prompt
“given the json attached, convert the json into a set of junit tests using webdriver and page objects to recreate the flows recorded by the user. The json file is a recording made in Chrome Dev Tools recorder”.
The process took about 5 minutes to run.
- It opened the web page a few times,
- created some Page Objects.
- Found that the menu interaction was hard.
- Decided to jump straight to the triangle page and automate that way
- It did replicate the functional interaction included in the recording
- it did generate assertions for the results
- I could have used the test code generate with some small amendments.
Lessons Learned
SAAS tools which take this approach will use Agents, which are essentially longer prompts. I should have added into the prompt more conditions:
“given the json attached, convert the json into a set of junit tests using webdriver and page objects to recreate the flows recorded by the user. The json file is a recording made in Chrome Dev Tools recorder. In the test code, do not write any findElement code, always abstract the location and interaction into a page object. Use existing page objects where possible and try to re-use existing methods. Make sure that any page object methods you use are used in a test so we have coverage of the interactions.”
Over time, if I wanted to continue to automate using this process, I would refine the prompt further until I got the results I wanted, and possibly encode this prompt as an agent.
Would I do this?
Unlikely that I would do this in the real world.
This seems like the replication of a record and playback approach.
I do create Page Objects and have created initial execution coverage using AI.
I have some empathy for AI tooling scanning applications and building flows through the application to automatically interact and generate a wide variety of data, to increase the scope of coverage across a few simple flows. I can see potential in augmenting a team to allow the team to go off and do more detailed stuff. Provided the AI tooling doesn’t get noisy and keep distracting the team.
For any functionality as simple as I experimented with here, I hope that when people evaluate SAAS tools, that they also compare the paid tools to the capabilities of Open Source tools.
Nothing wrong with paid tools. They are incredibly valuable when they make hard things simple, and add value when they make hard things easier.
But, when the activity is simple and the functionality differences between Open Source and paid are small, I hope companies consider “what if we upskilled our staff a little such that we could use the Open Source tools”.
