THE AUTOMATION SPRINT

Test Automation: from initial implementation
through to long-term sustainability

Bill Echlin

The Automation Sprint

Copyright © 2025 Test Management Systems Ltd

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Test Management Systems Ltd or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Test Management Systems Ltd have endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Test Management Systems Ltd cannot guarantee the accuracy of this information.

First published: January 2025

Published by Test Management Systems Ltd.

Test Management System Ltd

The Counting House

9 High Street

Tring

HP23 5TE

UNITED KINGDOM

ISBN 978-1-0369-1964-1

Contents

Preface

Chapter 1: Introduction

Background

The Automation Sprint

A Two-Track Development Model

The Test Development Pipeline

Skills and Resources

The Trouble with Test Automation

"The Automation Sprint" Components

Chapter 2: The Two Track Development Model

The Framework Track (Developers)

The Test Track (Testers)

Interlinking the Framework Track and Test Tracks

The Framework Build and Enhancements

Test Development and Maintenance Process

Chapter 3: The Sprint Objective

Five Contiguous Days or Five Distributed Days

Five Contiguous Days

Five Distributed Days

Which Option Is Best for You?

The Automation Sprint Overview

Chapter 4: Day 0 - Sprint Preparation

Identify the Challenge

Pick the Team

Select the Tool Set

Allocate the Time and Resources

Setup the Test Environment

Chapter 5: Day 1 - Requirements & Specification

Capture Your Requirements

Writing the Specification

Identifying Tests

Test Case Identification Example

Prototype Build

Pick Your Development Model

Chapter 6: Day 2 - Prototype Development

Develop a Skeleton Prototype

1. Define the Core Set of Test Cases that the
Framework Should Be Tested with

2. Setup CI Environment

Chapter 7: Day 3 - Review and Test

The Demo

Test Core Framework

Test Adding Tests

Define Priority Enhancements/Fixes

Chapter 8: Day 4 - Iterate

Test Within a CI Environment

Refine Your Requirements

Develop Enhancements

Build a Pack of Example Tests

Chapter 9: Day 5 - Continuous Test Setup

Write the Core Set of Automated Tests Within the Framework

Stability Execution Runs

So What Next?

Chapter 10: The Test Development Pipeline

Maintaining Quality and Stability

A Stable Foundation for Growth

Chapter 11: Identifying and Developing New Tests

Overview

Identifying New Tests for Automation

Integration with Sprint Planning

Collaborating with Framework Developers

Conclusion

Chapter 12: Workflow, Quality Gates and Standards

Overview

Workflow Definition

Test Development Rules

Standards For Test Development

Conclusion

Chapter 13: Continuous Test Review Cycles

Overview

Identify and Remove Redundant or Low-Value Tests

Check for Tests that Could Be Consolidated or
Simplified

Validate that Tests Still Align with Business
Requirements

Conclusion

Chapter 14: Health Metrics and Monitoring

Overview

Types of Metrics

Conclusion

Chapter 15: Knowledge Management System

Overview

Training Materials

Knowledge Transfer Sessions

Troubleshooting Guides

Clear Ownership and Escalation Paths

Conclusion

Chapter 16: Update and Regression Strategy

Overview

Scheduling Framework Updates

Process for Testing Framework Changes

Strategy for Updating Test Data

Plan for Handling Application Interface Changes

Conclusion

Chapter 17: The Automation Sprint Conclusion

Appendices

01 - Preparation Check List

02 - Writing The Framework Requirements

03 - Writing The Framework Specification

Preface

It's a common challenge for test teams: how to implement effective test automation that delivers lasting value without becoming a maintenance burden? This book presents a comprehensive approach to test automation that addresses this challenge. We'll address three interconnected components. Those three components are the Automation Sprint, the Two-Track Development Model and the Test Development Pipeline. These elements provide a practical framework for creating and maintaining successful test automation initiatives.

Test Automation Success

The Two-Track Development Model is an automation framework development approach that separates concerns between testing and development tracks. It introduces a degree of separation of effort that allows developers and testers to maintain close collaboration. This model recognises that effective test automation requires deep testing expertise and deep development skills. Embracing both skill sets is the best way to deliver a successful automation solution in a reasonable time frame. By clearly defining roles and responsibilities and ensuring continuous communication, teams move faster and deliver a more robust test automation solution.

The Five-Day Automation Sprint transforms framework development from an open-ended project into a focused, time-boxed effort with clear deliverables. This intensive approach aims to break down the complex task of creating an automation framework into manageable daily objectives. The goal is to move from an initial concept to a working framework within a week. The sprint format ensures that teams maintain momentum while validating their automation approach through practical usage. We're looking to avoid the pitfall of perfectionism that often delays framework implementation and adoption.

The Test Development Pipeline addresses the crucial question of long-term framework maintenance and test development. Knowledge management, framework updates, and quality controls all contribute to an automation process that continues to deliver value over time. This approach to framework maintenance helps teams avoid the gradual decay that often undermines automation initiatives. It's a way to ensure that automated testing remains a reliable part of the quality process even as applications and teams evolve.

These three components work together to provide a complete solution for teams looking to implement sustainable test automation. Whether you're starting a new automation initiative or improving existing automation efforts, the approach described in this book will help you support an organisation's quality objectives.

Chapter 1

Introduction

Background

"The Automation Sprint" is born of the widespread frustration many teams experience when implementing effective test automation solutions. It's focused on the implementation process rather than the technology involved.

Test Automation Sprint Components

We've called the overall approach "The Automation Sprint." It's a step-by-step approach from initial implementation through long-term sustainability. One of its central pillars is the "Automation Sprint" phase. This sprint phase forms the core around which we wrap the Two Track Development Model and the Test Development Pipeline.

The Automation Sprint

We've found that dedicated five-day sprints are the most successful approach to implementing test automation solutions. These sprints follow a proven, step-by-step process. The sprint methodology is effective because it directly addresses a common failure point in test automation projects: trying to do too much over an extended period of time.

Teams often expect to produce the perfect test automation solution at their first attempt, failing to define a bold implementation time frame. This leads to a drawn-out implementation phase where attempts are made to automate everything. You then end up with a half-finished framework that delivers poor test coverage and unreliable automated tests. We avoid this by breaking down the initial automation project into one focused, achievable sprint. This sprint follows a step-by-step, iterative process undertaken over five days.

A Two-Track Development Model

The key to automation success is the "Two-Track Development Model." This model bridges the gap between implementation and a sustainable test development process. Critically, we acknowledge that successful test automation requires both development and testing expertise.

By separating framework development from test creation, teams move faster during sprint-based implementations. Technical support provided through the Test Development Pipeline helps ensure long-term maintainability. This division allows skilled developers to build robust and scalable architecture while testers can concentrate on creating valuable test scenarios using simplified interfaces. This approach accelerates initial development and promotes a sustainable test development model. Framework improvements and test development proceed in parallel, each track maintaining its own velocity.

The Test Development Pipeline

After the implementation sprint(s), the "Test Development Pipeline" kicks in to ensure the project's ongoing success. A robust framework counts for nothing without dedicated maintenance and the development of reliable tests. This pipeline is the backbone of sustainable test automation. It establishes the rhythm and routines needed to maintain the efficiency and value of automated testing long after initial implementation. It ensures that test assets evolve with the application, creating a systematic approach that transforms test automation from a one-time project into a living part of the development process.

Skills and Resources

The success of these three concepts depends on skills and resources. Your team likely already possesses most of the skills needed for test automation. However, you should identify and address any skills gaps up front. Only then can we focus on clearly defined goals and the deliverables that provide the needed automation capability.

It's all very well having the right skills. However, if you can't apply those skills and allocate the resources, you won't achieve your objectives. This is why we aim to allocate and deploy those resources as part of a sprint and focus the right resources on the right tasks. Developers on the framework and testers on the tests.

"The Automation Sprint" is a proven process for getting the job done and maintaining it in the long run!

The Trouble with Test Automation

Everybody in a software development project wants test automation. It is a desired end goal promising rapid testing with little resources. To deliver on this goal, the test team needs to be able to add tests to an automation framework with the same ease as creating manual tests. However, that's the ideal. Even getting close will make a big difference.

Most teams fail, hitting problems with the implementation and maintenance phases.

Thus, many automation initiatives fail to deliver on their promises. Teams often dive in enthusiastically but quickly become overwhelmed by the technical complexity of building a robust framework. Why is this? Manual testers may struggle with programming concepts that are far removed from their core skills. Developers may create technically sound but overly complex solutions that complicate adding and maintaining test tests for the testers.

Even when automation is successful, challenges often arise. Test suites become brittle and break with every minor application change. This results in increasing maintenance costs that spiral as more tests are added. All this creates a growing burden that eventually eclipses the time saved by automation. Hence, the promise of faster testing cycles often becomes a project millstone, consuming resources while delivering diminishing returns.

But you've heard all of this before. What you need is real-world solutions!

These common problems highlight the need for practical, real-world solutions. These failures are not inevitable; they're symptoms of treating test automation as a purely technical challenge rather than a socio-technical one. Traditional approaches often ignore human factors, such as the diverse skills within a team, the learning curves involved, and the need for sustainable processes. With its three core components, the "Automation Sprint" approach offers a solution for all this.

The "Automation Sprint" starts by separating framework development from test creation. It uses an automation implementation approach with focused sprints, where teams can play to their strengths to build momentum and avoid common pitfalls. This approach ensures that technical excellence doesn't come at the cost of testing expertise and that rapid progress doesn't sacrifice long-term sustainability.

This isn't a new concept. The BBD (Behaviour-Driven Development) approach exists already, where someone from the business, development, and/or test teams writes a Gherkin script. Let's be honest, though: Just how many projects really work with the "Three Amigos" (Business, Development, and Testing) to define BDD scripts that can be turned into automation scripts? Not many!

Many testers and test teams are left to define BDD test scenarios and then produce the underlying automation code. This is why "The Automation Sprint" approach aims to avoid BDD's complexities, maximise team skills, and focus people, time, and effort on delivering a realistic test automation capability.

"The Automation Sprint" Components

The "Automation Sprint" approach originally comprised a series of prescriptive steps within a standard sprint used for many software development projects. It has since evolved and now includes three core components:

1. The Two-Track Development Model

2. The Automation Sprint

3. The Test Development Pipeline

In the following sections, we pick up on these three components.

In Part 1, we’ll explore ways to divide framework and test development work, match the right skills to develop the more complex framework code, optimise the testers' role in defining framework needs, and develop the scripts that need to be written.

In Part 2, we will examine a fast-paced approach to the "Sprint Base Implementation" process for developing the automation framework. We will discuss both contiguous and distributed 5-day sprints, considering them within a single dedicated sprint and spread over the 5 days out, coexisting within another existing sprint.

Part 3 describes the "Test Development Pipeline" and the essential components needed for it to work in the long term, including different approaches for allocating this work. We look at approaches from both a Sprint and Kanban perspective. You'll need to identify a test case development process that fits your team and your organisation.

We'll see how "The Two Track Development Model" weaves its way through both the Sprint-Based Implementation and the Test Development Pipeline.

Chapter 2

The Two Track Development Model

Many test automation initiatives fail. Five main reasons contribute to this failure rate:

1. Over-ambitious scope leading to overwhelm

2. Lack of maintenance strategy causing test debt

3. Poor tool selection creates long-term problems

4. Insufficient focus on knowledge transfer

5. Treating automation as a "side project"

"The Automation Sprint" is an approach designed to mitigate these potential pitfalls using the "Two Track Development Model" at its heart. This two-track model straddles all aspects of the implementation and test development. Specifically, it includes:

▪ A dedicated sprint with a well-defined, achievable scope.

▪ Tool selection that meets the demands of both the testers and developers in a project.

▪ Developing a system built by developers with a test development/maintenance approach easily managed by a tester with moderate coding skills.  

▪ A "Two Track Development Model" where the developer deals with the framework complexities and the Tester focuses on test development/maintenance.

▪ Implementation of a training program that ensures knowledge transfer across the whole team.

▪ The allocation of resources as part of a dedicated and well-scoped sprint

"The Automation Sprint" can be viewed as a sprint to deliver a product to the tester(s) and/or test team, followed by implementing a Test Development Pipeline that supports the tester. A pipeline that enables the Tester to deliver automated tests quickly. This focused effort provides the tester(s) with solutions and support to allow them to write and execute automated tests. Everyone involved in this Sprint and the Pipeline should view the tester as the customer and end user. Whilst the tester is closely involved in the Sprint, the team members prioritise meeting the tester's needs.

The main goals are to deliver:

1. a system enabling the tester to add automated tests easily - no complex coding!

2. a process enabling the tester to request additional automation functionality - avoiding complex coding!

3. systems and processes allowing the tester to execute tests and report on results

Remember, the tester is the end customer. The goal is to deliver a system that simplifies the writing of automated tests - avoiding complex and time-consuming coding. All built around processes that allow the tester to request and receive more complex enhancements to these systems from coders (internal or external). Remembering also that coders are best placed to deliver enhancements and functionality. This makes implementing automated tests quick and easy (for example, developers write modules to interact with complex grid components). All of this wrapped in systems (e.g. CI/CD tools) and processes (e.g. source code control) that makes for easy to manage and maintain automated tests.

Many test automation projects fail because the entire project is delegated to a developer uninterested in test automation or to a tester lacking some of the skills to develop a high-quality test automation framework. Another reason is that typically, the project manager (or the person with the authority to allocate resources) gives little thought to allocating the correct skills and resources to ensure the project's success.

Most automation projects require diverse skill sets specific to multiple roles and people. One of the key concepts in "The Automation Sprint" is breaking up the framework build and the test development. Once the build is complete, the delineation between framework maintenance and test maintenance is continued. Different roles undertake different aspects of the project. We will explore this demarcation next.

This simple but fundamental approach to creating effective test automation solutions is based on the "Two Track Development Model." It divides the task of building out test automation capabilities into two tracks.

The Two Track Development Model

1. Framework build and maintenance

2. Writing and maintaining automated tests

These two tasks require distinctly different skills, so in "The Automation Sprint", we divide the tasks into two tracks with two teams (or two people for a smaller project). With the "Two-Track Development Model", we design a system and processes where framework development and test creation run in parallel. There is a clear separation between the two tracks where we have:

Framework Team:

-   Skilled developers focusing on building robust, reusable components

Test Team:

-    Testers focusing on test logic and business scenarios using simplified interfaces delivered by the framework team

The following sections provide a more detailed exploration of these two aspects.

The Framework Track (Developers)

The Framework Track requires a skilled developer(s) to focus on:

Architecture Quality — Developers build a robust, scalable framework without compromising technical capabilities due to limited skills. They properly implement design patterns, develop error handling, and build in robust logging and reporting, ensuring that these often-overlooked aspects are implemented effectively rather than being missed by less experienced coders.

Technical Debt Prevention — Experienced developers are better equipped to:

▪ Write maintainable, well-structured code

▪ Write code to organisationally defined standards

▪ Implement proper version control practices

▪ Plan for future scalability

Performance Optimisation — Developers implement efficient solutions for common bottlenecks:

▪ Smart waiting strategies for dynamic elements

▪ Optimised database access patterns

▪ Efficient test data management

▪ Resource clean up and browser session handling

Note: Some of these capabilities might already be built into your chosen automation tool. However, developers with a deep understanding of coding must oversee their implementation.

The Test Track (Testers)

The Test Track requires skilled testers who are focusing on the following:

Business Logic - Testers concentrate on what they do best:

▪ Designing effective test scenarios

▪ Maintaining test data

▪ Ensuring proper coverage of business rules

▪ Identifying edge cases from their domain knowledge

Add Tests Fast - By working with simplified interfaces, testers can:

▪ Start writing automated tests much sooner

▪ Focus on test logic rather than implementation details

▪ Build on their existing testing knowledge

▪ Expand their coding knowledge from the already implemented framework

Higher Quality Tests - When testers aren't struggling with complex coding concepts, they can:

▪ Write more comprehensive test cases

▪ Better document the test's business purpose

▪ Maintain better traceability to requirements

▪ Spend more time on test design rather than debugging

Note: In this context, we are not talking about skilled test automation engineers. We are talking about skilled testers who can analyse business requirements, design tests, etc.

With these two pieces in place, the interlinking of the Framework Track and the Test Track is critical now.

Interlinking the Framework Track and Test Tracks

Both tracks must run in parallel and communicate effectively for this approach to succeed. To ensure this, your team must follow three core concepts embedded in the processes that address the technical, personal and process aspects of building your test automation capability. They are as follows:

1. Technical: "Building the Bridge Layer":

Create a simplified interface layer between the framework and test scripts:

▪ Use business-oriented method names that match how testers think

▪ Implement strong parameter validation to prevent common errors

▪ Create helper methods that combine multiple framework calls into single, logical operations

▪ Build in logging and debugging features that are understandable to non-developers

2. Personal: "Simplified Test Syntax":

Design a test writing approach that feels natural to manual testers:

▪ Use a syntax that reads like plain English

▪ Create method names that match testing terminology (e.g. confirmButtonDisplayed() instead of   assertElementExists())

▪ Build reusable business logic based on test components that encapsulate common sequences of actions

▪ Implement smart defaults to reduce the amount of code the tester needs to write.

3. Process: "Knowledge Transfer and Support Model":

Establish a sustainable system for collaboration:

▪ Regular, sprint-style, stand-up meetings to discuss progress and issues

▪ Documentation that focuses on "how to use" rather than "how it works"

▪ Template libraries for common test patterns agreed upon between the developer and tester

▪ Code review process that focuses on test logic rather than implementation details

While the above three points are important, remember that a more important aspect will be a strong working relationship between developers and testers. Running parallel tracks is a less tangible but still important aspect.

The Framework Build and Enhancements

Experience shows that 'testers' don't possess the deep development skills to implement complex automation frameworks. What might be even more controversial is to say that many 'test automation engineers' lack these skills either. A test automation engineer may be good at writing and maintaining automated tests. Still, it does not mean that they possess the required skills to build an architecturally sound automation framework in the first place.

You need a skilled developer with a solid understanding of programming fundamentals, OOP principles, design patterns, and an ability to construct code with sound architectural foundations in mind. The ability to design modular and structurally sound code is a must.

This initial build phase is a one-off task that you need to get right. Get a good developer, not a tester or a test automation engineer.

To get a good developer, you can use internal resources or outsource to freelancers. If you have internal resources, that's great. However, I've often found that even when these internal resources are allocated, they find it difficult to prioritise test automation above other work they've been allocated.

For this reason, outsourcing is a good approach. Outsourcing yields superior results. Allocating this framework build task to good freelance developers. This doesn't necessarily mean employing contract developers at high rates. You'll find many great developers on freelancer websites like Upwork.com or Freelancer.com. You get top-quality developers who will create automation frameworks far better and faster than your tester can. Not only this but you can find developers that have a good understanding of the tech stack that you're automating.

For example, let's say you need to test an application that is written with React. Say you've also chosen Playwright with Typescript as your automation tool. You can post a well-defined job specification and find a freelance developer with good React and Typescript skills on a freelancer website. State that you need an automation module to deal with interacting with React JS Grid, and that this developer must be capable of delivering code quickly.

The key to all of this is getting the right people with the right skills to undertake the right jobs. As a result, the right people with the right skills build the right test automation framework!

Test Development and Maintenance Process

The framework build stage aims to deliver a system that makes it easy for the tester to develop and maintain automated tests. In this setup, Testers have three main tasks.

1. Understanding business scenarios that need testing

2. Designing tests to test these business scenarios

3. Writing/maintaining automated tests at speed

Notice that this list does not include building and extending the automation framework. Adding this task to the list of tasks that the tester has to accomplish usually pushes automation projects over the limits, i.e., it often leads to "Likely to Fail" automation projects.

The reason may be either the tester struggles to deal with the complexity of the framework coding required or he/she spends so much time dealing with the complexity that the project effectively stalls. This is why automation projects become painfully slow in terms of building out the number of stable automated tests.

The "Automation Sprint" addresses this by intelligently dividing the work between developers (either internal or external) and the tester team.

The next chapter details how to structure a sprint to develop and implement a core framework. "The Test Development Pipeline" will be discussed later in the book.

Chapter 3

The Sprint Objective

This initial piece of work is not about producing a complete, fully functional working test automation framework. It's about creating a working prototype that proves the concept and confirms that the proposed architecture will scale. It's about setting the groundwork with a handful of fully working automated tests. This groundwork then providing the spring board that defines the way of working and the future functionality you that you'll need in place.

The 5 Day Automation Sprint

We approach this by breaking the framework's build down into a number of preparation tasks, followed by five sprint days. Then, to continue building momentum, we finish off with review and planning sessions.

Preparation:

▪ Identify the Challenge

▪ Pick the Team

▪ Select the Tool Set

▪ Allocate the Time/Resources

▪ Setup the test environment

Day 1: Requirements & Specification

▪ Capture Requirements (tester)

▪ Write the Specification (tester)

▪ Identify initial test cases to implement (tester)

▪ Kick off the Prototype Build (developer)

Day 2: Prototype Development

▪ Develop skeleton prototype (developer)

▪ Define a small set of test cases framework should be tested with     (tester)

▪ Setup CI environment (tester)

Day 3: Review and Test

▪ Demo (developer)

▪ Test Core Framework (tester)

▪ Test Adding Tests (tester)

▪ Define Priority Enhancements/Fixes (tester)

Day 4: Iterate

▪ Implement Fixes and Develop enhancements (developer)

▪ Define Core Set of automated Tests within the framework (tester)

▪ Setup CI Reporting (tester and developer)

Day 5: Continuous Test

▪ Write Core Set of automated Tests within the framework (tester)

▪ Stability Execution Run (tester)

▪ Review the original challenge

▪ Asses results against initial challenge

▪ What next?

Below, we provide an overview of this sprint structure. Then, we look at what's involved for each day in turn. First, though, we discuss the pros and cons of a five-day contiguous sprint vs. distributing five days over a longer period of time.

Five Contiguous Days or Five Distributed Days

There are two ways to allocate these five days: contiguous and distributed. Both offer distinct advantages and challenges. The key, regardless of approach, is to factor in nominated full days with dedicated resources for those days.

The contiguous approach provides intense focus and rapid iteration. It is ideal for teams that can dedicate resources exclusively to framework development for a full week. This concentrated effort often leads to faster completion and more cohesive solutions.

In contrast, the distributed approach, where you're spreading the five days across a period, offers greater flexibility and easier resource allocation. This makes it more suitable for teams with ongoing commitments or limited resource availability. While the distributed approach may sacrifice some efficiency due to context switching, it can be more practical.

The choice between approaches ultimately depends on organisational constraints, resource availability, and team structure. We discuss both approaches in more detail below.

Five Contiguous Days

The concentrated approach of five consecutive days offers powerful advantages in terms of focus and momentum. Teams can fully immerse themselves in framework development without context switching or competing priorities. This intense focus often leads to faster problem-solving and a more cohesive team. Both development and test tracks maintain awareness of decisions and progress. This uninterrupted allocation of time allows for rapid iteration between development and testing tracks. You'll find that you'll benefit from the immediate feedback and quick resolution of issues as they arise.

However, the contiguous approach demands significant resource commitment. Organisations must be willing and able to dedicate key team members exclusively to the framework development effort for a full week. This can be challenging in environments with ongoing project commitments or limited resources. Teams also need to ensure that all necessary stakeholders are available throughout the week. Delays due to unavailability can disrupt the sprint's tight timeline.

Five Distributed Days

Spreading the framework development across a longer period offers greater flexibility in resource allocation. Teams can integrate framework development into their existing sprint structure, which clearly makes it easier to balance with other commitments. This approach can be particularly beneficial when working with team members who have ongoing responsibilities. It also allows more time for reflection and research between steps, which leads to more considered decisions about framework design and implementation.

The distributed approach does introduce some challenges, though. Context switching between framework development and other tasks will reduce momentum. Teams may need to spend additional time at the start of each framework development day refreshing their understanding of previous progress and decisions. There's also a risk of losing continuity in the development process. This is especially true with longer time gaps between framework development days. Communication between tracks may require more formal documentation to maintain alignment across the distributed timeline.

Which Option Is Best for You?

The choice between these approaches often depends on organisational constraints and team structure. Teams with dedicated resources and strong management support might benefit most from the focused energy of contiguous days. Conversely, teams operating within already established sprint structures or with limited resource flexibility might find the distributed approach more practical.

Regardless of the chosen approach, success depends on maintaining clear communication between tracks and ensuring that framework development receives total focus on the allocated days. Teams should carefully consider their specific circumstances and constraints when deciding which approach best suits their needs.

As already stated, the critical point is to factor in nominated full days with dedicated resources for each of the five days.

The Automation Sprint Overview

The five-day automation sprint leverages the Two Track Development Model to transform a testing challenge into a working automation framework through focused, parallel efforts. This intensive approach pairs a testing track (responsible for requirements, specifications, and validation) with a development track (responsible for framework implementation).

The sprint begins with a preparation stage, including tool selection and team setup. It then moves through a structured sequence of framework development, testing, and refinement days. Each day builds upon the previous one, with clear hand-offs between tracks and continuous validation of the framework's capabilities. The parallel nature of the two-track model maximises productivity, allowing for rapid development and immediate feedback.

By the sprint's end, the team emerges with not just a working framework but also the automated testing infrastructure. This includes continuous testing, reporting, and a core set of automated tests. The compressed timeline demands careful resource allocation and clear communication between dev and test tracks. When properly executed, this will deliver a practical, validated automation framework that meets real-world testing needs.

The success of this approach relies heavily on the dedication and coordination of both tracks. Understand, though, that this framework development approach is an iterative process. You may need to repeat and iterate with multiple sprints (we discuss this below). Yet, whilst five days might seem aggressive for developing a complete testing framework, the focused nature of the sprint and the two-track model make it eminently achievable.

Chapter 4

Day 0 - Sprint Preparation

A bit of preparation will set the foundation for success before launching into the five-day automation sprint. Teams must first pinpoint the specific automation challenge they aim to solve. You'll want to ensure that solving this challenge will really add value to your test process.

The selection of team members will be crucial. The two-track development model requires both a skilled tester to write the framework specification and an experienced developer for implementation. Teams also need to evaluate and choose their automation tools carefully. You should consider factors like supported tech stacks, coding languages, integration capabilities, and maintenance requirements.

We've put together a checklist to help you ensure you've covered all the preparation tasks before you start your sprint. This "Preparation Check List" is in the appendices in the end of this book.

Identify the Challenge

Identify the automation challenge you're undertaking, and you'll find it far easier to focus on what needs to be delivered during the Automation Sprint. Look beyond the simple desire to "automate testing" and dig deep into the specific problems you aim to solve. This might include addressing slow manual testing cycles, reducing regression testing bottlenecks, or improving test coverage across multiple platforms. By clearly defining the problem(s) you must solve, you can better shape your automation approach and measure its effectiveness.

You should be answering the questions:

•           Why are we doing this project?

•           Where do we want to be six months to a year from now?

•           How will we know if this has been a success?

Setting precise, measurable results creates a focused path for the automation sprint. It's not always possible, but establishing specific metrics will help. You may be aiming to reduce the execution time for a regression pack by a certain percentage. Or you're looking at increasing test coverage to a target level. Or even simpler, enabling testing across a defined set of browsers or devices. These concrete goals guide the framework's development and provide clear benchmarks to evaluate the sprint's success. Without these well-defined targets, teams risk building a framework that, while functional, may not address their most pressing testing challenges or deliver the intended value to their testing process.

This should take no more than 30 minutes to an hour to complete. Once completed, it will serve as a guide when you start defining requirements and writing specifications.

Pick the Team

One of the most critical aspects of delivering a test automation solution that works is making it as easy as possible for testers to add automated test cases. It's getting the balance right between the required complexity to interact with your application and the ease with which tests can be added. That's why this developer/tester pairing, two-track approach, is so critical. The developer deals with the coding complexity. The tester guides the developer and implements the tests.

It's a similar approach to the BDD concept that's widely accepted. However, it doesn't necessarily need the complexity of implementing natural language constructs. The high-level test step statements can be written in code but are designed to hide as much complexity as possible. In removing this layer, the tester needs to understand some core coding concepts. The tester needs to be able to code, but not to the same degree as the developer. The developer needs to implement the architecture and implement the more complex methods/routines.

This approach relies on us having three main roles in the project:

Product Owner:      

This role is responsible for resourcing the project, defining the challenge, and setting long-term and short-term goals. This role could be filled by a test manager or a scrum master. It's someone who has the authority to allocate resources and will be responsible for the overall delivery of the working product.

Tester:

The tester is responsible for defining the requirements and helping outline the specifications. Once the framework has been delivered, the tester will also be responsible for testing it. Ultimately, the tester will be writing the automated tests and so needs to be very clear about the framework that the developer needs to create.

Developer:

The developer is responsible for building the framework and implementing all the low-level code. The developer will create the initial framework and then support and extend it as the project evolves.

There are also optional roles that you should consider filling.

Trainer:          

If you plan to extend this automation project to many testers, you will probably need to consider some sort of training package. This need not be a dedicated trainer but someone who has the capacity/talent to teach new automation engineers the core coding principles required and how to write/maintain new automated tests.

Stakeholders:            

Any other person with an interest in the effective delivery of your test automation project. This could include people from a scrum team who are interested in seeing automated testing delivered. Perhaps others who have a vested interest in seeing an automated regression test pack reduce a product's test cycle time. These stakeholders only need to see test result reports. Either way, it's important to consider their needs and factor them in from the start.

The key to all of this is the dynamic between the tester and the developer. Ultimately, the tester delivers the usual test development and test execution capability to the team. In this automation setup, they will need to drive the developer to get the required capability in place. They will need to challenge the developer to produce a system that enables fast and easy development of the automated tests. This is the key bit for the tester. To oversee the implementation of this capability, not to implement this capability themselves. The tester is the developer's customer. The developer is the supplier to the tester. This is a reversal of the usual dev/test relationship. It's essential to get right with "The Automation Sprint" approach.

Select the Tool Set

Selecting the right automation toolset requires careful consideration of both immediate needs and long-term sustainability. Teams must evaluate tools based on their technical requirements, such as tech stack support, programming language compatibility, support for target platforms, and integration capabilities with existing development tools. Special attention should be given to the learning curve associated with each tool, as the five-day sprint format demands quick adoption by both the development and testing tracks. The chosen toolset should align well with the team's technical expertise while meeting the project's automation objectives.

Cost considerations and community support are vital in tool selection but shouldn't overshadow practical usability. Teams should prioritise tools with good documentation, active user communities, and proven track records in similar testing scenarios. The selected tools must also demonstrate good maintainability characteristics, as the framework must evolve beyond the initial sprint. While popular tools often seem like safe choices, teams should focus on finding tools that address their defined challenges rather than following industry trends. This targeted approach ensures the selected toolset will effectively support the immediate sprint goals and long-term testing needs.

Allocate the Time and Resources

Setting aside dedicated time and resources is a critical success factor for the five-day automation sprint. The two-track model demands full commitment from the tester and developer, requiring them to clear their schedules of other responsibilities during the sprint. Organisations must understand that any compromise on this dedicated focus (such as allowing team members to work on different projects simultaneously) can derail the tight timeline and reduce the sprint's effectiveness. The sprint's intensive nature requires not just allocating people but also ensuring they have uninterrupted time to collaborate and solve problems as they arise.

Beyond personnel allocation, teams must secure the necessary infrastructure and support resources before the sprint begins. This includes having development and test environments ready, ensuring access to required licenses and tools, and arranging for any needed subject matter expert availability. Teams should also plan for adequate meeting spaces or virtual collaboration tools to support frequent communication between the testing and development tracks. While five days might seem like a short time investment, the resource allocation must be treated as a serious commitment to avoid the common pitfall of partial dedication that can lead to incomplete or poorly implemented frameworks.

Setup the Test Environment

Setting up a robust test environment for framework development brings testers face-to-face with an ironic twist - they must now create a testing environment to test their own testing tools. This meta-testing scenario requires careful consideration of how to validate the framework itself. Testing the framework works correctly before it's used to test actual applications. The environment needs to include sample applications or components that will serve as test subjects, allowing the framework to be exercised against real-world scenarios.

Creating this test environment demands attention to both technical and practical considerations. Teams must establish version control for the framework code, set up appropriate test data, and ensure the environment mirrors the conditions where the framework will ultimately operate. The testing track needs to think through scenarios for validating framework features, much like when testing application features. You need to be aware that there's an added layer of complexity here. You're testing the tools you'll use for future testing. This self-referential nature of testing a test framework often helps testers gain valuable perspective on how developers feel when creating testable code, fostering better collaboration between the two tracks during the sprint.

Chapter 5

Day 1 - Requirements & Specification

The first day focuses on establishing a clear vision and laying the technical groundwork. The tester dives deep into capturing detailed requirements, documenting how the framework should function and what problems it needs to solve. They craft a comprehensive specification that serves as the development blueprint while identifying an initial test case that will validate the framework's core functionality. Meanwhile, the developer begins building a prototype based on the emerging specifications, setting up the basic structure that will grow into the whole framework.

Capture Your Requirements

It may seem excessive, especially as us engineering types like to jump straight into solutions. However, it is worth spending a bit of time defining your requirements. That is, think through what exactly you'll need from your test automation framework.

I know some of you will think that you're following the Sprint methodology, so why aren't you going with user stories and all that "As a user..." stuff? Well, I'm writing the specification as the user. In this instance, I am the user. Personally, I find it easier (and I think you will too) saying, "This is what I want ..."

Please don't skip this step! So many times, I've seen this upfront work save days of wasted work on the back end. Equally, don't spend days working on this. In "The Automation Sprint," your team should be able to complete this work in under two hours with a Whiteboard and Post-It notes. Either...

1. Divide up the topics below across the sprint team. One or two areas each.

or

2. Swarm each topic, letting everyone contribute ideas for each topic.

Next, I list some areas that will help you elicit a good set of requirements.

Stakeholder conversations:

Before writing anything, talk to the people who will use the framework. Talk to business analysts, developers, testers, DevOps engineers, etc). Understand their pain points, what they want to automate, and their technical capabilities. Also speak with project managers about delivery timelines and business goals.

Research existing solutions:

Look at any current testing approaches, even manual ones, across your organisation. Examine similar frameworks in the industry and their documentation. This helps identify both good practices to adopt and pitfalls to avoid.

Document the current landscape:

Map out your existing application architecture, tech stack, and deployment processes. This helps ensure your framework requirements align with your actual infrastructure and tooling.

Create a progressive outline:

Start broad, then refine. Begin with significant sections like "Purpose", "Technical Stack", "Core Features", "Integration Requirements". Under each, list bullet points. These points can later be expanded into detailed paragraphs.

Imagine how you'll use it:

Consider who will use the framework and how. For example: "I need to create data-driven tests with the data in this format". Or "When triggering in a CI pipeline, I will need to pass in these parameters."

Prioritise requirements:

Not everything needs to be in version 1.0. Use MoSCoW prioritisation (Must have, Should have, Could have, Won't have) to organise requirements by importance and feasibility.

Validate interactively:

Share drafts with stakeholders early and often. Technical teams can spot implementation challenges, while business stakeholders can confirm if it meets their needs.

This document doesn't have to be some massive tomb of text that people will baulk at reading. Stick to one or two pages. Keep it succinct and to the point.

You'll find an example of the Requirements document in Appendix 02.

Writing the Specification

You're going to get a developer to build your framework. This means your team needs to write a good specification document. It doesn't have to be pages and pages long. Just something that will guide and constrain the work that will be undertaken. You should outline the functional capability you need this framework to deliver.

A good specification document should start with a high-level overview paragraph clearly stating the framework's purpose. Detail the core business problems it aims to solve and the key technologies involved. This sets the context for your developer and helps them understand the fundamental goals.

The bulk of the document should break down your requirements into distinct areas - technical architecture, test capabilities, integrations, reporting, and maintenance considerations. Each area should be described in clear, specific language without room for misinterpretation. Avoid vague terms like "fast" or "user-friendly" - instead, provide measurable criteria like "we should be able to execute tests against a React JS application" or "framework should include detailed setup scripts with examples."

Also, consider the appropriate level of detail required for your developer's needs. Technical teams need specific implementation guidelines. Use consistent terminology throughout, and remember that a good specification is prescriptive enough to guide development but flexible enough to allow for technical problem-solving during implementation. Consider including examples that add clarity, particularly for complex requirements like handling specific application components.

As you write your specification, you can use the following checklist to ensure that you have covered all the key points.

☐     Purpose-driven framing
Start by clearly stating what problem the framework solves and its intended benefits for the team and project.

☐     Technical precision
Specify exact technologies, versions, and tools (like "TypeScript 5.0" rather than just "TypeScript") to avoid ambiguity.

☐     Scope definition
Explicitly state what the framework will and won't handle, preventing scope creep and setting clear expectations.

☐     Component-specific details
Break down requirements for testing specific components (like we did with the Grid), including their unique challenges and requirements.

☐     Integration points
Clearly outline how the framework should integrate with existing systems, CI/CD pipelines, and other tools in the ecosystem.

☐     Non-functional requirements
Include performance expectations, browser compatibility requirements, and scalability needs.

☐     Implementation guidelines
Provide specific guidance on design patterns, code organisation, and architectural approaches without being overly prescriptive.

☐     Maintainability considerations
Address requirements for documentation, code standards, and knowledge transfer to ensure long-term sustainability.

☐     Testing approach
Define the types of tests to be supported and how they should be organised and executed.

☐     Success criteria
Establish measurable criteria for determining when the framework meets requirements.

☐     Future-proofing
Include considerations for extensibility and how the framework should handle future additions or changes.

☐     Error handling and reporting
Define expectations for the framework's handling and communicating of failures, including logging and reporting requirements.

This may seem like a lot of work. It's not! Compared to developing and maintaining a framework yourself, it's a fraction of the cost and takes a fraction of the time. It also helps build a good ongoing relationship with your developer. You want to develop this relationship as it'll also be key in the "Test Development Phase" of your automation project. Rather than waste days trying to write code for other complex components in your automation framework, you'll be able to leverage the help of this development resource.

Outsourcing to a dedicated developer can be fast and will deliver a far better framework. That's why you want to get the fundamentals in place right from the start, and build that relationship with the developer that's going to support you.

You'll find an example of a specification document in Appendix 03.

Identifying Tests

There are two parts to this. First, we can use some tests to help build the framework. Second, once the initial framework is complete, we'll need a process that helps us identify and prioritise tests that should be automated.

Identifying Tests for the Framework Build

Focus on selecting a small set of tests that exercise different aspects of both the application and the framework capabilities. I would recommend an approach that encompasses three types of tests:

1. Simple Happy Path

Start with a simple, stable "happy path" test that everyone knows works—like a basic login flow. This helps validate the core framework setup and basic patterns without getting caught up in complex scenarios. It's perfect for proving the framework, the test environment, and the execution pipeline.

2. Complex Elements

Look to add a test involving grid or table interactions - these typically involve multiple UI interactions and data validations. This helps establish patterns for handling complex elements and data verification. Looking at our earlier grid testing requirements, this could validate basic sorting or filtering.

3. Business Workflow

Finally, pick a test involving a complete business workflow with multiple steps - like creating a new record that appears in search results. This helps validate how the framework handles test dependencies, state management, and multi-step processes.

These three test types should cover enough variety to prove the framework design while being simple enough to implement quickly. The key is choosing stable, well-understood test cases that help establish good automation patterns rather than trying to tackle the most complex scenarios first.

Test Case Identification Example

Here are specific example tests in each category that would help validate your framework:

Simple "Happy Path" Test:

-     Log in with valid credentials

-     Verify successful login (correct user name displayed, redirected to dashboard)

-     Log out

✓         This validates basic navigation, element interaction, and assertion patterns.

Grid/Table Interaction Test:

-     Navigate to a data grid page (e.g., customer list)

-     Sort by customer name column

-     Filter for customers in a specific region

-     Verify grid results match filter criteria

-     Export grid data and verify contents

✓         This establishes patterns for complex element handling and data verification.

Complete Business Workflow:

-    Create a new customer record with multiple fields (text, drop-down, date picker)

-    Save the record and verify the success message

-     Navigate to the customer search

-     Search for the newly created customer

-     Open and verify all details match the input data

✓         This validates state management, data handling, and navigation between features.

Bonus Test (if time permits):

-     API call to create test data

-     UI verification of created data

-     Update via UI

-     API verification of update

✓         This helps establish patterns for API/UI hybrid automation.

Each of these tests exercises different framework capabilities while remaining straightforward enough to implement quickly. They give you a solid foundation to build upon while validating your framework design decisions.

Prototype Build

Now, we're ready to start the build. We have:

•           the requirements

•           the specification

•           the required toolset

•           the initial test cases to automate

We have enough detail for our developer to get started. When starting the build, we must be conscious of the development model. Are we outsourcing this work to a specialist developer? Or are we passing this work on to someone internal to our organisation? Either way, that resource should be lined up and ready to take on this work.

Pick Your Development Model

When working with an internal development team, the handover of requirements and specifications benefits from direct, face-to-face collaboration. The testing track should schedule a detailed walk-through session with the developer. Here, you can explain the requirements, demonstrate the initial test cases, and answer questions in real time. This immediate feedback loop clarifies ambiguous points quickly and helps the developer understand not just the "what" but the "why" behind specific requirements. The testing track should remain readily available throughout the first day as the developer begins to prototype the framework structure.

Working with outsourced development presents a different set of challenges and requires more thorough upfront documentation. The testing track needs to create more detailed technical specifications. Immediate clarification may not be possible due to time zones or communication delays. This means your specifications should include concrete examples, pseudo-code where appropriate, and clear acceptance criteria for each requirement. Setting up a shared collaboration space (like a wiki or documentation repository) helps maintain a single source of truth for requirements. The testing track should also establish clear communication channels and schedule regular asynchronous check-ins to ensure the developer can get clarification when needed.

In both scenarios, the testing track should provide sample test cases demonstrating the range of testing scenarios the framework needs to support. These are practical examples that help developers understand the framework's required capabilities. However, with outsourced development, these examples need to be more comprehensive and include edge cases that might otherwise come up naturally in casual conversations with an internal team.

Whether internal or outsourced, the testing track should establish clear milestone checkpoints for the project, ensuring the prototype development stays aligned with the sprint's aggressive timeline.

Chapter 6

Day 2 - Prototype Development

On the second day, both tracks move into high gear with parallel activities.

The developer focuses on expanding the prototype, adding core features and establishing the basic architecture. At the same time, the tester defines a set of test cases that will validate the framework's essential functions. Together, they set up the continuous integration environment, laying the groundwork for automated building and testing of the framework.

This dual-track approach keeps development aligned with testing needs while maintaining rapid progress.

Develop a Skeleton Prototype

The second day of the sprint creates a perfect opportunity for parallel work streams that maximise productivity. While the developer focuses on building the framework based on day one's specifications, the testing track focuses on two critical preparatory tasks that ensure smooth framework validation and integration.

1. Define the Core Set of Test Cases that the Framework Should Be Tested with

The testing track begins by defining a comprehensive set of test cases that will validate the framework itself - a crucial step in ensuring that the testing tool can be trusted. These test cases should cover the fundamental capabilities outlined in the requirements, such as handling different types of UI elements, various data inputs, and different testing scenarios. The test cases should also include edge cases and error conditions to verify the framework's robustness.

This collection of test cases serves dual purposes: they will validate the framework's functionality and serve as examples for future users.

2. Setup CI Environment

Simultaneously, the testing track collaborates with the developer to establish a continuous integration environment. This involves setting up the build pipeline, configuring source control integration, and establishing automated framework testing.

The CI environment must handle both the framework code and the test cases that validate it. Special attention should be given to reporting mechanisms that highlight any issues. You should be able to distinguish between framework failures and legitimate test failures. This infrastructure work, though potentially challenging, is crucial for maintaining framework quality as development progresses through the sprint and beyond.

The parallel nature of these activities - framework development, test case definition, and CI setup - embodies the efficiency of the two-track model. While the developer implements the core functionality, the testing track prepares the groundwork for thorough validation and sustainable framework maintenance. This approach ensures that by the end of day two, the team will have a prototype framework and the infrastructure and test cases needed to validate and maintain it.

Chapter 7

Day 3 - Review and Test

The middle day centres on validation and refinement through hands-on testing and feedback. The developer demonstrates the current state of the framework, allowing the tester to begin testing of core features. The tester explores the process of adding new tests, conducting a detailed usability assessment, and documenting required improvements. Based on this feedback, the developer starts implementing enhancements while maintaining the framework's stability and core functionality.

The Demo

This is the pivot point in the sprint where theory meets practice, and the framework faces its first real test. The developer starts with a framework demo, walking through its architecture, key features, and implementation details. This demonstration isn't just a show-and-tell session - it's an interactive knowledge transfer where the testing track can ask questions, challenge assumptions, and begin to understand how they'll work with the framework in practice.

Test Core Framework

Following the demo, the testing track dives into hands-on testing of the core framework features using the test cases defined on day two. This initial testing phase often reveals practical usability issues that weren't apparent in the requirements phase. The tester explores how intuitive it is to create new tests and how well the framework handles different testing scenarios. Take time to review the original requirements and asses the implementation against these requirements.

This exploration frequently uncovers gaps between theoretical design and practical usage that must be addressed. If this is the case, make sure you document and highlight these issues quickly. The sooner you can provide feedback to the development track, the sooner your developer can take corrective action.

Test Adding Tests

Usability in this context is about how usable the framework is for the tester. Usability testing represents a structured evaluation of the framework from a tester's perspective. The testing track systematically works through common testing scenarios, documenting friction points, unclear behaviours, or missing features. This isn't about finding bugs - it's about identifying ways to make the framework more efficient and user-friendly. Questions like "How many steps does it take to create a basic test?" or "How clear are the error messages?" help identify areas for improvement.

Testing the process of adding new automated tests represents a critical validation phase that goes beyond just checking framework functionality. The testing track creates various types of test scripts, starting with simple scenarios and progressively moving to more complex test cases. This hands-on experience helps identify any barriers or friction points in the test creation process, such as overly complex syntax, missing helper functions, or unclear documentation.

The tester becomes the framework's first user, experiencing first-hand what future testers will encounter when building automated tests.

Define Priority Enhancements/Fixes

The testing track compiles a prioritised list of enhancement requests based on the testing and usability assessment. These include additional helper functions, improved error handling, or simplified interfaces for common tasks. The developer can implement these enhancements immediately, taking advantage of the remaining time in day three. This rapid feedback and development cycle is crucial for ensuring the framework will be genuinely helpful in real-world testing scenarios, not just theoretically complete.

Prioritising enhancements and fixes require carefully balancing immediate needs and long-term usability. The testing track categorises issues found during testing, distinguishing between critical fixes that could block framework adoption and nice-to-have enhancements that would improve efficiency. Priority is typically given to issues that affect core functionality, test creation workflow, or framework stability. The tester must consider their own experience and how other testers with varying skill levels might interact with the framework.

Chapter 8

Day 4 - Iterate

Day four marks the transition from prototype to practical implementation. You'll probably have to accept that the initial version of your framework doesn't meet all your needs perfectly. In all likelihood, your framework will include some significant bugs. Accept the need for rapid refinement based on real usage experience and iterate to improve.

While the development track implements priority enhancements identified from the previous day's testing, the testing track works on creating a set of automated tests that serve both as framework validation and future reference implementations.

Test Within a CI Environment

This is where you'll need to set up and run your continuous testing environment with your reporting infrastructure. You'll need this to test your tests! Tomorrow's session will focus on stability. You'll need this in place if you're going to develop the required levels of execution stability. You don't need anything complex here—a simple one-click or scheduled execution capability.

Testing within your CI environment will help ensure clear visibility into the framework's health and test execution results. These parallel framework and test tracks enable the framework to evolve quickly while maintaining stability and usability. At the end of the day, you'll see not just an improved framework but also the infrastructure and example tests you'll need to support wider adoption.

Refine Your Requirements

The fourth day forces you to acknowledge the fundamental truth in software development - initial prototypes rarely meet all needs perfectly. So today, you'll also want to focus on refining the framework by defining a targeted iteration cycle. You'll start thinking ahead and collate enhancement requests into future planned cycles. In short, identify what can be done this week (with the time left) and what needs to be pushed out. Remember that you need both tracks working in parallel to improve and validate the framework's capabilities.

The development track tackles the enhancement list compiled from day three's testing, making necessary adjustments to improve framework functionality. These enhancements range from simple usability improvements to more substantial architectural changes. Wherever needs changing, the testing track will decide based on its practical experience with the framework. The developer prioritises changes that unlock key functionality or remove barriers to effective test creation. You want to ensure that each enhancement contributes meaningfully to the framework's usability.

Develop Enhancements

As enhancement requests flow to the development track, the developer faces the challenge of implementing improvements while maintaining framework stability. The two-track model shows its strength here, as the developer can make changes while the testing track continues exploration and validation. The developer focuses on quick, targeted enhancements that deliver immediate value, implementing them in order of priority as defined by the testing track. This rapid iteration between development and testing ensures that improvements are technically sound and practically beneficial. Each enhancement receives immediate validation from the testing track with the continuous testing rig.

The parallel work between defining and implementing enhancements creates a dynamic feedback loop where improvements can be tested and refined quickly. This approach allows the framework to evolve rapidly during the sprint while ensuring that each change improves the testing experience. The testing track can quickly validate implemented enhancements and provide immediate feedback, allowing for further refinement.

Build a Pack of Example Tests

While enhancements are implemented, the testing track continues, creating a core set of automated tests using the framework. This isn't just about testing the application - it's about building a library of reference test cases from which future framework users can learn. The testing track creates tests demonstrating different framework features and testing patterns. You're effectively building a practical guide to framework usage through real-world examples. These tests serve multiple purposes: they validate the framework's capabilities, provide examples for documentation, and form the beginning of the team's automated test suite.

The two tracks collaborate closely on setting up a comprehensive CI reporting environment, ensuring that framework health and test execution results are visible. This involves configuring dashboards that display test execution results, framework build status, and (possibly) code quality metrics. The reporting setup must clearly distinguish between framework issues and legitimate test failures, helping future users quickly identify the root cause of any problems. This infrastructure work is crucial for the framework's long-term maintainability and adoption. It'll provide clear visibility into both the framework's stability and the health of the tests built with it.

Remember, stability and reliability are everything. Without stability and reliability, people will quickly lose faith in your automation solution.

Chapter 9

Day 5 - Continuous Test Setup

The final day focuses on validating and assessing the sprint's success. You'll achieve this through running repetitive tests. The tester verifies recent enhancements and fixes. Then, we conduct baseline test execution runs to establish a good level of stability. The team reviews the original challenge that sparked the automation sprint, comparing their solution against initial goals. This assessment helps determine whether you've met your goals and identifies areas needing attention in future iterations.

Write the Core Set of Automated Tests Within the Framework

The testing track has already dedicated some time to developing a set of automated tests. Now, you need to start expanding this pack. We won't build a massive regression pack in just one day! This is more about adding a handful of new tests and then starting to plan and prioritise what you'll be working on over the coming months.

These tests you write today will not only continue to validate the framework's functionality but also start to add to the regression pack you need. Of course, they will provide yet more working examples for future framework users. This test set should cover various testing scenarios, from simple element interactions to interactions with more complex elements. We're effectively creating a reference implementation that showcases framework best practices and common testing patterns.

Creating this core test set reveals practical insights about framework usage that might not have been apparent during development. The testing track focuses on building tests utilising different framework features, creating diverse examples from which future users can learn. We might include tests that handle data-driven scenarios, cross-browser validations, error conditions, and complex user interactions. Special attention is given to test structure and organisation, establishing patterns other testers can follow when adding their tests.

You'll also check how easy it is to add tests. The easier you make it, the easier and quicker it will be to get other testers on board. Consider future enhancements that can simplify test development and creation.

It's about solidifying a baseline which others can use and learn from now.

Stability Execution Runs

Test stability is a massive key to successful test automation. Tests will fail because you haven't put enough effort into 'hardening' them. You MUST start diligently evaluating every failure. If a test fails because it's a flaky test, either discard it (for now) or put effort into enhancing/fixing the test. This is a habit you MUST get into. The instability of tests will kill all the good work you've put into this so far.

The stability execution phase involves running these core tests repeatedly to validate the framework's reliability under sustained use. Multiple execution runs help identify conditions that you haven't yet considered. The testing track monitors these runs for various metrics: execution time consistency, resource usage patterns, and any intermittent failures that might indicate test stability issues. This phase often reveals subtle issues that only become apparent under repeated execution, such as problems with test clean up, resource management, or state handling between tests.

Don't get disheartened by this phase. Developers fix and enhance the production code they release. You'll need to get into the habit of fixing and enhancing the automated tests you release. The trick is to be diligent and not to expect tests to magically start working on the next run. Either discard the test (for now) or fix it!

During the stability runs, particular attention should be paid to the framework's error handling and reporting capabilities. In this testing phase, you look for error conditions to verify that the framework provides clear, actionable feedback when things go wrong. This includes validating that test failures are correctly reported, error messages are meaningful, and debugging information is sufficient to diagnose issues quickly. The stability testing phase also helps verify that the framework's logging and reporting mechanisms scale appropriately under continuous use. This ensures that teams can effectively monitor and maintain their growing test suites.

So What Next?

You've completed the Five-Day Implementation sprint and are ready to implement "The Test Development Pipeline."

Or are you?

Before moving on, consider some of the outputs from today and the other sprint days. You should at least have:

▪ Prioritised list of framework fixes and enhancement requests

▪ Identified areas for enhancement that will improve usability

▪ Prioritised list of tests to write

At the start (the Preparation phase), you defined a "Challenge." You wanted to meet this "Challenge" objective with your short, medium, and long-term test automation goals. You have a core set of tests and have completed some stability runs. How is what you've built holding up against your original challenge?

You'll probably realise that you haven't achieved everything you wanted. It's human nature to expect a bit too much. However, you will have made significant progress. Looking back, it's usually easy to figure out the next steps. Typically, you'll see that you have two options:

1. Iterate with another 5 Day "Framework Implementation Sprint"

2. Start "The Test Development Pipeline" phase

Which of these you choose is usually dictated by answering this simple question:

"Is what we've built ready to be used by other testers?"

Yes, these other testers will need some education, but is your framework implementation stable and reliable enough for them to use? If you believe it is, move on to "The Test Development Pipeline" phase. If not, step back and plan another "Framework Implementation Sprint."

Chapter 10

The Test Development Pipeline

Your "Test Development Pipeline" is designed to wrap a consistent process around creating new tests and maintaining existing test cases. This consistent process should be followed by everyone adding new tests. The key is maintaining a regular execution cadence to ensure you have "stable and accurate automated tests at all times". You should be fixing or removing unstable or inaccurate tests immediately! This is the only way you'll deliver a single point of truth that team members can rely on and trust. Unstable or inaccurate test results kill confidence in test automation. There has to be total commitment to delivering quality new tests and maintaining the stability of existing tests.

Maintaining Quality and Stability

Maintaining quality and stability isn't difficult if you commit to defining and following a well thought out "Test Development Process". This process should address six key aspects:

1.  "Identifying and Developing New Tests"

- A Systematic approach to test expansion

2.  "Workflow, Quality Gates and Standards"

- Clear criteria for adding or modifying tests

3.  Continuous Test Review Cycles"

- Regular evaluation of test relevance and effectiveness

4.  "Health Metrics and Monitoring"

- Systematic tracking of automation effectiveness

5.  "Knowledge Management System"

- Structured approach to maintaining expertise

6.  "Update and Regression Strategy"

- Systematic approach to maintaining test assets

The Test Development Pipeline

A Stable Foundation for Growth

The "Test Development Pipeline" creates a stable foundation for test automation growth and maintenance. It starts by identifying the new tests we need and ensuring our automation keeps up with product changes. Regular reviews help us catch issues early, remove tests we don't need anymore, and keep our test suite lean and effective. It's important to think through workflows for developing and testing your tests too. Standards and quality checks ensure both framework developers and testers follow consistent practices.

By tracking key metrics and fostering knowledge sharing between teams, we build a healthy system instead of being cursed by a lack of maintenance and development effort. The Pipeline turns routine maintenance into a streamlined process where framework updates, data management, and test improvements are part of an embedded process. This structured approach keeps our automation stable and accurate. It's an approach that leads to confidence in your automated tests and, ultimately, confidence in the releases of your products.

In the following sections, we drill into these six key aspects, outlining processes to follow. We also provide checklists that will help you keep track.

Chapter 11

Identifying and Developing New Tests

Overview

Identifying and developing new automated tests requires a systematic, collaborative approach. You'll aim to align testing efforts with business needs and development priorities, balancing resources available against a never-ending list of potential tests that could be automated.

With regular analysis of upcoming features and application changes, teams can proactively identify opportunities for expanding test coverage. A structured prioritisation process will help you focus automation efforts on high-value scenarios. While also considering factors like execution frequency, reliability requirements, and maintenance costs.

Integrating test automation planning directly into sprint cycles can help develop and implement new tests in parallel with feature development. This integrated approach ensures that test automation evolves alongside the application, continuously delivering value to the business and customers.

Identifying New Tests for Automation

When the framework is in place and running, we'll want a process for identifying which manual tests should be prioritised for automation.

We can start by gathering test metrics from your existing manual test cycles - look at execution frequency, time taken, complexity, and failure rates. Tests running frequently (like smoke tests and critical path regression) should be high on the automation list. For example, if your team runs specific login flow tests daily, these are prime candidates.

Evaluate tests based on business risk and impact. Prioritise core functionality that could severely impact users or business operations. This might include payment processing, data security features, or key user workflows directly affecting revenue.

Consider test stability and predictability. Good automation candidates have consistent steps and expected results. Tests that require lots of human judgment (like UI aesthetic validation) or have highly variable outcomes are usually poor automation candidates. Additionally, look at the data requirements - tests with stable, manageable test data are easier to automate.

Analyse the technical feasibility of automation. Some elements to consider include:

▪ Is the application UI/API stable enough for reliable automation?

▪ Are there good, consistent element locators available?

▪ Does the test require a complex setup or external system integration?

▪ Is the required test data accessible programmatically?

▪ Are there any technical limitations that would make automation difficult?

Create a prioritisation matrix combining the following factors:

▪ High frequency + High business impact + High stability = Top priority

▪ High frequency + High business impact + Low stability = Medium priority

▪ Low frequency + Low business impact = Low priority or remain manual

Adding these as fields to your test management tool is usually not difficult. Along with a simple "Is the test automated?" field, this will give you the ability to filter 'Top priority' test cases that are 'not yet automated'. Then, add these to either a Sprint board or a Kanban board to track work and implementation.

Integration with Sprint Planning

When it comes to allocating resources and planning the work associated with developing automated tests, you have two main approaches:

1. SCRUM: planning work as part of an existing scrum team

2. KANBAN: organising work as part of a dedicated automation effort

The Scrum Approach

In many organisations, testers work as part of mainstream scrum teams, balancing their time between manual testing and automation work. These teams integrate automation work directly into their sprint planning sessions. They'll be identifying automation candidates during backlog refinement and story planning. Test automation tasks get story points like any other work, competing for sprint capacity alongside feature development and manual testing.

On the plus side, this approach ensures automation stays closely aligned with new feature development. On the downside, it can mean automation resource allocation remains a lower priority in the sprint, and automation moves slower than desired when sprint pressures mount.

The Kanban Approach

The alternative is to maintain dedicated automation resources that work in a Kanban flow. This way, you separate automation work from the main development sprints. This model works well when organisations have committed to a significant automation initiative and can dedicate full-time resources.

Using Kanban rather than sprints makes sense when it's easier to write automated tests against existing stable functionality. Sometimes, automation tasks vary significantly in size and complexity, making planning into fixed sprint cycles hard. The Kanban approach allows for a continuous flow of automation work, with framework developers and test creators pulling new tasks as capacity allows.

The key challenge with this model is getting senior management to agree to dedicated resources in the first place. You may also find challenges ensuring communication channels with the scrum team are maintained. You must ensure you stay informed about new features and changes that need automation coverage.

The Combined Approach

Often the biggest challenge is getting started with test automation. An approach that's well worth considering is starting out with Kanban. You might begin with a Kanban process utilising dedicated test automation resources. This helps make sure that you build out that initial suite of automated tests. Crucially, it allows you to prove the benefits of automation fast. This then strengthens your hand when it comes to requesting further resources later on.

Once the process, systems, framework, and initial automation suite are in place and running, you can switch back to the embedded Scrum approach. With the benefits of automation proven and the team's skills embedded, a switch back to the Scrum model is likely to be more successful. Arguing for significant resource allocation within a sprint is far easier when you can demonstrate benefits from earlier successes.

Collaborating with Framework Developers

As mentioned above, you may want to enhance the framework and add more complex methods/routines to support the implementation of particular automated tests. The key here is to maintain this focus on "A Two Track Development Model" (as discussed earlier).

When planning new automated tests, use both tracks of your development model to get the best results. Test designers focus on what needs testing and how to test it, while framework developers handle any technical upgrades required to support these new tests. This split lets each group focus on what they do best, making the whole process smoother and faster.

If testers need new tools or methods to test something properly, they work directly with the framework developers. The framework developers create these new features in a way that keeps things simple for testers to use while ensuring the code stays clean and efficient.

For example, if testers need to check complex data in a new part of the system, framework developers might create easy-to-use methods that hide the technical complexity. This teamwork means we can tackle more complex testing needs without slowing down or compromising quality.

Testers must focus on what needs testing, not how to build the framework. They should spend their time writing new tests, defining what data needs to be handled, and what results should be checked. Avoid getting caught up in complex coding tasks. Testers use their deep knowledge of the product to define test requirements and acceptance criteria.

When framework enhancements are needed, this transparent handoff to the framework team means testers can focus on designing practical tests. It's a more efficient approach that plays to everyone's strengths and helps avoid the common trap of testers getting bogged down in technical challenges they're not equipped to handle.

Conclusion

The successful identification and development of new automated tests depends on many things. One is maintaining a balanced, proactive approach that combines strategic planning with practical execution. Another is fostering close collaboration between teams, integrating automation planning into the development cycle, and maintaining clear prioritisation criteria.

Separating test development from framework development will always be a significant challenge. Just make sure that you remain focused on delivering maximum value with the resources you have at your disposal. Focusing on scenarios that provide meaningful validation of important functionality.

Chapter 12

Workflow, Quality Gates and Standards

Overview

A system of workflows, quality gates, and standards forms the foundation for maintaining high-quality test automation projects. This is about establishing consistent processes for test development and framework modifications and ensuring that all contributors follow established processes.

Test development rules and standardised templates will streamline the creation of new tests while maintaining consistency. Together, these elements create a quality-focused ecosystem that promotes sustainable test automation practices while maintaining the framework's long-term value to the organisation.

Workflow Definition

Your test development workflow should be designed to ensure high-quality working tests are developed and fed into the execution engine. This workflow should be defined around your toolset for managing source code control, executing tests, and reporting test results (e.g., Git, GitHub, and Allure). We outline a typical workflow below.

The Test Development Workflow

This workflow depends on one central source code repository that contains all the automated test cases for one Application. The main/master branch should be maintained as the stable/reliable source of test cases to run on schedules (this ensures that the execution machine is kept in a clean state). Pipelines within the repository for the main branch are configured to run different sets of test cases (e.g. different test suites).

The key to this reliable and consistent execution is keeping the test execution machines in a known good state so that they do NOT block execution runs. For this reason, the process outlined depends on only known stable tests being executed on schedules. All tests in development should be run manually so that the state of the test execution machines can be monitored while tests are “tested.”

Development branches should be created when new tests are required to increase reliability levels. Associated pipelines can be designed to execute these tests and "Test the tests" on the execution machines. These pipelines should ONLY ever be triggered manually. Testers should make sure that execution runs do NOT leave the execution machine in a 'hung' or 'unknown' state. Only when tests have been "tested" and proven to be reliable (e.g. clean up consistently after execution) should they be merged to the Main branch and executed with a pipeline on a schedule.

You should always have one 'main' execution branch, with a known good set of automated tests, that deliver a reliable set of test results that everyone can depend on. No un-tested tests or known un-stable tests should pollute the 'main' test set.

To ensure that all testers and developers conform to this, consider defining a set of test development rules and standards.

Test Development Rules

Developing a set of rules around your workflow will help your team ensure that tests run reliably and consistently with accurate test results reported. An example set of rules might be developed from the following:

▪ One repository dedicated to managing automated tests for ONE application

▪ The Main branch is where stable / reliable test cases will be run from

▪ The Main branch is protected and can only be updated with a pull request

▪ Tests from the Main branch must reliably clean up after the tests have been executed (it’s okay for the tests to fail, but the setup and clean up must be completed reliably!)

▪ Nothing gets committed to the main branch unless it’s been run 5 times daily for 3 days and proven stable.

▪ Scheduled executions are ONLY implemented for the main branch (nothing gets added to a schedule unless it’s thoroughly tested and on the stable main branch)

▪ Tests are developed on ‘Development’ branches with pipelines ONLY triggered manually

▪ After running tests from the development branch, users must ensure the execution machine is left clean (ready for scheduled runs to execute).

You may impose these rules through the configuration of your toolset (e.g., Git, GitHub, etc.). Or you may just publicise these rules and expect people to follow them implicitly.

Standards For Test Development

✓       Process

✓      Design

✓      Code

✓      Quality

Process

The following 'standard' test development process could be used to help ensure team members reliably develop stable automated tests. When a new automated test needs to be developed, the tester must walk through each of these steps:

▪ Create a development branch off the main branch

▪ Develop and test your tests ‘locally’

▪ Write a pipeline that executes your development tests on the execution machine (this pipeline should never be run on a schedule)

▪ Execute and test your tests on the ‘Test Execution Machine’ (tests should only be triggered manually, and execution Machines returned to a stable state on completion)

▪ Regression test existing automated test cases locally

Regression test existing automated test cases on the test execution machine *

▪ Once tests have been successfully regression tested, create a Pull Request (code should be reviewed and only merged when the reviewer has checked regression results)

▪ New pipelines should be configured to run new tests on schedule

▪ New tests and existing tests should be run on the execution machines and monitored for stability/reliability

▪ Reporting pipeline should be updated to include new tests (see “Adding Additional Test Runs” section below)

Note that this process allows for the development and testing of common code across tests and projects. It ensures that common code is regression tested against existing tests on a branch before being committed to the stable ‘main’ branch.

Design

Test Design standards help your team organise and structure your test code. As your test pack grows, it's very easy for everything to become disorganised. Providing guidance on things like naming conventions helps keep everything in place. Your team could define a set of design standards to include.

▪ Common naming conventions for test files and test cases

▪ Required level of documentation for each test

▪ Guidelines for test data management

▪ Standard approach for handling test prerequisites

▪ Rules for organising tests into logical suites

▪ Clear criteria for what make a good automated test

You can use the above as a starting point and add/develop as your process matures and your team's level of experience grows. What you may find useful, though, is defining these rules in a file contained within the test automation code base (e.g., the readme.md file held within your Git repository). Making this information easily accessible helps to reinforce the importance of conforming to standards.

Code

Coding standards have been around almost as long as people have been coding. Your organisation may already have a set of coding standards that developers follow. If this is the case, follow them. At a minimum, though, you'll want to consider:

▪ Consistent method naming patterns that reflect testing actions

▪ Proper use of framework utilities and helper functions

▪ Error handling and logging requirements

▪ Guidelines for comments and inline documentation

▪ Standards for handling waits and synchronisation

▪ Rules for managing test environment variables

Much of this will result from the framework's design and implementation. In this case, it's a case of formalising these points and ensuring they are communicated to anyone writing automated tests.

Quality

We're in the game of helping the teams we work with meet certain quality levels. To this extent, it would be remiss not to ensure that we set and conform to our own quality standards. It's worth considering and addressing the following points:

▪ Require code review process for new tests

▪ Ensure performance benchmarks for tests are met

▪ Set reliability criteria (tests must pass consistently)

▪ Set guidelines for test isolation and independence

▪ Ensure standards for cleanup after test execution are enforced

Again, much of this can be built into the workflow imposed by your team's toolset. However, other aspects (like test clean-up) are harder to nail down. Don't ignore them, though. Ultimately, if you're seeing issues in any of these areas, raise bugs and enhancement requests against your test automation code.

Ultimately, you should meet, if not exceed, the same standards your developers aim for!

Conclusion

The trick is not to create reams of documentation that your testers need to read and follow. Instead, provide examples with comments that can be used as templates to copy when a tester starts creating new tests. Templates save people time and help to encourage the following of the standards you've set.

Remember, the goal is not to create rules and documentation.

The goal is to create long-term test stability and reliability.

Chapter 13

Continuous Test Review Cycles

Overview

Regular Test Review Cycles are crucial for maintaining a high standard of quality and relevance across your automated testing suites. Regular evaluation of test relevance and effectiveness ensures that the test automation strategy aligns with the application's evolving requirements and business goals. This process helps identify obsolete tests and update existing ones. This is one of those tasks that isn't that exciting, but if left unchecked, it will result in a bigger job over time. It is best kept on top of with short, regular reviews.

Identify and Remove Redundant or Low-Value Tests

The importance of Continuous Test Review Cycles can be illustrated through several key benefits. Firstly, it helps avoid test suite bloat, where the number of tests grows over time, but their value does not increase proportionally. Test suite bloat will lead to increased execution time and increased maintenance costs. It may also decrease the signal-to-noise ratio, making it harder to identify real issues.

Secondly, implementing this practice ensures that tests remain robust and less flaky. Again, it provides reliable feedback to developers. By regularly reviewing tests, your team can refactor flaky tests or eliminate them if they no longer provide value. It's a way of adapting to your changing code base, ensuring that tests remain relevant and focused on the most critical areas of the application.

Teams should adopt several practices to effectively approach continuous test review cycles. A good starting point is to integrate these reviews into the development and testing workflows. Make them a regular part of sprint retrospectives or incorporate them into the definition of done for new features. Test coverage tools can be used to identify untested code paths and redundant tests. Test metrics, such as pass/fail rates, test execution times, and code coverage percentages, should be monitored to inform decisions on which tests to review and refactor.

Check for Tests that Could Be Consolidated or Simplified

During these Test Review Cycles, it's essential to identify opportunities to consolidate or simplify tests. Sometimes, a single test will give you coverage of multiple areas of functionality. Maintaining one test that provides extensive coverage should be easier than managing many different tests.

Test consolidation might involve combining multiple tests that have overlapping coverage or that could be more effectively tested as a single unit. This process helps in reducing redundancy and the overall number of tests. The benefit is that you'll see decreasing execution times and resource consumption. Similarly, simplifying tests means making them more straightforward to understand. This may involve reducing complexity, improving readability, and removing unnecessary steps or assertions that do not contribute to the test's primary objective.

Consolidating and simplifying tests can lead to a more streamlined automation suite. For example, if multiple tests are setting up similar environments or testing similar functionality with slight variations, they could be refactored. Refactored into a single parameterised test that covers all scenarios. This reduces duplication and makes it easier to identify the root cause when a test fails.

Simplifying tests can also improve the stability of the test suite by removing the flakiness associated with complex, multi-step tests that are brittle and prone to failure. Simplified tests are typically faster to run and easier to debug, which can significantly enhance the feedback loop for developers.

Regularly reviewing the test suite for consolidation and simplification opportunities requires discipline. You should encourage your team to practice good coding standards and refactoring techniques within their test code. Automated code analysis tools can assist in this process by flagging complex or duplicated code. Pair programming and peer reviews are also effective in identifying overly complex tests that can be broken down or merged. By committing to these practices, teams can maintain a lean, effective test automation suite. This ensures that each test is purposeful, clear, and efficient in verifying the application's behaviour.

Validate that Tests Still Align with Business Requirements

This isn't just about the health of your test suite, though. You need to be thinking bigger in terms of fostering a culture of collective ownership of the test suite. Developers, QA engineers, and other stakeholders should collaborate to maintain the quality and relevance of the test cases. Validating that tests still align with business objectives as they evolve is critical. There's a significant overhead in maintaining tests that serve no purpose. The software under test is updated to meet new market demands and regulatory changes or to incorporate feedback from users. These updates often result in changes to features and functionality that can render existing tests obsolete or inadequate. Therefore, it's essential to ensure that all stakeholders continuously review automated tests and update them to reflect the current business needs.

When tests are in sync with business requirements, they serve as a reliable measure of whether the software delivers the intended value to its users. This alignment helps prevent scenarios where development efforts pass all technical tests but fail to satisfy the business criteria, which could lead to costly rework or, worse, a product that does not meet market needs. Furthermore, maintaining this alignment helps prioritise testing efforts, focusing on critical business paths and risk areas. This all helps optimise the deployment of your testing resources.

A systematic approach is needed to maintain alignment with business requirements. This includes regular communication between business analysts, product owners, developers, and testers to review the existing test cases in the context of current business goals. Changes in business requirements should be reflected in sprint plans and documentation, ensuring that everyone involved has a clear and up-to-date understanding of what needs to be tested.

Conclusion

In conclusion, Continuous Test Review Cycles are vital to a robust test automation strategy. By regularly assessing and refining the test suite, teams can ensure that their automated tests are efficient, reliable, and aligned with the current state of the application and business needs. A deliberate approach to these cycles, integrated into the team's workflow and supported by the right tools and practices, will lead to a more maintainable test suite.

Chapter 14

Health Metrics and Monitoring

Overview

You'll want to monitor the health of both your automation framework and the processes wrapped around it. Health metrics and monitoring serve as the diagnostic backbone for assessing your automated testing processes' effectiveness, efficiency, and reliability. By collecting and analysing key performance indicators such as test execution times, failure rates, maintenance efforts, coverage metrics, and the prevalence of false results, you'll gain valuable insights into the state of your test suites. These metrics not only illuminate areas that require optimisation or attention but also help maintain the integrity and value of the test automation over time. Effective monitoring ensures that automated testing remains a robust, trustworthy asset in the continuous delivery pipeline, facilitating the rapid delivery of high-quality software.

Types of Metrics

Test Execution Time Trends

Tracking test execution time trends will help reveal how quickly and efficiently tests are being run. Over time, these trends can highlight slowdowns that may result from increased test suite complexity, less efficient code, or infrastructure issues. Monitoring these trends helps teams maintain swift feedback loops and address any bottlenecks promptly to ensure that testing remains a facilitator, not a hindrance, to the development process.

Failure Rate Patterns

Analysing failure rate patterns in automated tests is crucial for understanding the stability and reliability of the test suite and the application under test. A high or increasing failure rate could indicate unstable application code, flaky tests, or environmental issues. Observing these patterns over time helps teams identify problematic areas requiring attention. It's essential to distinguish between actual defects and test-related issues.

Maintenance Time per Test

Monitoring the maintenance time per test provides insights into the effort required to keep the test suite up-to-date and functioning correctly. This metric helps in assessing the cost-effectiveness of the test automation strategy. Tests that consistently require a high maintenance effort may need to be refactored or removed. Keeping maintenance time in check ensures that the test suite remains lean and manageable, contributing positively to the overall development velocity.

Coverage Metrics

Coverage metrics, such as code coverage or requirements coverage, measure how much the test suite exercises the software. These metrics are essential for identifying untested parts of the application and ensuring that the tests cover critical functionalities. High coverage metrics often correlate with higher software quality, but balancing the pursuit of coverage with the actual value delivered by each test is essential.

False Positive/Negative Rates

The rates of false positives (tests that fail without a corresponding defect in the application) and false negatives (tests that pass despite the presence of defects) are indicators of the effectiveness of a test automation suite. High rates of either can undermine trust in the testing process and lead to wasted time investigating non-issues or overlooking real problems. Keeping these rates low is essential for maintaining confidence in automated testing and ensuring that it serves its purpose of accurately detecting defects.

Framework Usage Statistics

Collecting statistics on the framework's usage, environments and tools within the test automation process can inform decisions about best practices, training needs, and resource allocation. Framework usage statistics can highlight the most commonly used features, the proficiency levels of team members with those features, and potential over-reliance on outdated or inefficient technologies. This data can drive continuous improvement in the test automation strategy, ensuring the team has the best tools for their needs.

Conclusion

In conclusion, investing effort in collating Health Metrics and Monitoring statistics helps with the continuous improvement and sustainability of test automation. By diligently tracking and analysing metrics such as test execution time trends, failure rate patterns, etc, organisations can fine-tune their testing processes to achieve optimal performance. These insights allow for informed decision-making, enabling teams to proactively address issues before they escalate. It'll help you refine your test strategy and make sure you invest wisely in your test automation infrastructure. A robust set of health metrics ultimately empowers teams to deliver quality software consistently, with confidence and efficiency, while continually aligning with evolving business goals and technological landscapes.

Chapter 15

Knowledge Management System

Overview

Even the best test automation setup is worth very little unless people know how to use it properly. Your Knowledge Management System will be key to user adoption. In short, if you're going to survive past the initial implementation, you need to start imparting knowledge. You'll want to think about a system that goes beyond simple documentation.

This system must encompass comprehensive training materials that help new team members quickly become productive with the framework, alongside regular knowledge transfer sessions that keep development and testing tracks aligned on framework capabilities. In addition to maintaining current documentation, you'll want practical examples that demonstrate framework usage. Detailed troubleshooting guides will help users overcome common roadblocks and save time. Together, the elements of your Knowledge Management System will create a sustainable support structure that maximises the framework's long-term value and promotes consistent testing practices.

Training Materials

You will need training materials! The key is striking a delicate balance between breadth and depth, providing high-level conceptual understanding and detailed practical guidance. These materials should start with framework fundamentals, explaining core concepts and basic usage patterns, before progressing to more advanced topics like custom test helpers and complex testing scenarios.

To build a practical training library, you'll want to include a mix of written documentation, video demonstrations, and hands-on exercises that give new team members practical experience with the framework. These materials should also cover framework architecture and design decisions, helping users understand how to use the framework and why it works the way it does.

The most effective training materials will incorporate real-world examples taken directly from your existing test suites. These examples demonstrate best practices, common patterns, and practical solutions to typical testing challenges. Rather than presenting isolated code snippets, you should walk through complete test implementations, explaining the thinking behind different approaches and highlighting potential pitfalls to avoid.

With a good combination of theory and practice, you'll help ensure that new team members can quickly become productive framework users while developing good testing habits.

Knowledge Transfer Sessions

Knowledge transfer sessions will serve as vital connectors between the development and testing tracks. You need to continue to foster collaboration beyond the initial sprint. Regular bi-weekly or monthly sessions provide a structured forum where both tracks can share insights, discuss framework updates, and address emerging challenges.

The development track uses these sessions to introduce new framework features or architectural changes. The testing track shares practical experiences and identifies areas where the framework might need enhancement. This ongoing dialogue ensures that framework evolution remains grounded in real testing needs while keeping all team members informed about new capabilities.

A typical session might include demonstrations of new framework features, discussions of recent testing challenges, and collaborative problem-solving for complex testing scenarios. Teams often find value in rotating session leadership between tracks, with testers sometimes leading deep dives into specific testing approaches. Developers explaining technical implementation details. The key to successful knowledge transfer sessions lies in their regularity and focus on practical, actionable information.

Session content should be recorded or documented for future reference. This will contribute to the team's growing knowledge base and help maintain continuity even as team composition changes over time.

Troubleshooting Guides

Good troubleshooting guides will save time and effort and make a good addition to your training material. Your team will always run into common issues and roadblocks. Troubleshooting guides will help with productivity and avoid disruptions to other team members.

You'll want to build these guides from real-world problems encountered during framework usage, documenting both the symptoms of each issue and resolution paths. Rather than simply listing solutions, comprehensive troubleshooting guides explain the underlying causes of common problems, helping users develop a deeper understanding of the framework. This approach allows users to resolve immediate issues and builds their capacity to diagnose and solve similar problems independently.

A well-structured troubleshooting guide organises issues into clear categories, such as setup problems, runtime errors, or test stability issues. Make it easy for users to locate relevant information quickly. Each entry should include specific error messages or behaviour patterns that help users confirm they're looking at the right issue. This should be followed by step-by-step resolution instructions and, where applicable, preventive measures to avoid similar problems in the future.

Also, include (or create separately) the details about debugging techniques specific to the framework. Covering such things as how to use logging features effectively or how to isolate test dependencies that might be causing failures.

Typically, you'll write this guide in a Wiki, but just make sure people are regularly making updates. You'll need to ensure you keep it relevant if you're going to ensure its effectiveness. Keep in mind that this is all about reducing the framework support overhead.

Clear Ownership and Escalation Paths

Consider defining ownership and escalation paths depending on your team's size and your automation framework's reach. Sometimes, this is implicitly understood in smaller teams, but it can be important to formally define it in larger setups.

It'll help ensure that framework issues are resolved efficiently without creating bottlenecks or confusion. The framework ownership model should define specific responsibilities for the development and testing tracks. Aim to outline who owns core framework maintenance, test helper development, continuous integration setup, and documentation updates. Each area should have a primary owner (perhaps even a backup).

Consider also defining explicit criteria for when issues should be escalated from the testing track to the development track. The escalation path should include clear communication channels and expected response times for different severity levels. If you're running a 24x7 operation with high dependencies on your automation framework, consider procedures for handling urgent issues. Formalised procedures will help minimise the impact on test execution or delivery timelines.

It's all about ensuring that framework users know precisely where to turn when they need support above and beyond what's covered in the documentation and troubleshooting guides.

Conclusion

Your Knowledge Management System can transform your test automation framework from an automation tool into a sustainable testing solution that grows with the organisation. You're aiming to create an environment where automation knowledge is preserved and shared. A systematic approach to knowledge management will help ensure that the framework remains accessible to new team members.

Build a living ecosystem of test automation knowledge, and you'll have something that supports both immediate testing needs and long-term framework sustainability.

Chapter 16

Update and Regression Strategy

Overview

As you might have realised by now, much of the focus must be on making your automation strategy sustainable. A large part of that sustainability will come from ensuring that the framework and its test assets remain reliable and effective over time. For this, you'll want to consider aspects around scheduling updates, testing framework changes, and test data maintenance. You'll also want to think about adapting to application interface changes and consistent regression testing of existing automated tests.

By methodically addressing each of these areas, your team can maintain the health of your automation framework. You want to avoid the gradual decay that often plagues test automation efforts. Addressing these points helps ensure that test automation is not a one-time effort but a living system requiring ongoing maintenance.

The more you depend on your test automation framework, the more you'll realise that it needs to be managed and maintained in a similar way to any other production system people depend on!

Scheduling Framework Updates

Framework updates will require a structured, predictable schedule. Aim to balance the need for new features and improvements against the stability requirements of existing test suites. Look to establish regular maintenance windows for different types of updates, categorising them based on their potential impact. Critical security patches and bug fixes might follow an expedited process, while feature enhancements and non-critical updates can be batched into scheduled releases. Don't forget to account for adequate testing time before changes are promoted to a main branch or formal automation runs, where they could affect other team members.

Version control practices play a crucial role in managing these updates, with clear branching strategies and merge procedures that maintain framework stability. Think about your team establishing a release calendar that aligns with their sprint cycles and testing needs. There is nothing worse than having major framework updates that coincide with critical testing phases or release deadlines.

Finally, consider designated periods for technical debt reduction and infrastructure improvements. Preventing the accumulation of outdated dependencies or obsolete code patterns that could hinder future maintenance. Prevention is always better than cure.

Process for Testing Framework Changes

You're in the testing domain, right? Well, there should be a process around testing framework changes! As your team becomes increasingly dependent on your automation solution, a thorough approach to testing changes will need to be considered. Each framework modification must pass through a defined test and validation phase.

This testing process should utilise the core set of automated tests developed during the initial sprint as regression tests. You'll need to ensure that framework changes don't inadvertently break existing functionality. Also, remember to keep an eye on usability and maintainability. And then, of course, there's performance testing. Your testing process should include performance validation to verify that framework changes don't negatively impact test execution times or resource usage.

Regular communication between the testing and development tracks during this process will help identify issues early. It's your responsibility to ensure that changes truly enhance the framework's value to its users. So, keep all these factors in mind when testing framework changes.

Strategy for Updating Test Data

Test data management requires a well-defined change management strategy, too. One that ensures automated tests have access to reliable, up-to-date data that accurately reflects real-world scenarios. Your approach here should address the maintenance of existing test data and the process of introducing new data.

Your team should establish clear procedures for data refresh cycles. Consider including aspects like how frequently test data should be updated, who is responsible for maintaining different data sets, and how to validate that updated data maintains the necessary testing coverage.

It's easy to forget that test data should be versioned too. If you're looking to recreate issues then test versioning and data versioning are both critical. Data versioning allows teams to roll back to previous data sets if needed and maintain consistency across different testing environments.

Your test data strategy should also account for different types of test data requirements. Consider static reference data as well as dynamic transaction data that changes frequently. You may be looking at tools that will automate the processes for data generation. The aim here being to reduce the manual overhead. Aim to reduce the manual overhead of data maintenance while ensuring consistency.

Then, there's the area of handling sensitive data. You'll need clear procedures for masking or anonymising production data when it's used for testing purposes. Add to this the provisioning of specialised data sets needed for specific testing scenarios, such as edge cases or error conditions.

Managing test data can often be as challenging as building your test automation system. Be sure to put the time and effort into this area, as the effectiveness of your automation framework will depend on it!

Plan for Handling Application Interface Changes

The stability of test automation suites is very much dependent on the stability of the user interface or API that you're testing. It can be hugely frustrating trying to keep up with UI and API changes when a project is in flux.

Having a plan for managing interface changes can help maintain automation framework stability and prevent test maintenance nightmares. Look to establish a proactive approach to detecting and responding to UI changes. Collaborate often and early with the development team to gain early awareness of upcoming interface modifications.

Where possible, consider implementing a locator strategy that minimises the impact of UI changes. This is often easier said than done! However, a conscious approach to identifying stable identifiers will help. Aim to maintain a centralised repository of element locators that can be quickly updated when changes occur. Aim to include automated checks that can identify potential interface changes early in the development cycle before they cause widespread test failures.

Consider folding element identification into unit tests. Aim to get developer buy-in and support. Developers could, for example, work on simple locator tests as part of their unit tests. Then, those developers will be alerted to changes that will break existing automated tests really early in the development life cycle.

This area will always be a significant challenge in your test automation journey. However, with some upfront thought, you can find ways to minimise the impact of this challenge.

Conclusion

You will need a maintenance strategy to deal with framework updates, test data management, and interface changes. The more effort you put in here, the better your chances of creating a sustainable foundation for test automation success.

An integrated approach will help ensure that framework components stay current and reliable while test data remains relevant and effective. You'll be able to respond quickly to application changes and maintain automation stability.

Ultimately, this is about protecting the organisation's investment in automation by preventing that gradual decay that often undermines long-term automation initiatives!

Chapter 17

The Automation Sprint Conclusion

Success in test automation requires more than technical expertise - it demands a structured approach combining collaboration, focus, and long-term commitment. The Two Track Development Model establishes a foundation by splitting this endeavour between testers and developers. Two tracks to bring two distinct skill sets to bear on a difficult challenge. This separation of concerns, coupled with close collaboration, enables teams to capitalise on the unique skills of testers and developers.

The Five-Day Automation Sprint aims to bring the focus of testers and developers together to crack the development of this automation framework in one main sprint. By breaking down the complex task of creating a test automation framework into focused daily objectives, teams can move from initial requirements to a working framework quickly. It is a challenging time scale but achievable when the right skills and focus are committed.

The structured approach to a dedicated sprint helps ensure that both tracks remain aligned while progressing continuously. Each day builds upon the accomplishments of the previous one. This intensive but focused effort helps teams avoid the common pitfall of framework development that drags on indefinitely without delivering tangible results.

With the framework in place, attention turns to automated test development. The Test Development Pipeline addresses the challenge of maintaining and evolving the automation framework beyond its initial implementation, as well as the challenge of building new automation tests on a regular cadence.

It's about getting the systems and processes in place so that teams can ensure their automation investment continues to deliver value over time. A structured approach to framework maintenance helps prevent the gradual decay that often undermines automation initiatives. It ensures that your framework remains reliable and effective as applications evolve and team members change.

Together, these three pillars - the Two Track Model, the Automation Sprint, and the Test Development Pipeline - provide a complete framework for successful test automation implementation. By following this structured approach, teams can avoid common automation pitfalls while building and maintaining an effective test automation solution. A solution that supports your quality objectives. Success lies in following these practices individually and implementing them as an integrated system. A system that promotes continuous improvement and sustainable growth.

The ultimate measure of success in test automation isn't just the creation of automated tests - it's the establishment of a sustainable automation practice. Establish consistent practices, and you'll deliver value as applications evolve and business needs change. Get this right, and you'll achieve your organisation's quality objectives.

Appendices

Preparation Check List

Writing The Framework Requirements

Writing The Framework Specification

01

Preparation Check List

Goal

☐        What problem are you hoping to fix

☐        What results are you hoping to achieve

☐        Have you tried this before

Suitability

☐        Types of applications

☐        Proof of concepts

☐        Commonality between applications

Team

☐        Experience/skills

☐        Coding

☐        Test definition/selection

☐        Developers and Testers?

Organisational

☐        Team structure

☐        Resource allocation

☐        MIS

Technical

☐        Test environments

☐        Test data

☐        Data driven

☐        Test accounts

☐        Coding languages

Context

☐        Unit testing

☐        Api testing

☐        Module testing

Setup

☐        Source code control

☐        CI/CD

☐        Test specification

☐        Test management

☐        Virtual environments

☐        Testing tests

☐        Categories (sanity, core, full)

Reporting

☐        Test results

☐        Aggregation

☐        Dashboards

☐        Report distribution

02

Writing The Framework Requirements

Below you'll find a short example framework requirements document. This doesn't have to be a massive tomb of text. Just a list of what's needed, what it needs to be able to test and the environment within which it should exist.

Test Automation Framework Requirements Document

Stakeholder Analysis: Through discussions with the development team, QA engineers, and product owners, we've identified key requirements including the need for rapid test creation, robust reporting, and integration with existing CI/CD pipelines. The DevOps team specifically highlighted the importance of containerized execution, while QA emphasized the need for data-driven testing capabilities and easy maintenance of test scripts.

Current Solution Assessment: Our evaluation of existing testing approaches revealed manual testing processes for regression, limited API testing using Postman collections, and basic UI testing using outdated Selenium scripts. Industry research shows Playwright emerging as a preferred choice for modern web applications, with strong TypeScript support and built-in cross-browser capabilities that align with our needs.

Technical Landscape: The application under test is a React-based web platform using Material UI components, REST APIs, and OAuth2 authentication. It's deployed across four environments (DEV, QA, STAGING, PROD) using Azure DevOps pipelines. Current testing tools include Jest for unit testing and Cypress for limited component testing, with test results stored in Azure Test Plans.

Framework Structure: The proposed automation framework will be built on an open source platform that supports TypeScript. We will stick to implementing the proven Page Object Model pattern. Core features include environment configuration management, reusable test utilities, custom commands for common operations, and integration with existing CI/CD pipelines. The framework will support parallel execution and cross-browser testing.

Test Scenarios and User Stories: Key testing requirements include: "As a QA engineer, I need to create data-driven tests efficiently", "As a developer, I need to execute tests locally with minimal setup", "As a DevOps engineer, I need test results integrated into build pipelines", and "As a product owner, I need clear test reports showing feature coverage and stability."

Priority Matrix: Must-have features include basic test execution, reporting, and CI/CD integration. Should-have features include parallel execution and cross-browser support. Could-have features include visual testing and performance metrics. Won't-have features include mobile testing and load testing capabilities in the initial release.

Validation Plan: The framework development will follow an iterative approach with regular reviews. Initial focus will be on creating a proof of concept testing critical user journeys, followed by expanding test coverage and implementing advanced features. Success criteria include reduced regression testing time, improved test reliability, and increased test coverage.

Some, including me, find it useful to create lists of bullet points. If you go with bullet points it does tend to make it easier to tick off which requirements have been addressed in the specifications document you write.

03

Writing The Framework Specification

An example Framework Specification is provided below.

Test Automation Framework Specification: Playwright

The framework should be built using TypeScript and Playwright, implementing the Page Object Model design pattern for improved maintainability and reusability. It should support parallel test execution across multiple browsers (Chrome, Firefox, Safari) and include configuration management for different environments (DEV, QA, STAGING, PROD) using environment variables.

The framework must include a robust reporting mechanism that generates detailed HTML reports showing test execution results, including screenshots for failed tests, execution time, and error logs. Test data management should be implemented using external JSON/CSV files, with support for data-driven testing and the ability to maintain separate test data sets for different environments.

Authentication handling should be implemented efficiently, with the ability to maintain and reuse authentication states across test sessions. The framework should include retry mechanisms for flaky tests and automatic waits for dynamic elements, reducing test flakiness. Custom commands and utilities should be created for commonly used operations to reduce code duplication.

A well-structured logging system should be implemented using Winston or similar logging libraries, with different log levels (INFO, DEBUG, ERROR) and the ability to write logs to both console and files. The framework should integrate with CI/CD pipelines (specifically Jenkins) and include scripts for both local execution and pipeline runs.

Outline documentation must be provided that covers setup instructions, coding standards, best practices for writing tests, and examples of commonly used patterns. The framework should support pre-commit hooks for code formatting (using Prettier) and linting (using ESLint), maintaining code quality standards. Test execution should be configurable to run specific suites, individual tests, or all tests, with support for test tagging and filtering.

Then you'll want to think about specific parts of your "application under test" that will need to be automated. Let's take our React JS Grid that needs to be tested. We should include in our specification details for a class that will help us interact with this grid. An example follows:

The framework must include a specialized Grid testing utility class that handles common grid operations and verifications. This utility should support dynamic locator strategies for grid cells, headers, and rows using data-tested attributes. Grid interactions should include the ability to sort columns, filter data, verify cell contents, validate row counts, and handle pagination controls. The framework should implement methods to extract grid data into structured formats (arrays/objects) for comprehensive data comparisons, including the ability to verify sorted states and filtered results.

Key grid testing capabilities should include: column header verification, data sorting validation (ascending/descending), filter application and verification across multiple columns, pagination navigation and state verification, dynamic row counting, cell content validation including formatted values (dates, currencies, numbers), and grid state persistence verification. The framework should also handle grid loading states and implement smart waits for grid data updates after user interactions.

The grid testing implementation must support virtualized scrolling scenarios where not all rows are rendered in the DOM simultaneously. Methods should be provided to programmatically scroll the grid and verify data integrity during scrolling operations. Performance testing utilities should be included to measure and verify grid rendering times and response times for operations like sorting and filtering. Special consideration should be given to accessibility testing of the grid component, including verification of ARIA attributes and keyboard navigation support.

Again, you may find it more useful to provide a bullet list of functionality that should be implemented in your specification document.