Test Management Blog

It’s well known that keeping track of results from testcases against builds, configurations, environments, etc can become quite difficult to manage. It’s one of the reasons we usually implement test management tools. These tools enable us to log our results against these different aspects of our testing and then quickly produce traceability reports to show our coverage. It’s just that the permutations and combinations can become quite difficult to track, even with the right tools.

If we have just 1 testcase, which we need to run against 2 platforms, 2 operating systems along with three configurations relevant to our application we already have 12 permutations. That’s 12 permutations for just 1 testcase. And with this example we’re not even considering different versions of the application being checked. This would add even more permutations.

The whole thing can quickly get out of hand. Certainly this will get out of hand if you’re using Excel and don’t have some sort of test management system in place to help. It also helps if we break this down and focus on the relationship between just three variables

1. version of application
2. configurations
3. tests

Where the configuration entity might relate to different operating systems, platforms, environments, etc. So we define one record for each configuration where a configuration could cover Windows XP, Intel, etc. In this way we can manage the traceability matrix a little easier.

We can see this in practice with the test management tool QAComplete in this video. Here we look at how QAComplete deals with these different aspects in practice. We see in this video how QAComplete manages the relationship between sets, configurations and versions of the application.

The benefits of this approach are threefold..

1. you can view a configuration and see all past results against a configuration.
2. you can view a test case and see each configuration it was run against, along with the result.
3. you can see which tests have been run for a particular version of the application against a particular configuration

The key here is to ensure that you define the configurations in a meaningful way. If you get this right (perhaps using custom fields for tracking different aspects like Os, platform, etc) then your reporting and traceability will deliver the information you need.

And this is the crux of this topic. A lot depends on the information you need to get out after the testing is complete. The ways in which the different tools track and report on this information differs. As a consequence the reports your chosen test management tool delivers may or may not meet your needs.

Re-running tests to check that bugs are fixed is an essential part of the test management process. Teams grow, the amount of data generated by completed testcases increases and the number of raised defects expands too. As a result it becomes more difficult to keep track of the retesting. There are several different ways to approach this.

Firstly we can identify all failed cases from a previous cycle and just re run those testcases. This approach has one serious flaw in that defects raised outside of the QA effort don’t then get retested. Having said that this is the simplest approach to execute. We talk more about this approach in practice in this managing retests blog post.

The second approach is based on finding all defects that are in a resolved state and making sure they have been tested before they are closed out. This is usually covered by having defect status values like ‘fixed’ and ‘verified’. Where the developer moves a defect to a fixed status. The QA engineer then pulls out all the bugs that are fixed and is responsible for checking the fixes. Once checked the state is moved to verified. The problem with this approach is that it can be difficult for the QA engineer to identify tests that are relevant.

This is where a test management system with good capabilities for managing traceability comes in to play. Whilst maintaining traceability can seem like a significant overhead at the start it’s situations like this where that overhead pays off significantly. With good traceability in place we can identify the following:

1. From defects in a fixed state we can see which tests we ran that resulted in the defect being raised. If you have a significant amount defects to verify then identifying the relevant testcases can be a time consuming manual process.

2. where we have defects in a fixed state and there is no link to a testcase we know we have to write the testcase before we can verify the defect fix. This is a really important step in the process because it enables us to build a regression pack that can be used to verify futures builds of the application.

A good test management tool should support both of these approaches. In the following video we look at how QAComplete can be used to identify all defects that are fixed and then identify relevant testcases.

Retesting is an absolutely essential part of the test management process. Once the volume of data starts increasing, having a good system in place to identify and track of retests ensures that no software is shipped with bugs that haven’t really been fixed. Key to this though is understanding the way to filter and extract the right data from your test management tool.

In many cases we’ll need our test management tool to aggregate and report on results from various perspectives. For example we may need to aggregate across different projects, application modules, runs or cycles. Whilst this sounds simple in theory, it’s not always so simple in practice. Much depends on how you structure things within your test management tool and how well that tool can cope with aggregating results.

Another area of complexity can develop when you want to see what the last run status values are for a group of tests across different releases or builds of a product. We argue against The Aggregated Test Management Results Report in this post. However there are times when this can be useful. Perhaps you just need to get a feel for the status of all results across a series of releases.

This sort of information is never quite as simple to gather as you may expect. Complexities arise around how you determine which records should be included in your report. If you’re not grouping by the results based on a single release then what are you grouping them by? A cycle, functional area, date, etc? All of these could be considered valid groupings in certain scenarios. It’s important though not to forget that it can mi-represent the quality of your product at a particular point in time.

The exact quality of an application can only ever be related to a specific build of the product. If you’re aggregating across different builds then, ultimately, you are trying to report on a quality status of a product/build that has never existed.

Either way sometimes it is useful to create statistics based on results aggregated by some other category. Within QAComplete we have a number of options out of the box that we can employ.

1. Within the library we can group by a folder structure. When we view testcases in that folder structure we can see the last run status for all records. This then give us an indication of quality aggregated by functional area across different releases for the last run testcases.

2. On the library test management dashboard we can view the last run status of ALL testcases (and perhaps apply a filter based on functional area or some other grouping). Then, for example, when we drill down on this report we’ll see last run statuses where we had failures across all releases. This can be useful for identifying retests for subsequent builds.

The limitation with QAComplete is that this is all reported based around the ‘last run status’. Usually this is enough but if you need to aggregate by some other category then this becomes more difficult. For example if you define a cycle which covers more than one release/build this concept isn’t supported within QAComplete. In addition to this there is no capability to report across projects. To create cross project statistics requires the creation of custom reports which may mean getting your hands dirty with crystal reports and SQL.

Ultimately with test management we’d argue that you have to be careful with aggregated reports. Especially where you are grouping results across different builds/iterations of the product. This type of aggregation can give a misleading indication of product status. Having said that, often it’s good to see which testcases failed on old builds of the application so that you can create sets of retest for subsequent release to validate fixes and ensure failed testcases are now passing. The subject of identifying testcases for retesting conveniently being the next topic in this series of test management complexity blog posts.