Test Management Blog

All test management tools have the concept of assigning testcases. Some provide more capabilities than others. Certain tools provide just about every capability you can imagine but do this with a level of complexity which can be quite difficult to manage. Others have limitations that may or may not impact your ability to implement the process you need. In this article we look in detail at how your test management tool implements assignment at the set, case and step level.

Key points you should consider in regards to how a test management tool implements the concepts of assignment are:

1. At which level do I want to carry out assignment (e.g. step, case and set).
2. Do I need to be able to define different assignments of individual test steps within a case.
3. Do I need to be able to define assignment of individual testcases within a test set.
4. At run time how does assignment of the same test set to different configurations work.
5. Can assignment of these different components be changed at run time.

In the following example we look at how QAComplete and ALMComplete implement the concept of assigning testcases and sets. Cases are contained within a Library where we carry out the development and store our testcases ready for use in a set. Each set will contain multiple testcases and can then be executed against a release and a configuration. It should be noted that assignment can not be set at the step level or for test cases within a set. All the steps within a testcase are assigned at the testcase level. And all the tests within a set are assigned at the set level. With the test management tool QAComplete, default fields include both owner and assigned to fields for tracking assignment.

Both cases and sets have an owner (the person who created the record) and the person it is assigned to (the person developing or executing). Within the library individual tests may be assigned to someone. The owner may be the person responsible for maintaining the set and the assigned to the person responsible for executing the set. If you want to have different people run the same set against different configurations or releases then you have to change the assignment prior to each run of the set.

The concepts around assiging testcases to a tester sound simple. In practice though there are many ways to approach this and there are many ways the different tools provide this capability. The key is understanding these different approaches. Then making sure you select and set up your test management tool to cover the right process flow for you and your team.

Over the next seven weeks we’re going to take a look in detail at 7 complex aspects of test management. We’re going to look at how we can address these complexities with some of the common tools available today. In doing so we’ll explore some of the weaknesses and strengths of popular tools and see how best to set them up to overcome these issues. In short we’re looking to help guide you on the best way to implement your chosen test management tool so that it better supports your process.

In this series of weekly blog posts we’re going to examine the following…

1. Assignment

At some point in the process we need to assign a test to a tester. The question is are you looking to assign it at the step, case or set level? Do you need to allocate different tests within a set to different qa engineers? How do you track the difference between assignment for writing and executing the testcase? All of these points complicate a seemingly simple concept. When you look at the different ways different tools support this capability you’ll find it’s not quite as straightforward as you might first think.

2. Version Control

We version control the code we release. In many cases we need to be controlling versions of the test cases we  write and execute too. When working with version control you need to consider if the replication of different versions impacts already executed tests. You need to understand if you need to be able to choose the different versions you use at execution time. And finally if you can compare the different versions easily.

3. Parametrisation

Writing similar testcases repetitively becomes tedious very quickly. It’s open to mistakes and is also a poor use of the QA engineers time. To help address this some tools provide the ability to parameterise test cases. You write one testcase then scale up with data that creates many permutations. It’s a nice concept but one that’s difficult to execute effectively. That difficulty sometimes being compounded by the way in which the tools implement this feature.

4. Libraries

The concept of having a library of reusable test cases is as old as the discipline of software testing itself. How your test management tool allows you to manage this library makes a big difference in your capability to implement your QA process. Can updates in the library be replicated to testcases already assigned? Can you replicate updates across projects? Is the version control on the library linked into the executed testcases? All points worth considering when you decide how best to administer your tests library.

5. Result Aggregation

The software products that we’re working on are complex. The complexity of these products needs to be modeled within our test setup to a degree. Core modules of the product we’re QA’ing will have a set of core testcases. Specific modules, specific testcases. When modeling this setup within your test management tool there comes a point where your ability to report on results becomes more complex. Can you aggregate results to deliver the high level reports you need for your clients?

6. Retesting

Tests fail. Bugs get raised. It’s a clear cut part of the process. What’s not quite so clear cut is how we approach retesting of the fixed bugs. Do we delve into our defect tracking tool and look for all bugs that are fixed? Then write testcases for each fixed defect? Or do we search for failed test cases and then re run these? Sounds simple but pulling out that information from your test management tool isn’t always that easy.

7. Configurations and Releases

Every test is run against a release of the product and a specific configuration of the product. Sounds simple. Then you start looking a daily builds, different platforms and operating systems to run the tests on. Before you know it a these few variables start to result in thousands of permutations. Keeping track of this is absolutely key to your test process. How easy is this to track? It largely depends on the capabilities of the tools you use.

Most of these points are core processes that we all follow as a matter of course. There’s more to most of them than first meets the eye though. When we look to support that process with a  tool we can find that the process the tool imposes doesn’t quite match our requirements. This is the reason it’s important to examine these aspects prior to implementing a test management tool. Or at least understand the consequences of bending your process to meet the approach your chosen tool imposes. In this upcoming series we’ll examine all of these points in detail and help you find your way through these test management complexities.

Successful test management is about improving processes, technology and teams. However, if you don’t keep your eye on the end goal it’s easy to spend time and effort improving these areas yet see little difference in the quality of products you’re working on. And that’s the key. Any improvement efforts need to be targeted towards delivering an improvement in the quality of the products you are testing. Too many projects fail to deliver those improvements because they lose sight of this. The following 7 rules will help you focus on delivering a successful test management solution.

Rule 1: Focus on the Product Quality Goal

This is all ultimately about improving product quality. It’s not about technology. Improving tracking, reporting and productivity are all important but they’re not the main reasons we should be looking to implement a test management solution. If we see this as an opportunity to improve quality this enables QA teams to look for solutions that help us find more defects and find them earlier. If we look at this as an opportunity to improve tracking, productivity etc we may well achieve this but we may well miss the key point. Which is that we’re looking to improve the quality of the products we’re working on. Keep this at the front of your mind whilst you’re working to improve processes. If you do you’re far more likely to implement the right solutions in the manner that best works for your team.

Rule 2: Executive Involvement

Executives tend not to find software testing very exciting. They see it as a necessary evil. It gobbles up resources and money. And at the end doesn’t deliver a tangible product that can generate revenue. Only those executives that have been bitten by a poor product release (usually associated with negative press for your company) seem to truly grasp the importance of our role in the product development process. Yet the executives are usually the ones that will be authorising any significant changes to your processes and sanctioning the purchase of any tools. Executives also have a big influence on the empowerment of the QA team in improving their processes. Just two or three knock backs from a couple of executives can result in a QA team giving up on any future desire to improve and enhance their working practices.

Rule 3: Understand your Business

Many teams expect to implement a successful test management process without even understanding their business processes. We execute tests. That’s our business process. How difficult can it be to find a system that helps support that? It’s never that simple in real life. Over years the QA team will have developed processes (formal or informal) for interacting with developers, customers, end users, technical authors, etc. All of these interactions require some degree of process. If you don’t understand these processes how can you expect to implement a test management solution that will support them?

Rule 4: Look for the Benefits

Many QA teams select a solution based on the features it provides rather than the benefits it will deliver. Defining your business processes is important. So is understanding what benefits your improvements will deliver. You could be implementing a system that provides traceability between tests executed and requirements. This is only going to be a benefit if you have good requirements definition in the first place.

Rule 5: Support

You are not going to implement a solution successfully if your staff can’t support it. You should consider selecting a solution based on a technology platform that your IT department is familiar with. If the solution you want doesn’t fit with your technology platform then consider going with a SaaS solution. Next ensure that you have QA staff who know the system inside out. You’ll need staff that can support others in the team when it’s needed. Blind testers following blind testers is not a good situation. Successful test management is about having people in your team that understand why and how to use the solutions you implement.

Rule 6: Little Steps

Don’t go for the big bang implementation approach. Deploy your test management system as a series of small projects. Implementing change, especially in a test team, is difficult. It takes time for QA engineers to adjust, to learn and for them to appreciate the benefits. Smaller projects also increase the chances of success because your team can see the incremental improvements and start to feel the benefits sooner. These little successes build momentum. Momentum builds enthusiasm. Once you’ve got momentum and enthusiasm you’ve got a powerful combination for improving your QA processes.

Rule 7: Evaluation

Commitment is key. Success comes with improvements over the long term. Having a group of people that are responsible for a successful test management setup is essential. This group should be responsible for collating feedback, evaluating and maintaining the system. They should engender a belief that the system needs continuous adjustment and improvement. Not just a set it up and leave it for the next 10 years approach (before you have another mad spurt of effort to improve things).

Teams that implement successful test management systems know that it’s all about improving the quality of the software that you are testing. That means focusing your processes, your technology and your testers on the goal of improving that quality. Some get caught up in implementing the latest sparkly technology. This is usually in an attempt to just improve process, moral or some other relevant part of the system. It’s less likely to succeed. Successful test management is all about keeping the ultimate goal at the front of your mind.

Retesting is a key part of the test management process. However, there is a right and a wrong way to approach this. You run some testcases as part of one cycle. Some fail and you look to run them again as part of the next cycle. When they pass the temptation is to go back to the original testcase record and change the result to a pass. This is especially true where there was just a small fix or config change required to resolve the issue. This is the wrong way to approach this.

It’s imperative as part of the test management cycle to make sure each execution is tracked and recorded independently. The scenario is this.

1. You execute against a specific build and configuration.
2. You record a result of fail.

Note that at this point the failed result is logged against this specific build and configuration.

3. You get the development team to fix the bug.
4. You rerun the test and record a pass.

At point 3 you have a fix. You have a change in the application that resolved an issue. When you rerun at point 4 you are testing something different. This is important! Even if the change to resolve the issue was a minor configuration change this constitutes a different release/config/build. It’s imperative that this difference is tracked. Just changing the previous result to a pass is not the right approach. In this scenario you’re effectivly recording the pass against the wrong release/config/build. The key concept to grasp here is that…

Good test management practice dictates that each time you run the testcase you record the exact release, build/iteration and configuration that you are testing against.

If a developer has made a change, however minor, then any future retests need to be tracked as separate result records. So this presents the question ‘how do we track this in real life?’ The following example uses the test management application QAComplete to demonstrate this principal.

The steps we followed in this video example are…

1. Identify the failed testcases you need to rerun
2. Create a filter based on the ‘last run status’
3. Create a set for the next release
4. Included testcases in the set which are identified by the filter

With this approach we have a set specifically to retest all of your failed testcases from the first cycle of testing. This can be considered good test management practice because we’re always tracking the right results against the right versions, builds and configurations of the software.

There are a number of core test management activities that you will be performing as part of any QA process.  What we cover here is good software testing practice and should form the bedrock of any QA teams setup.  The activities we’ll look at are :

Test Management Activities

1. Developing testcases in a library.
2. Defining releases and configurations that we’ll use during the execution.
3. Grouping into sets ready for execution.
4. Execution.

In this discussion we’re going to look specifically at carrying out those activities with QAComplete. With QAComplete we’ll see how we can streamline this approach. We’ll see how this can improve on the process that we’d normally see with an excel based approach.

The first part of this process is developing the tests. We’re focusing on the test management activity of writing good testcases. At this point we’re not thinking at all about the execution phase. It’s important to keep these two test management activities completely separate. If we don’t the temptation is to write what we see whilst we are using the application. So the case content is driven by what the application does. This of course defeats the whole point of testing. We’re defining what we expect to see before we check that the application conforms to our expectations.

The second part is to define the releases and configurations that we expect to run our testcases against. There is no point running any checks if you are not recording or tracking what you are running against. The first aspects we need to track is the version, build or iteration of the product. The second aspect is the configuration of the system or software (e.g. the operating system we’re running the product on). So the test management activity we need to concern ourselves with here is identifying the release and configuration combinations we need to work with.

Once we’ve written our testcases and defined releases/configurations we should look at grouping our tests in to sets. This helps on several levels. Firstly a group of similar testcases is easier to manage and simpler to maintain. At execution time a group of similar tests can be quicker to run. Whilst there are no hard and fast rules on how to go about grouping the following grouping approaches could be considered:

i. type of test (e.g. integration, system, etc)
ii. feature or module
iii. process
iv. scenario
v. complexity

Clearly there are many ways to group here. However, the most important point to consider is how you would like to see your reports structured once you are finished. If you need to report on progress based on product features then grouping by feature may be a good approach.

With sets created we can then move on to the execution. At run time the test management activities we’ll be focusing on will be selecting the set to run and specifying which configuration/release we’ll run the set against. Once we’ve identified these attributes we can step though each testcase and record the results.

In principal these 4 core test management activities are quite simple to grasp. As always though the devil is in the detail. Developing and writing the testcases can be considered an art form and requires a good degree of skill. Identifying releases and configurations, whilst easy enough, is complicated massively by the combinations and permutations. Again skill and experience come in to play here. Grouping into sets is probably the easiest of the tasks here. Just keep in mind your reporting requirements. And then finally we come on to the execution. If we’ve completed the other tasks well this should be pretty straightforward. With all of this keep in mind that these core test management activities will remain constant and will provide a good framework to help you manage the underlying complexities.