Test Management Blog

This is a follow up to the previous post on Approaches to User Training for Test Management Systems. We thought we’d take a look at one of the most important reasons for training. That is increasing user adoption. Getting your QA team to use your test management system, and use it correctly, is key to your success. Increasing user adoption with training does, however, depend on a few significant elements.

It’s about the process – A lot of test management training is based around the functionality of the system. We track this piece of data, at this point in the process, in this field. We record the results in this field and log the environment tested against here, etc, etc. The reason we do this is to support the test process that the organisation has adopted. For example we track the phase of testing that a test is run in so that we see where in the development lifecycle we’re capturing issues.

As such your QA team will need to understand not only the business process but also how the system maps to that process. A significant part of the training should therefore be focused on educating people about the process. This way they can see how the system is configured to enhance that test management process.

It’s a sales task – Your QA team need to know that it’s worth investing their time and effort on this. Any training needs to sell the benefits and advantages of using the test management solution. That’s not just the benefits to the organistaion but the benefits to the users too. On the other side of the coin training also needs to overcome the objections which usually surround the implementation of new systems. There will always be someone who sees a new system as a solution for tracking them and monitoring what they are doing. Your sales pitch needs to overcome these objections and demonstrate the benefits.

It’s a two way street – During the requirments capture and implementation phases you can be sure that you won’t have resolved every issue and captured every important requirement. The training phase is one of the best opportunities you’ll have to get open and honest feedback from users. People are typically more open to raising points of concern and issues, one on one, with a trainer. People aren’t so keen on dealing with some outside consultant who’s responsible for implementing the new test management system. So the training phase is an ideal forum to capture that feedback and feed it into immediate enhancements or future improvement cycles.

What we are trying to get across here is that we should look at training as a forum to increase user adoption. If we approach it from that angle then we can see other significant factors that can increase the overall success. When you see training from this perspective then you can appreciate the importance in the context of the overall success of your test management system implementation.

A large part of any new test management system implementation is training. Delivery of effective training is key to the uptake and correct usage of any system you put in place. There are several options or approaches to training that should be considered. Each approach having it’s own advantages and disadvantages.

Training Mentors
Typically delivered internally this type of training is usually based in the real world. It involves having your trainer work side by side with the students in a live environment. The aim is to have the mentor(s) help staff by walking them through the system as they work normally. The big advantage here is that the students learning is reinforced by working in a familiar environment. The down side is that the effective mentor to student ratio needs to be quite low (e.g. one mentor to 1 or 2 students) for this to be effective. This of course can add to the cost or time scales if you have the recommended low ratio. Alternativley if you have many students to one mentor the teaching tends to get diluted.

Training Delivered Internally
Here the company implementing the test management system uses their own internal trainers to teach the students. The big advantage of this is that the internal trainer usually knows the companies internal processes and systems very well. In this way he or she can help cover scenarios that are important to company with depth and authority.

Usually to implement this approach an external trainer will train the internal trainer. The down side to this tends to be that many companies don’t have a dedicated internal training team. So they pass the job onto someone in the organisation that has little experience training. It also relies on the internal trainer understanding the test management tools well.

Training Delivered Externally
In this scenario the vendor supplying the test management tool offers a training package along side the purchase of the software. Here you can expect the vendor to have experienced trainers that understand the product inside out. Conversely they are unlikely to understand all of the companies internal processes and systems. As such they are restricted in the extent to which they can relate the training to the companies real business practices.

Remote Training
Many enterprise level solutions these days will come with some sort of computer based training package. This may be standalone video tutorials or training guides. Clearly the down side to this is that it relies on the student putting aside time to study and learn. Difficult when you have a day job demanding your time.

A good compromise on the remote training front can be delivery of training remotely by an external trainer. This type of training is usually carried out with tools like Go To Meeting or WebEx. Here the external trainer can share their desk top and talk directly with the students over a conference call. This approach means students have to dedicate time to the training. Of course they also get a specialist training who’s very familiar with the product. Without face to face interaction it’s unwise to allow each session to go over 2 hours. Students’ attention soon starts to drift without that direct interaction with the trainer.

Summary
There are many training options available beyond the usual “get an external instructor” solution. All of them have their benefits and disadvantages. So the option you choose will usually be dictated by a number of factors. Firstly do you have the resources in-house to undertake the training to a decent standard. Secondly how crucial is it for the students to learn about the test management system in “your” business context. If you have little resources and the context isn’t important then remote web based training may be the best solution. If you have the resources and context is important then your own internal trainer may be the way to go. Whichever option you go for it’s extremely important that you never underestimate the importance training has on the uptake of you test management system.

It’s always interesting taking a look back over the year to examine some of the significant software failures. Whilst companies rarely allude to the causes behind these failures it’s easy to argue that poor software testing is likely to contribute significantly. The trouble with blaming this on software testing is that it usually means the QA team takes the wrap. And in taking the wrap we’re pushed into blaming it on poor process (e.g. test management process), lack of resources or even poor requirements. Naturally the QA team feel aggrieved that they are being singled out. And rightly so. Product quality is the responsibility of the whole product team not just the QA team.

So when we see failures, like the three examples that follow, it’s difficult not to feel a large degree of empathy for the some of the software testers. These people are likely to be bearing the brunt of the failure. I’ve worked in financial companies where failures cost millions in lost revenue. I’ve seen testers fired on the spot in the witch hunt that follows. I’ve seen, miraculously, testing budgets doubled after such a failure. I’ve seen boards of companies suddenly understand why testing is so important. No doubt these three companies are currently following that same cycle.

1. US financial conglomerate fined millions for covering up bug

A fund that used computer models to work out its trading approach was found to have a significant bug. A bug that resulted in investors loosing millions of dollars. When questioning these losses investors were fobbed off with explanations around market volatility. The real reason was a software defect in the trading algorithm. Whilst it is alleged that employees found the issue and acted to resolve it the company alleges that they tried to hide the issue. What ever the background the company was fined heavily by the SEC (Securities and Exchange Commission).

2. System issues result in ATM downtime for one of Japans largest bank

Issues on the ATM network for one of Japans largest banks resulted in a complete outage of the network. With thousands of machines out of action for nearly a day customers were left having to withdraw from branches only. This was compounded with failures to make salary payments and a million unprocessed payments. In conjunction Internet banking facilities were offline for three days.

3. Australian ATM defect gives customers extra money

With 40 ATMs giving out significant sums of money by mistake this Australian bank’s customers thought they had hit the jackpot. Apparently with the machines operating in a standby mode customers could withdraw funds without being prevented by any account limits. With this issue lasting more than 5 hours large queues formed with customers withdrawing funds well past limits set on their accounts.

There’s a predictable pattern when high profile failures like this happen. Blame the test team. They blame poor process (like the test management systems in place), lack of resources and poor requirement definition. Senior management wake up to the importance of the QA function. Budgets double to ensure it doesn’t happen again. When the dust settles budgets get cut as part of a company wide efficiency drive. Failures happen due to lack of resources/commitment to the QA process. The cycle starts all over again.

If these examples highlight one thing it’s the complexity of integrated systems. No longer are we testing a single system in isolation but we need to be testing massively integrated systems that all rely on this interconnectedness to function correctly. Some may argue that this complexity is going beyond our ability to comprehend and test effectively. Perhaps we’re at a point where no amount of well thought out test management process and software testing skill is going to prevent these types of failures?

The Test Management solution QAComplete 9.7 has been out for around 3 months now. In that time we’ve had the opportunity to implement for many clients and evaluate the new feature set in the real world. Billed as a solution comparable to HP’s Quality Center there are some interesting new features to help teams manage their QA process here.

With 3 months of use under our belts our impression is that SmartBear have been listening to their client base. They have restructured the way testcases are managed but retained the core traceability and visibility features. Essentially it’s still easy to use but delivers a far richer feature set for the QA team.

So here are some of the key features from this release that stood out for us:

Releases

You can now define releases in terms of versions, builds, iterations (or using which ever nomenclature best suite your setups). With releases defined you can then link all other test management artifacts back to the release. Where this delivers big benefits is with traceability reports for a release. You can now quickly run a report to see which requirements have been implemented for a release, which defects are impacting the release and which sets are being executed. This does make it really easy to see at a glance exactly where you are up to with a release.

Test Management

The whole approach to testing in QAComplete has been restructured. No longer do you have a ‘Test Cases’ area but a whole new ‘Test Management’ area. This delivers a test library, configurations, sets and an execution area. This brings QAComplete right into the same space as HP’s Quality Center product with a similar feature line up. Not only that but also the capability to export your data from QC and import directly into QAComplete.

Library

The new library feature allows you to store all your testcases in one area of QAComplete. These can then be reused within sets. Tests are now version controlled and contain steps (a major omission in previous versions which is now rectified). The steps feature is easy to use. You can enter steps and expected results in much the same way as you would in a spreadsheet. In addition you can quickly reuse steps from other cases by dragging and dropping from other testcases in the library.

Configurations

Test management execution runs are now tracked against configurations. This makes it easy to trace runs against say different browser types. You can structure your configurations into folders, add custom fields and link to other artifacts like defects.

Test Sets

All of this comes together in the new test sets area. A set contains a collection of cases from the library. The test management set is then executed against a configuration and a release. So for example you might run your regression set against, release version 1 and XP/IE8 configuration. Each testcase within the set is then executed in turn, where each step within the case is given a pass or fail result. The first core concept to grasp here being that a set is linked to the releases it will be executed against and the configurations the tests will be run on (see below).

From here when you run the set you are given the chance to specifiy exactly which release and configuration you will run this against. Crucially here you can specifiy multiple combinations at the same time. So you could for example run against version 1 and version 2 of your product at the same time. Or you could run against config A and config B at the same time.

Execution

The new run and execution engine in QAComplete makes running the testcases very straight forward. Once you’ve specified the release and config to run against you can step through each testcase in the set and mark the individual steps as either pass or fail. Time taken to execute is tracked along with percentage complete.

Whilst each test case is given an individual status, the set is also given an status. So when you complete the set you can either pause the run, or end the run as a pass or fail.

In summary this is quite a prescriptive workflow that has been implemented within QAComplete’s test management engine. It’s either an approach that will work well for your process or it won’t. Having said that it’s an approach that has worked well within other popular test management tools for years now, so it’s well proven and well worth trying out.

All test management tools provide reporting to some degree. Most also support exporting of test results and test data. That data may include testcases, run status information, configuration settings, etc. Whatever the information you need to export the key question will be does the test management tool support the export format you need? Typically you can expect a tool to support CSV, Excel, Pdf, Word (or Rich Text Format) and XML. The following video shows how to export in these various formats from QAComplete.

You’ll see in this video that there are two core approaches to exporting. The first is to export directly from within the GUI to CSV format. This involves selecting the data you wish to export (usually by defining filters to select the test cases you need) and then selecting one of the two following options…

1. Export visible: using the ‘select fields’ feature you first select the fields you want to display on the listings page. It’s this data that is then exported when you select export visible data

2. Export all: regardless of the fields displayed or selected all data associated with the test case is exported.

Note that in QAComplete this export option only supports CSV format. If you need this in excel then just open the CSV file in Excel and then save again in Excel format.

The second option involves creating a test management report and then exporting the report data. QAComplete comes with various test management reports from which you can then export the underlying data. Note that you can only export the information that the reports suck out of QAComplete. If you don’t have a report format that covers the information you need then you can’t export the information you need. Having said that you can create your own reports with exactly the data you require. Once the information is displayed in a report you can then send this to CSV, Excel, Pdf, Word or XML formatted files. See the video to understand how this is achieved.

In short every QA team has differing requirements for exporting data. The key part is to make sure the test management tool you select supports an extensive range of formats so that you have the flexibility long term to produce the exported data your team is likely to need.