Test Management Blog

The Test Management solution QAComplete 9.7 has been out for around 3 months now. In that time we’ve had the opportunity to implement for many clients and evaluate the new feature set in the real world. Billed as a solution comparable to HP’s Quality Center there are some interesting new features to help teams manage their QA process here.

With 3 months of use under our belts our impression is that SmartBear have been listening to their client base. They have restructured the way testcases are managed but retained the core traceability and visibility features. Essentially it’s still easy to use but delivers a far richer feature set for the QA team.

So here are some of the key features from this release that stood out for us:

Releases

You can now define releases in terms of versions, builds, iterations (or using which ever nomenclature best suite your setups). With releases defined you can then link all other test management artifacts back to the release. Where this delivers big benefits is with traceability reports for a release. You can now quickly run a report to see which requirements have been implemented for a release, which defects are impacting the release and which sets are being executed. This does make it really easy to see at a glance exactly where you are up to with a release.

Test Management

The whole approach to testing in QAComplete has been restructured. No longer do you have a ‘Test Cases’ area but a whole new ‘Test Management’ area. This delivers a test library, configurations, sets and an execution area. This brings QAComplete right into the same space as HP’s Quality Center product with a similar feature line up. Not only that but also the capability to export your data from QC and import directly into QAComplete.

Library

The new library feature allows you to store all your testcases in one area of QAComplete. These can then be reused within sets. Tests are now version controlled and contain steps (a major omission in previous versions which is now rectified). The steps feature is easy to use. You can enter steps and expected results in much the same way as you would in a spreadsheet. In addition you can quickly reuse steps from other cases by dragging and dropping from other testcases in the library.

Configurations

Test management execution runs are now tracked against configurations. This makes it easy to trace runs against say different browser types. You can structure your configurations into folders, add custom fields and link to other artifacts like defects.

Test Sets

All of this comes together in the new test sets area. A set contains a collection of cases from the library. The test management set is then executed against a configuration and a release. So for example you might run your regression set against, release version 1 and XP/IE8 configuration. Each testcase within the set is then executed in turn, where each step within the case is given a pass or fail result. The first core concept to grasp here being that a set is linked to the releases it will be executed against and the configurations the tests will be run on (see below).

From here when you run the set you are given the chance to specifiy exactly which release and configuration you will run this against. Crucially here you can specifiy multiple combinations at the same time. So you could for example run against version 1 and version 2 of your product at the same time. Or you could run against config A and config B at the same time.

Execution

The new run and execution engine in QAComplete makes running the testcases very straight forward. Once you’ve specified the release and config to run against you can step through each testcase in the set and mark the individual steps as either pass or fail. Time taken to execute is tracked along with percentage complete.

Whilst each test case is given an individual status, the set is also given an status. So when you complete the set you can either pause the run, or end the run as a pass or fail.

In summary this is quite a prescriptive workflow that has been implemented within QAComplete’s test management engine. It’s either an approach that will work well for your process or it won’t. Having said that it’s an approach that has worked well within other popular test management tools for years now, so it’s well proven and well worth trying out.

All test management tools provide reporting to some degree. Most also support exporting of test results and test data. That data may include testcases, run status information, configuration settings, etc. Whatever the information you need to export the key question will be does the test management tool support the export format you need? Typically you can expect a tool to support CSV, Excel, Pdf, Word (or Rich Text Format) and XML. The following video shows how to export in these various formats from QAComplete.

You’ll see in this video that there are two core approaches to exporting. The first is to export directly from within the GUI to CSV format. This involves selecting the data you wish to export (usually by defining filters to select the test cases you need) and then selecting one of the two following options…

1. Export visible: using the ‘select fields’ feature you first select the fields you want to display on the listings page. It’s this data that is then exported when you select export visible data

2. Export all: regardless of the fields displayed or selected all data associated with the test case is exported.

Note that in QAComplete this export option only supports CSV format. If you need this in excel then just open the CSV file in Excel and then save again in Excel format.

The second option involves creating a test management report and then exporting the report data. QAComplete comes with various test management reports from which you can then export the underlying data. Note that you can only export the information that the reports suck out of QAComplete. If you don’t have a report format that covers the information you need then you can’t export the information you need. Having said that you can create your own reports with exactly the data you require. Once the information is displayed in a report you can then send this to CSV, Excel, Pdf, Word or XML formatted files. See the video to understand how this is achieved.

In short every QA team has differing requirements for exporting data. The key part is to make sure the test management tool you select supports an extensive range of formats so that you have the flexibility long term to produce the exported data your team is likely to need.

On a fairly regular basis we get asked for a specific test management report that will show the aggregated results from multiple runs. The idea behind this is that ultimately you can show that every testcase was run and that every one passed.

So for example on test cycle 1 you may get…

my_test_case_A:  failed
my_test_case_B:  passed
my_test_case_C:  passed

Then on cycle 2 you get….

my_test_case_A:   passed

The aggregated test management information would then show the following:

my_test_case_A:  passed
my_test_case_B:  passed
my_test_case_C:  passed

So we’re looking for the latest result for each testcase and showing that result as if all the tests have been run in one go. Very misleading when you consider it from that perspective.

There is a certain level of attraction to this test management information. It’s one of those reports that can be categorised in the “that gives us and our customer a nice cosy feeling that everything has passed and the product is ready for release” reports.  From the QA engineers point of view it’s nice to show a list of results to your customer that’s simple and lists every case as passed. Customers see this as a nice to have too. However, If I was the client, receiving this, I would be horrified. Results reporting in this way hide key information, and misleads on a number of levels.

Firstly, it demonstrates that little, or no, attention has been paid to regression testing. Assuming you are just re-running failures then you’re probably not re-running tests that passed first time round. If you’ve got to do a rerun then that implies you’ve made changes to the application. If you’ve made changes to the application then you need to be regression testing. As a customer that receives such a management report all I can say is that I’ve got absolutely no visibility of any regression testing that’s taken place. I’m going to start questioning the worth of your current test management tool at this point.

Secondly it entices the QA team to aim for a goal of 100% of tests completed with 100% passes. This leads to a kind of “Oh how lovely, we have a perfect application to release” syndrome. The team should not be aiming for a goal of 100% passes they should be aiming to find defects. It’s too easy to think testing is complete when you’re inadvertently enticed into producing the perfect report rather than a perfect approach to finding bugs.

Thirdly, it implies that little or no attention is being given to version control of the application under test. Such a report demonstrates that all testcases have been run against “the” application rather than some run again version x of the application and others run against version y of the application. Your product might be under version control but you’re not recording tests against the version they were run against. Without this test management information you have no traceability and no repeatability within your process.

I suspect that most requests for such a report come either from an inexperienced QA team or the end client. A client that’s looking to see information conveyed in this way ought to be educated by the QA team. A team that’s providing information to the client in this way ought to be improving their test management process as a matter of urgency.