Test Management Blog

Have you read Quiet by Susan Cain yet? In it Cain refers to studies that show just how much peer pressure influences our decisions — a concept that has some interesting ramifications for us as software testers.

We’re all exposed to peer pressure. Even as testers we are subjected to peer pressure.

Peers in development roles question the defects that we uncover. Our project management peers question our test plans.

In an attempt to overcome issues with peer pressure, an advertising man named Alex Osborn invented brainstorming. He believed that creativity was stifled by the fear of judgment from colleagues. So he sought to remove the threat of criticism from teamwork.

Brainstorming has just 4 rules:

       1. Don’t judge or criticize ideas
       2. Put forward unusual ideas
       3. Aim for quantity of ideas
       4. Build on each other’s ideas

Osborne believed that brainstorming as a practice would produce more and better ideas. He also believed that brainstorming as a group would deliver better results than working alone.

Yet, according to several studies, there’s an argument to suggest that brainstorming doesn’t work. Why?

In 1963, Professor Adrian Furnhams at the University of Minnesota conducted a study on brainstorming — the results showed that more ideas are produced when people work on their own. It also showed that the ideas produced alone are of equal or higher quality. Adrian Furnham’s research led to the following conclusion: “evidence from science suggests that business people must be insane to use brainstorming groups.”

Further studies have produced similar results.

So what, you say? How often does anyone use brainstorming is software testing?

In my experience, not often.

It’s where this leads that’s important, though.

It turns out that even in a brainstorming environment, the fear of looking silly in front of our peers is still strong. So strong that it’s been given its own term — “evaluation apprehension.” There’s not a lot we can do about it, unfortunately. Neuroscience research suggests that the fear of judgment runs deep, so deep it may have a significant impact even for those of us in roles as software testers.

As humans we’re prone to conform. Gregory Berns, a neuroscientist, wanted to know if people conformed even though they knew their peers were wrong. Or (and this is where it gets interesting), if their perception had been altered.

In 2005, Berns came up with a complex way of measuring brain activity to work out the answer (more of which you can read about in the book Iconoclast — see below). The long and short was that peers somehow managed to change individuals’ perceptions!

This implies that peer pressure can actually change your view of a problem. Worse still, you are completely blind to how much your peers have influenced you. You have NO control over this.

Which leaves an interesting question to consider. Just how much influence do our peers (developers, project managers, etc.) really have over us?

For example, we find a defect in an application. We’re not sure if it really is a defect. It works, but it just doesn’t quite conform to what we think the end user would expect. Developers think it’s okay. Do we say nothing?

Just how much are our independent roles as assessors of quality influenced by our peers? How often have you let a bug go because a developer or project manager applied peer pressure? (Peer pressure that you didn’t even realise was being applied?)

If nothing else, it suggests that there’s a whole world of psychology and neuroscience that we can learn and grow from as we strive to be our best as software testers.

References:
Quiet: The Power of Introverts in a World That Can’t Stop Talking by Susan Cain
Iconoclast: A Neuroscientist Reveals How to Think Differently by Gregory Berns, Ph.D.

With the release of the 9.8.1 version of QAComplete and ALMComplete SmartBear software have included two new key test management features. The first is a versioning feature that allows you to select which version of a testcase is used in a run. The second is an assignment feature that allows you to assign at the test level. Lets look at both features in a more detail.

Versioning

QAComplete has always had the capability to automatically increment the version of a case as it’s changed and then use the latest version in new runs. With this new release more of this underlying versioning capability is exposed to the QA engineer to give them more control. You now have the ability to manage tests cases by…

1. listing the versions and select which version should be used in the subsequent runs. You can roll back to a previous version too (without deleting or removing any of the later versions).

2. seeing which versions are included in a set

3. seeing which testcase version was used in each and every run

4. comparing the different versions visually in a browser (although QAComplete will not highlight the differences for you yet)

5. delete previous versions IF they have NOT already been used in a run

These enhancements work well within the overall process that QAComplete imposes and work with little overhead in the day to day test management activities carried out by the QA engineer. They sit there in the background if you don’t need them but provide smart version control capabilities to help you should you need them.

Assignment

In previous editions of QAComplete once you’d assigned a run to a user. No other user would be able to edit or update this once it’s started. With release 9.8.1 there is now the capability for other testers to pick up runs and continue with them. You can also assign at the testcase level too (more about that later).

When the QA engineer starts a run he/she has exclusive ownership of the test management run. When another user attempts to access this he/she has no option to pick up testcases and execute them …

However, now with the 9.8.1 release if the engineer that starts out with the run pauses it, other engineers can then come in and pick it up to complete it. You can see this in the following screen shot where an engineer that the set is NOT assigned to picks up the run.

So if the person who starts the run pauses it, any other tester can come in and pick up that run. It does depend to a degree on the person who starts the run actually taking the time to pause it. Having said that, if you don’t complete a test management run it’s paused automatically which means others will be able to pick it up.

Adding to this assignment capability is also a feature which allows you to assign at the testcase level. So whilst the set is assigned to one person you can come in assign individual testcases to different QA engineers for a run.

A Few Other Features

There are also a few other neat features that have been added recently that will be of interest to the tester.

There is now a Test Run History which allows users to view and sort all runs that have been executed. This gives you the ability to delete, sort and view runs with ease.

Within the testcase listing area there is now a new ‘Create Test Set’ button. This allows you to search for cases (with all sorts of complex search criteria) and quickly include the tests you’ve found in a new set.

All in all this new release of QAComplete contains some pretty important updates to the test management capabilities provided by the tool. Capabilities which will certainly make some big differences to the QA team that demands advanced versioning and assignment capabilities.

These days a large part of successful project management and test management depends on the integration of tools. Tools that are used across different business functions need to be integrated to provide visibility and traceability across the business.

For example the Dev team and QA team might need different tools for test management and agile development. Yet the team as a whole need to be able to report on progress and quality across both functions. Yet with testers raising defects in one tool and developers looking at defects in another tool it’s difficult to keep track and keep in sync. In this sort of situation integration of development and QA tools makes sense.

We’re going to take a look at integrating one of leading test management tools, Quality Center, with one of the top agile development tools, VersionOne. The goal here to give testers the capability to raise defect records in Quality Center. These Quality Center records then get propagated to VersionOne where the developers get visibility of the issues raised. When developers update these records in VersionOne the updates get propagated back down to Quality Center where the QA team get visibility of current defect status. This means the QA team can then pick these defect fixes up for re-testing.

The big advantage with this kind of setup is that the development team get to use their tool of choice and the QA team get to use their preferred tool. The ideal would be to have a single agile tool that the QA team and development team both use. This would be nice but in reality most of the agile tools on the market today (VersionOne, Rally, Jira, etc) are lacking in good test management capabilities. And even as these tools start to implement more effective QA features they’ve got a very long way to go to catch up with Quality Center. So the concept of using integration tools to link the different applications is a good way to proceed if you want to get the best of both worlds.

There are a couple of options to integrate QC and V1. The first is a tool called OpsHub which we’ve talked about before. The second is the “VersionOne Quality Center Integration” tool. It’s this ‘community’ open source tool from VersionOne that we’re looking at here.

This utility is configured and run as a process on a windows server. It runs in the background polling the APIs of both systems. It will sync and update the following entities from both systems…

1. Test Cases
2. Defects

Currently there is no capability to process product back log items in V1 or requirements in QC for the test management.

The work flow processes that it covers are as follows:

1. Tests created in V1, and assigned to Quality Center, are sync’d down to Quality Center
2. Test status updated in QC will be propagated back to the corresponding testcase in V1
3. Defects created in QC will be sync’d up to V1
4. Defect status updated in V1 is propagated back to the corresponding defect in QC.

Setup is pretty straight forward with a ServiceHostConfigTool that comes as part of this integration tool. There are a few steps to get this integration setup…

1. Download and install the VersionOne Quality Center integration tool. This can be obtained from here..

http://community.versionone.com/Downloads/Lists/Platform%20Downloads/DispForm.aspx?ID=19

2. Then we need to make some changes to our VersionOne instance to prepare it for the QC integration. To do this we need to add QC as a global source in the system. A user must also be created within VersionOne to represent the Quality Center user.

3. Setup Quality center. You’ll need to go to the ‘Add Ins page’ and select the Quality Center Connectivity add in which allows QC to integrate with 3rd party tools.

4. Configure Quality Center by customizing the tests entity to add a custom user field to create the versionOne identifier field.

5. Then configure the Version One service host tool. From the files downloaded in step 1 run the following application

ServiceHostConfigTool.exe

This tool allows you to define the parameters for the integration between both tools. The setup consists of a number of pretty straightforward steps

1. configure connectivity to V1
2. setup the Defect field used to hold the defect id within V1
3. setup the testcase service aspects of the integration.
   (This allows you to define your pass/fail statuses)
4. define the QC user name in VersionOne.
5. define the project mapping between both systems
6. config the QC setup, with the QC URL, username and password.
7. Select the QC user name we’re using
8. specify the qc project name, the test folder in QC that will receive the records from V1.

We can then save the configuration before starting the service host process.

6. Then finally start up with version one service host and make sure everything is running correctly…

VersionOneServiceHost.exe

All in all it takes about 30 minutes to install, work out how the application hangs together and then resolve any issues. VersionOne instructions and setup video are very good. Couple of things to watch out for though…

a. You might have mandatory fields in QC testcases. This test management integration tool will try to create testcase in QC from V1 records. However, the mandatory fields are not present in V1 so the integration tool can’t create the record in QC (it fails because QC complains that mandatory data isn’t present).

b. The integration tool can’t work with custom fields in either tools. For example if you have a custom field for priority in both QC and V1 you can’t sync the field data between the two tools.

c. updates to descriptions or comments in records do not get propagated between the two applications. This can make it difficult to see the history of the records from some perspectives. Ultimately it means you end up looking at the record in the originating system which kind of defeats the object really.

In conclusion this is a pretty useful integration tool that comes as a free utility with VersionOne. It has some short comings in terms of not sync’ing product backlog items, and not dealing with custom fields. Having said that the source is open and it wouldn’t be difficult to add to this utility to deliver exactly what you need. Certainly work looking at if you need to integrate your test management solution with the instance of VersionOne that your development team are using.

I read an article the other day that questioned if collaboration actually impacts productivity. As software testers in the age of agile we’re encouraged to embrace the concept of collaboration as something that will improve the way in which we work. We’re pushed to collaborate within the test management process and other business processes. And we’re pushed into working in groups in open plan offices.

There’s probably some merit in this argument that collaboration might result in reduced productivity. In an open plan office we’re constantly interrupted by other people. We find it easier to waste time chatting with people. Getting things done in dedicated and focused work sessions has become more difficult. Some blame this on our desire to be more collaborative. There are probably pros and cons to both sides of this argument. As always there will be a balance to be struck.

There are however, many merits in integrating the tools you use. If your QA team need visibility of defect status in their test management application, or the dev team need to see source code check ins in their defect tracking application then integration is going to help. The aim being that each functional team gets to use their tool of choice which best serves their needs. Yet one tool that works for the development team may not be the right tool for the QA team. Jira is one example.

Jira is used by many development teams to track requirements and defects. The application of choice for many. Yet Jira doesn’t work well for managing test cases. The defect tracking workflow is not a good workflow for test management. So typically a test team ends up picking a different solution. At which point both functional teams end up with different silos of data. Data that in many cases both teams need access to. So we end up dipping in and out of two different tools and duplicating data.

Enter the new (or maybe not quite that new) breed of test management and ALM integration tools. Tools like OpsHub Integration Manager that allow you to synchronize data between completely different applications. OpsHub allows you to keep records maintained in two completely different applications in lock step. For example the QA team raise a defect in Quality Center. Trouble is the development team are using Jira and don’t get visibility of this new issue. With OpsHub configured the QC defect can be automatically created and updated in Jira. When a code change has been made to resolve the issue, and the issue status updated to resolved, that update is automatically sync’d back to QC. The QA engineer then sees this up to date status without having to check back in Jira.

The big advantage of this type of integration is that both parties see the same data even though it’s displayed in different tools. Testers don’t need to leave their test management tool of choice to raise defects in development tools. Team members don’t have to maintain duplicate records in different system manually. Everyone gets the right information, at the right time, in the right application, conveyed in the manner which best meets their needs.

Whilst setting up such integration tools might not be simple the benefits once you’re up and running are huge. In an upcoming post we’re going to look at how this comes together from a practical perspective and see how we can integrate our test management process with other processes that are critical to the project lifecycle. And whilst the argument continues about the potential drop in productivity caused by collaboration at physical level, we can move forward and exploit the definite benefits that tool integration provides.

I was asked by a colleague the other day to define what test management is. Having spent 20 years practicing it, I was surprised to discover that I couldn’t come up with a short pithy one liner to describe it quickly. With a bit of thought I came up with this though…

“The discipline of controlling and tracking the process of testing from the development of test cases through to the reporting on execution status”.

It’s not as easy as you might think to come up with a simple definition. The domain of QA is massive in itself and this aspect tends to impinge on just about every aspect of QA. So whilst a one sentence definition is nice to have it’s difficult to see how it can be truly representative. Having said that if we look at the definition in more detail perhaps there’s more to it than first meets the eye.

From a controlling point of view we’re looking at the definition, implementation and execution of that process. So we define the workflow that a testcase should follow during development. We look at how a system might be implemented using a tool. And we make sure that this system or a set of consistent actions are adhered to at run time.

From a tracking point of view we’re concerned with reporting on progress both from the development through to the running of tests. We might have a testcase development approach that incorporates a review stage for example. In which case we could track the review approved status of our records. We’d also clearly want to look at the results from our execution phase as part of our tracking.

To get a little bit more technical though the word managing means to handle or control something. In this case we’re talking about our testcases. So in short this could be thought of as just controlling or handling our testcases.

In terms of handling we looking at aspects like workflow, version control and traceability that surround the process. We may only have 2 workflow development status values (perhaps say ‘new’ and ‘complete’). For execution we may have just 2 status values (perhaps ‘pass’ and ‘fail’). Even if you’re keeping it that simple you’ve put some control around it by defining a workflow.

So whilst these definitions are nice they do belay the complexities involved with this discipline. The complexities surrounding version control, run histories and traceability for example. All of these concepts, and others, when examined in detail remind us just how complex test management is in real life. So whilst a simple definition might be nice to have it doesn’t exactly convey to others how complex and involved this discipline really is.