How to High Jump To High Quality-Part 2

In the last article in this series I had talked about the three steps that are needed in order to High Jump to High quality. These were :

1. Start  –  The preparatory phase in which test cases and charters are outlined and the necessary domain understood
2. Run    –  Getting test case execution started across one or more execution methods (Automation, AI automation, crowdsource & manual )
3. Jump – Getting Exploratory charters refined and improving the quality envelop with every regression that is carried out

At the end of the How to High Jump to High Quality we had included a short five question survey that many readers responded to. For those of you that replied to the survey a BIG THANK YOU!

This blog compares what the respondents of the survey said as compared to what we have been seeing when carrying out only test case based or exploratory testing versus doing a combination of both.

Five Question Survey on Test cases versus Exploratory testing

How much experience do you have in QA ?

It was nice to see that almost 50% of the respondents to the survey had more than five years of experience in QA. This creates among the survey respondents a strong basis of very experienced QA professionals mixed in with some people with lower amounts of experience and potentially a fresh perspective.

Which technique do you think yielded the maximum number of defects?

Almost 50% of the respondents felt that the combination of exploratory testing and test case based testing would yield the maximum number of defects. This is completely in line with what we found when running three parallel blind teams below.

How Likely are you to find all the defects every time with only exploratory testing, only test case based or with a combination of both?

The respondents were not as clear as to their choices in the case of this question. As we will layout in the rest of the article below this particular question has interesting ramifications. The question basically revolves around the guarantees of quality. That is which form of testing will yield the same number of defects every time when it is carried out with the same version of code? If you want to jump to what we have determined here is a link to the spot below with our results.

Research 1 Test Cases vs Exploratory vs Combined

Hypothesis: A combination of test case and exploratory testing gives the best quality result and high jump to the software quality.

Fixed Attributes:

• Same product
• Same version of software
• Same environment

First step was to understand the product and identify 500 test cases of the product for testing.

During the second step we created three test execution teams and each team had similar knowledge about the product as well as experience.

1. Team Exploratory – They used the exploratory approach of testing. They created exploratory charters and executed those charters to find defects.
2. Team Test Case – They executed only 500 test cases to find defects.
3. Team Combined – They executed 500 test cases and also created and executed exploratory charters for exploratory testing to find defects.

Let’s see how those defects from the three teams map out to test cases and how many were same and how many were unique as shown in Table 1 below.  

Table 1

Figure 3 Exploratory vs Combined vs Test Cases

Analysis:

• Exploratory is Hit and Miss testing ( Question 3 of the survey)

• It can help in improving quality but using it as a basis for testing is gambling.
• Team Combined missed one of the bugs that the Team Exploratory found for example  
• Team Exploratory found 5 of the 8 test case related bugs.
• Team Combined and Team Test Case found all of the same test case based bugs independently showing that the Test case approach is very reliable and predictable.

Conclusion

This concludes Part 2 of 3 of the Series How to High Jump to High Quality. In Part 1 I laid out the thesis on how an organization can significantly improve their quality in production with the Start-Run-Jump philosophy. In part 2 with the help of all the survey  respondents and  our internal  research we  have shown  two fundamental insights :

1. Combination of Exploratory and Test case based testing can increase your overall quality in production
2. Exploratory and test case based testing have different values. In particular over reliance on just exploratory testing can cause a reduction in quality as the reliability of test case based testing is higher. However test case based testing just confirms what you know works and exploratory gives you a view outside the boundary

In the next part of this series I will delve into the long run consequences of running exploratory and test case based testing in parallel. I found the results fascinating and we are gathering more and more data of our regression execution results both for analytics and also now to feed our multiple Artificial  Intelligence Engines. The kinds of questions I will try to answer in the third part of this series will be :

1. Over time which will yield more defects Exploratory or Test Case based testing?
2. Over time which one will yield  more Priority 1 Business impacting defects ?

If you like this blog series please like/follow us @Webomates or @Aseem. And of course if you are interested in learning more about our service Webomates CQ here’s a link to request a demo.

Tags: High Quality Testing, Software Testing