Test Case-Based Testing vs Exploratory Testing

So which one is better? Easy answer. You need to do both types of testing. No this is not a case of just doing more is better!! Exploratory testing and test-case based testing take different and complementary approaches for testing a product. Let’s start with basic definitions.

Test Case-Based Testing

In this approach the test team sits down and defines the test cases that they will be carrying out on the day of regression.

“A test case is a set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly.”

Typically the test case will have a series of steps with or without data being entered defined with one or more validation steps carried out to determine that the behavior of the system is occurring correctly.

Exploratory Testing

In Exploratory testing the team sits down prior to testing of the next build and creates a charter or a set of guidelines on where to explore. This can include high defect areas, new feature areas, areas where defects are fixed etc. Here is a more formal definition from ISTQB

“Exploratory testing is a hands-on approach in which testers are involved in minimum planning and maximum test execution. The planning involves the creation of a test charter, a short declaration of the scope of a short (1 to 2 hour) time-boxed test effort, the objectives and possible approaches to be used.”

In addition if you are looking for an excellent guide on how to perform Exploratory testing Elisabeth Hendrickson’s book Explore It!! is an easy yet phenomenal read.

Test Case vs Exploratory

The essence of the difference between the two techniques is between baseline functionality versus being adaptive.

Test case-based testing IF the test cases are written well is a great way to ensure that a basic consistent level of functionality is operational in the software. It is typically written once for a set of functionality and is only modified if the features are modified. And taken to the extreme in terms of coverage it can be sufficient. However, getting to 100% test case based coverage can be resource intensive in terms of both time and effort.

Exploratory testing on the other hand is by design an adaptive technique that can be used to significantly enhance the level of coverage without adding hundreds or thousands of test cases. In each regression the charters are initially selected that are likely to yield the highest defects and greatest understanding of the product. After a time boxed execution, the charters are refined once again based on the learnings from the first time boxed execution and the cycle repeats until the regression cycle is competed.

Pictorially module level coverage is shown on the right:

One caveat. If too many defects are being found by exploratory testing in comparison to test case-based testing this can be a strong indicator that an increased level of test case-based testing needs to be carried out. For each module the test case-based testing gives a baseline of consistent test cases and the exploratory testing augments the base testing.

In a similar manner end to end test cases give a baseline set of testing and exploratory testing can be used to augment the coverage model. At times I have seen teams swing from all test cased base testing to only exploratory testing and vice versa. Neither alone is optimal.

Why Choose?

Test case based testing and Exploratory testing are both excellent techniques for reducing the number of defects found in production. Together they serve as two great complementary techniques that build on each other. Ideally, we would like to do both.

Webomates offers a regression testing service that uses manual testing (test case- and exploratory-based), automation and crowdsourcing together to guarantee all test cases (even modified ones) are executed in ONE day!

Tags: Exploratory testing, Test case