Overlapped Regression for Agile Teams

Webomates’ CQ is composed of three different services:

• A full regression using both Exploratory and Test-case based testing
• Overnight – which is a guaranteed execution based solely on a test case
• Smoke tests that are pure automation executions

Full Regression

The full regression is a very thorough execution and always increases the envelope of quality. A depiction of a full regression, covering a product composed of 4 modules, is shown below. A full regression is composed of two different parts:

1. Test case based: The blue bars show the area of the product-module that is being covered by documented test cases. This is a baseline of guaranteed capabilities that are validated by extremely well-defined test cases with concrete execution steps and multiple validation points.
2. Exploratory based: The orange dots show defects that are found in a particular regression run. Exploratory testing is typically either charter based (a charter is a guidance for what area to test) or completely random. Execution is time based and the intent is to explore the software as much as possible and discover defects.

However, it is HEAVY in terms of both time (24 hours) and in terms of effort/cost.

Overnight Regression

In an Overnight Regression, solely test-case based regression is carried out on only a single module.

With Webomates’ execution guarantee ALL module test cases will be executed, including modified test cases. This is important to note as test cases that change due to a defect fix or feature change will also be validated! Thus, an agile development team working on a particular module, on a specific day, can do a check-in in the evening and come back the following morning confident that all the baseline test cases for the module are working! By the morning, if any defects are discovered, they are available and ready for the development team to review. And the same can be done for every other module.

Smoke Tests

Smoke tests are a subset of the test cases that are completely automated and can be run at any point in time by the customer him/herself.  Smoke tests are structured to be end-to-end as well as module focused. The main advantage is that they can be invoked at any point in time and can be completed within 15 minutes. However, there is no exploratory testing included, there is no execution guarantee and no defects are created. The development team needs to look at the Pass/Fail report to determine the result of the regression run. Agile teams use a smoke test multiple times throughout the day to assess the state of the build prior to invoking an Overnight test on the module under development.

Comparative Table

The figure below shows how many of our customers’ agile teams use overlapped regression –  that is the combination of Full, Overnight and Smoke to achieve high levels of quality without very large amounts of effort or cost.

In the above scenario, the customer has subscribed for 500 test cases. They are running per week :

1. 1 Full regression,
2. 3 overnight on different modules
3. Multiple daily smoke on the different module or full system.

Conclusion

Although the most thorough process for this customer would be to do a daily full regression that is not the most optimal scenario. By combining smoke, overnight and full regression the customer is better able to expand the quality envelop with full regressions while maintaining the baseline of test-case based regressions that are targeted at the module level that development is currently working on.

If you are interested in learning more about Webomates’ CQ service please click on the link and schedule a demo.

At Webomates, we have applied this technique in multiple different domains and successfully delivered with a quality defects. We specialize in taking many different software creators’ builds across domains, on 18 different Browser/OS/Mobile platforms and providing them with defects in 24 hours or less. If you are interested in a demo click here Webomates CQ

Tags: Agile, Agile Testing, Continuous Quality, Feature Testing, Overnight, Regression Testing, Smoke testing, Software Testing, Test-case based testing