CI/CD Without Regression is Like a Brownie Sundae Without The Ice Cream!

As brownie sundaes are a really important topic I am going to wax eloquently on my recipe for a perfect brownie sundae:

1. A Brownie – it should be soft, rich and moist
2. Ice cream purists insist that it should be vanilla ice cream but I am afraid that here I really go out on a limb and insist that it be chocolate ice cream
3. Fudge – correct richness, sugar level and thickness is incredibly important in fudge
4. Caramel – like the fudge the wrong caramel, or worse still fake caramel, destroys the experience
5. Nuts (optional) – Ok it’s hard to get nuts wrong if you want them on the sundae

At this point in time you must be wondering what this has to do with CI/CD? Well here is the thing. CI/CD is like the brownie. The entire toolchain that you setup, whether you are using Jenkins, Bamboo or some other platform, and the effort that the development team takes to put the software into modules that can be built and deployed is the brownie!

But you still need the ice cream and that basically boils down to the validation that you can carry out as a part of the CI/CD tool chain. There are a couple of methods that can be used to create validation suites. Taking the analogy a bit further – should you use chocolate or vanilla ice cream or better yet BOTH? No, I have never had chocolate and vanilla in the same brownie sundae …you can only take an analogy so far! But it makes sense to use unit testing and system testing in your CI/CD validation.

Vanilla and Chocolate are Both Needed!

From the start unit tests have been considered as a crucial part of a CI/CD implementation. However, the question quickly arose as to whether unit testing was sufficient for continuous deployment, especially all the way to production? Could it cover all type of validation especially for web-based systems?  Much of the industry has moved towards using both. For example, this article from Infoworld (by author Isaac Sacolick) delves into using continuous testing after deployment and using system automation tests to perform regressions in addition to unit tests that are created by the developer.  The article suggests going even further by integrating performance and security tests into the CI/CD chain.

CI/CD chains have slowly started incorporating system tests into their suites. There are a number of reasons why system tests were not there from the beginning in CI/CD systems. These are:

1. CI/CD was a developer driven initiative. System test has historically been part of QA and thus was not included.
2. Unit tests were perceived as sufficient to carry out code validation, at least at the developer or integration level, although some organizations have taken this trend even further and pushed to production with no system test. Most people agree that system test, performance test and security should be carried out as mentioned above in the Infoword article.
3. Time for execution – A system test or a regression test can take a tremendous amount of time days or weeks in many organizations). However, with automation systems built on Selenium or even more comprehensive services like Webomates CQ this is no longer a valid reason. Webomates CQ offers a CI/CD service that guarantees that all test cases are executed in 15 minutes….regardless of the number of automation or AI automation test cases. Read more about it here.
4. Cost of execution – The cost of developing automation test cases, like the cost of developing unit test cases, is high. Services like Webomates CQ offer this capability at a far more reasonable cost.
5. Software Automation can be really flaky (yeah that’s a technical term). Automation, particularly Selenium, can have time out issues, and is very sensitive to locator changes etc. That is, it’s flaky. To combat this, Webomates CQ offers tools for automation execution analysis for our CI/CD service as well as guaranteed execution for all test cases in our higher-level services like Overnight and 24 hour service.  Combining CI/CD for development with Overnight for integration and the 24 hour service for system test we refer to as overlapped regression and can provide a tremendous increase in quality at far lower price points.

When should You Use System Test with CI/CD?

As early as possible! Following shift left (as outlined in this article by Forbes author Adrian Bridgewater) principles the goal is to test as early as humanly possible. Ideally when the developer first checks in their code, even before an integration test, they should be able to get a high degree of validation from their unit test and system test executions.

The key is that it should be easy and quick.  A typical development life cycle starts off with low effort on quality in the initial stages of development and typically ends with very high focus on quality at the end of the development stage. As the figure above shows, starting to test early and often is key to reducing defects found downstream in production or even worse by customers.  

Quick – While I Take a Coffee Break

For a developer’s productivity not to be impacted the CI/CD execution in totality needs to be fast. Our target is to have the following happen in the time that it takes you to have a coffee break:

1. Code has been compiled and re-linked
2. New software has been deployed
3. Unit tests were carried out
4. System tests were executed
5. Analysis of the results are provided (Look at Fudge)

Easy

The easier it is to add system testing to the CI/CD execution chain the less time the developer has to spend on making this happen and the more time he/she can spend on creating features! In a perfect world all the developers should need to do is to add one line of scripting in the CI/CD script to invoke the system test execution. This line would specify:

1. The test cases that need to be executed by  module. This allows a tailored set of test cases to be executed rather than the entire system test suites.
2. The deployment environment that the software is being deployed to.
3. The number of retries for automation failure test cases that should be executed.

Webomates CQ offers this integration today.

Where’s the Fudge and Caramel?

An astute brownie sundae lover would have noticed by this point in time that I have not quite reached the other essential ingredients, namely fudge and caramel. Well the fudge for us would be analysis and the caramel would be exploratory testing.

Analysis

The temptation to say that you have to fudge the analysis is really high at this stage! 😊 And often that is what can happen if it takes the developer too long to figure out if anything is broken and then to identify what is broken. So, they fudge it and push the build!

After all, what the developer wants to know is whether the work that (s)he has done in the last few hours has broken the software build in any way or can they proceed. Also, if they have broken something then any information that can rapidly help identify the defect will help.  Examples of such help are:

1. Probability of defect
2. Area of UI
3. Module/line of code

Webomates CQ provides a defect prediction that helps a developer rapidly recognize whether there is a new bug that has been introduced.

Exploratory Testing (Optional Caramel?)

Here I am really going to go out on a limb and challenge current dogma.  If there was the ability to have an exploratory test carried out as part of the CI/CD chain would you choose to execute it?

Test case based testing only goes so far and is pretty rigid. It is a really good way to verify that what you know and have defined works correctly. Exploratory testing goes beyond the known envelope to try things out that are not heavily documented and scripted. It’s an incredibly valuable way to get an idea of the unknown and constantly increase the level of quality that you have. If you want to learn more about the differences between Exploratory and Test case based scripting read this blog that I wrote earlier on the subject.

Webomates CQ offers exploratory testing as part of its offering today.

Conclusion

CI/CD is a rising wave that technology companies are adopting all over the world. A lot of the value revolves in building and testing frequently so that incremental change can be verified more quickly and deployment to production can occur faster. However, in order to achieve this goal, verification at the system level is important. Using your existing system automation, or adding in a service like Webomates CQ for CI/CD can far more quickly and easily help realize the goal of rapid and reliable deployments in the field. Like with my brownie sundae, all the parts are integral to achieving the perfect balance and perfect tasting sundae!

If you are interested in learning more about Webomates’ CQ service please click here and schedule a demo or reach out to us at info@webomates.com.

Tags: Regression Testing, Software Testing