Accelerate Success with AI-Powered Test Automation – Smarter, Faster, Flawless

Start free trial

Automation always out of date

A relatively unknown fact that is often overlooked outside of the software testing space is that the cost of software test automation is only 20% in the setup phase and the cost of maintenance of the software test automation is 80% of the overall cost.

This is illustrated in the diagram below where a test suite of 500 test cases is shown for testing a Web User Interface application with traditional automation and automation test resources. The cost for setup and then the ongoing costs for maintenance of the test suite. Naturally, the costs vary on a number of factors:

  1. If there is no ongoing development after the test suite is created then there is no maintenance cost.

  2. The more agile the process the more frequent the releases the HIGHER the cost of the maintenance. Unfortunately, the industry is moving to a much higher speed to market of releases with the FAANGT/MAANGT companies (Facebook/Meta, Amazon, Apple, Google & Tesla) releasing features multiple times a day across their product portfolios in many cases

Thus for the model shown below a 30% script breakage per month has been taken. This breakage is caused due to:

  1. Feature change or addition.
  2. Locators.
  3. TimeOut Errors.
  4. Test Data Change Errors.

As the model shows, creation of 500 test scripts takes about 50 person days. The automation is ready on Jan1st. Then the maintenance with 30% failure also in person days…..15 person days month. In the first year alone the cost of maintenance is more than 3X the setup costs ( 50 person days versus 180)

The maintenance cost by the way is still 2X the setup cost in Year 1 (50 person-days setups versus 100 person-days for maintenance. But the real issue is that most software teams are being pushed to release features monthly or weekly and some teams are even being asked to do it DAILY!!!

Priceline - Overview

Priceline.com is an online travel agency for finding discount rates for travel-related purchases such as airline tickets and hotel stays. The company facilitates the provision of travel services from its suppliers to its clients. Priceline.com is headquartered in Norwalk, Connecticut, United States and is wholly owned by Booking Holdings, which also owns Kayak.com, Booking.com and other sites.

Priceline had spent a large amount of time and effort on their software test automation. However, as they became more and more agile, cloud-based, and using a DevOps process they were finding that the individual software development teams were using the automation less and less.

As they started releasing features to production multiple times a day per software development team the problem that they faced was the software test automation was never up-to-date. So they carried out a manual spot-check test prior to releasing a feature and tested the feature more thoroughly when the automation caught up.

Thus they were at the forefront of the industry in terms of feature velocity and the introduction of features faster to the market. At the same time, their defect leakage had increased with associated reputation risk and business impact.

Priceline initially deployed Webomates CQ for only one area in their website. And that too only in production. In three months of operation:

  • 168 defects were proposed by the Webomates CQ AI system and TestOps teams.
  • These defects were accepted as valid by Priceline
  • The Priceline team classified 38 Defects as Priority 1 defect

Webomates CQ was then deployed for multiple teams within Priceline’s website. Webomates CQ was executed in:

  • Production - As a continuous automated sanity test to validate the key test cases/functionality always worked
  • Staging - As the final test to dramatically reduce defect leakage into production
  • Integration - to shift left testing earlier in the sprint cycle

Priceline was able to leverage the power of AI-based Intelligent Test Automation that automatically AI heals automation scripts keeping them always up-to-date. They were able to deploy this revolutionary Testing as a Service platform in weeks.

Would you like the same results?

Talk to Us

To learn more about how to keep automation scripts up to date