Crowdsourced Testing : Introduction and Benefits

Crowdsourcing is a relatively new method for carrying out regression testing. Let’s start by defining what crowdsourcing is:

” Crowdsourcing is the practice of getting a task or project executed by enlisting the services of a large number of people via the Internet. There are several systems on the market such as Amazon’s Mechanical Turk that can be used to distribute regression tasks to thousands of people. Technically this is manual testing, but the key value is that the manual tests can be carried out with on-demand resources and the scale can be very large.

Curated or non-curated crowdsourcing

Curated means do you want resources that are selected/tested/filtered for a skill. In the case of curated regression testing the people that are selected are those with a QA background, with various offerings providing resources with upto 5 to 10 years of experience in QA manual testing. In non- curated resources anyone that has a computer and can read and execute the instructions that are provided for completing the task are utilized for executing the task.

Test case based testing- crowdsource style

There are systems on the market that require the creation of test cases by the customer. Others offer test creation as part of the service for a cost. In either case once the test cases are created, when the next software build is available, the test cases are distributed to crowdsource resources. This allows for hundreds or even thousands of test cases to be carried out in parallel. Although not as fast as scaled automation testing, the result is that test case based regression can be carried out far faster than typical manual testing, in hours as opposed to days. There are several issues that have to be thought through in this case:

1. Quality of testers – Was it accurate?
2. Fraud – Was the work done?
3. Platform availability – Do testers have the requisite browser, OS and mobile platforms?
4. Test case writing effort

Exploratory testing-  crowdsource style

Here testers are focused on finding defects without any script or guidance. The wrinkle here is that you are working with an arbitrary pool of resources on the Internet. The companies that offer this service typically uses a curated set of QA crowdsource resources. The focus is on finding defects in a focused amount of time.  There are a couple of concerns that come up with these kinds of services:

1. Quality of testers
2. Fraud
3. Number of defects found
4. Qualify of defects – the severity/priority of defects for the customer

Crowdsource vs Manual vs Automation

Why Choose?

Clearly, crowdsourcing offers different values as compared to manual and automation testing. Combining all three forms of execution offers a higher quality, faster overall execution time and the highest quality of defects.

Webomates offers a regression testing service that uses manual, automation and crowdsourcing together to guarantee all test cases(even modified ones) are executed in ONE day with exploratory testing! In fact, the output is a list of prioritized defects for software teams to review and fix.

Tags: Crowd-sourcing, Knowledge, Regression Testing