How to reduce the number of Test Cases?

How to reduce 4320 test cases into 3 using Pairwise Testing?

Hieu Le

INTRODUCTION

Designing test cases for the applications having multiple parameters is challenged. Sometimes there are so many Test Cases need to be created and you might feel it difficult deciding which one should be taken into account, which one should be skipped. Let's have a look at this example from http://www.softwaretestinghelp.com/what-is-pairwise-testing/

Car Ordering Application

  • The car ordering application allows for Buying and Selling cars. It should support trading in Delhi and Mumbai.

  • The application should have a registration numbers, may be valid or invalid. It should allow the trade of following cars: BMW, Audi, and Mercedes.

  • Two types of booking can be done: E-booking and In Store.

  • Orders can be placed only during trading hours.

Let’s list down the variables involved.

  • Order category: Buy, Sell

  • Location: Delhi, Mumbai

  • Car brand: BMW, Audi, Mercedes

  • Registration numbers: Valid (5000) or Invalid

  • Order type: E-Booking, In store

  • Order time:  Working hours, Non-working hours

If we want to test all possible valid combinations:
Number of Test Cases = 2 X 2 X 3 X 5000 X 2 X 2 = 240000 valid test cases combinations. There are also an infinite number of invalid combinations.
And that is the problem. A tester cannot write all of them due to the large number of combinations, as well as the limited amount of time. Luckily, there is a technique called Pairwise Testing mentioned and explained clearly in the example link above. In this blog post, I will apply it, plus the concept of Test Data (or Testing Data Sets) to reduce the number of Test Cases of a running project at Bloopark. In my case, it is for User Acceptance Test, but I think we can have the same approach for Automation Test.

TECHNIQUES AND TOOLS

Pairwise Testing also known as All-pairs testing is a testing approach taken for testing the software using combinatorial method. It's a method to test all the possible discrete combinations of the parameters involved. The idea is to have 100% test coverage with a reasonable number of test combinations. In the end, we have a 'best' test cases, not the 'entire' test cases, but we still ensure the test quality.

Read more:
http://www.softwaretestinghelp.com/what-is-pairwise-testing/
http://www.tutorialspoint.com/software_testing_dictionary/pairwise_testing.htm
 

There are several tools supporting Pairwise Testing http://www.pairwise.org/tools.asp, I use an online tool from https://inductive.no/.

Test Data (or Testing Data Sets) is data that is used to execute the tests.
E.g., the login feature needs 2 input parameters User Name and Password. The Test Data could be:

User Name

Password

user1

123

user1

456

user2

1357

user2

2468

user3

123456

user3

987654321


Read more:
http://www.tutorialspoint.com/software_testing_dictionary/test_data.htm
http://www.softwaretestingmentor.com/test-deliverables/what-is-test-data/

EXAMPLE

Let's see this example. I had to prepare test cases for the feature 'Inspection Voucher' in an ERP project. Here is the list of the main variables involved, with the most number of values is first, and the least is last. Besides, there are many other variables I did not put here because they are configuration parameters used in calculation formulas. 

  • Companies: Deutsche Bahn, Commerzbank, AXA, Canyon, Other

  • Inspection Voucher: Mandatory - Company pays,  Mandatory - Employee pays,  Optional - Employee pays, No Voucher

  • User: Manager, Employee, ABC Agent

  • Insurances: ABC, AG / AN, ABC AG

  • Type of bicycle: Fahrrad,  Pedelec, S-Pedelec

  • Wandlung der Rate: vst, Nvst

  • AG-Zuschuss: Fixed,  Fixed & Floating

  • MwSt. aus Geldwerten Vorteil: yes, no

Theoretically, at least I had to create 5 X 4 X 3 X 3 X 3 X 2 X 2 X 2 = 4320 test cases. I created 12 test cases based on 3 User types, 4 Inspection Voucher types and what I feel important to test. Regarding the Input values, they were not dynamic in each test, this can be solved with the Testing Data Sets in an Excel file that I am going to talk below. 

But now I am going to use Pairwise Testing and Test Data to re-design them.

Step 1: define the input parameters

Step 2: generate the best test cases
The tool uses an algorithm to do that automatically. As a result, I have 23 test cases. 

At the tab 'Analysis of Tests', you can see some interesting info of the relation between the number of test cases and the percentage of test coverage.

Step 3: define the number of test cases and test data
I should have created 23 different Test Cases but I can reduce them to 3 test cases thanks to:

  • Each user type (Employee, Manager, ABC Agent) has similar steps.

  • Test data

  • A separate excel file to handle the input / output parameters, and the calculation.

I consider a 'test cases in step 2 as 'testing data'. As you can see in the screenshot below, finally, I have:

  • Test case 1: Employee with 8 records of test data.

  • Test case 2: ABC Agent with 6 records of test data.

  • Test case 3: Manager with 6 records of test data.

TEST CASE 1: EMPLOYEE DOES THE INSPECTION VOUCHER PROCESS

Testing Data Set

Repeat the Test for each Testing Data record.
Use this Excel file to handle Input and Output values 

ID

Steps

Expected Result

1

click on "Neuer Auftrag" in employee dashboard

user should be redirected to order form
Name, Email and Personalnummer are pre-filled and not editable
Country should be pre-filled with Germany but editable

2

enter all personal information like phone, street, Zip code

3

select the bike type, brand as described in the Testing Data Set
add bike information like model, color, size and accessories

4

add VK price, UVP price as described in the Testing Data Set

5

click in "Was ist Inspection-Coupon"
click on the link at the bottom.
(for some personal reason, I am not allowed to share the link here)

user should be forward to the related website.

6

Use the Excel file to check these fields: AG-Zuschuss, Versicherungsrate_netto, Versicherungsrate_brutto, Leasingrate, Umwamdlungsrate, Geldwerter Vorteil, Tax - Geldwerter, Vorteil, Restwert, Restwert_netto

The values should match the Output values calculated in the Excel file.

7

select supplier from list

8

click in "confirm"
upload documents
click in "confirm" again

order should be successfully confirmed

9

Manager should confirm the order (frontend)

order should be successfully confirmed

10

go to backend as a ABC Agent
Use the Excel file to check UEV and ELV attachments in KAU

The values should match the Output values calculated in the Excel file.

11

confirm the KAU
confirm the LAU
receive the products

DLI is created

12

in DLI, click in transfer
input any serial number 12345
delivery date: "today"
confirm
go to Inspection Voucher module

The Inspection Vouchers can be auto generated or not, depending on the configuration. Check the Excel file for more info.

13

If Inspection Vouchers are generated in step 12, continue step 13, 14,15. Otherwise, the test ends here.
Use the Excel file to check the following information in the 3 vouchers

The values should match the Output values calculated in the Excel file.

14

click in print

the PDF Voucher should be downloaded

15

check in the pdf the following dynamic fields: ABC, XYZ, Rahmen-Nr, Übernahme, Einlöse Zeitraum, Wert.

the fields should match the previous information

TEST CASE 2: ABC AGENT DOES THE INSPECTION VOUCHER PROCESS

Testing Data Set:

Repeat the Test for each Testing Data record.
Use this Excel file to handle Input and Output values 

ID

Steps

Expected Result

1

In back end, go to Sales > Quotations > Create a Sales Order
Select values for Lieferadresse, Sales Order Line and other input values from the Excel file 

order should be saved

2

Use the Excel file to check these fields: AG-Zuschuss, Versicherungsrate_netto, Versicherungsrate_brutto, Inspektion Voucher, Leasingrate, Umwamdlungsrate, Geldwerter Vorteil, Tax - Geldwerter, Vorteil, Restwert, Restwert_netto

The values should match the Output values calculated in the Excel file.

3

click Confirm the Sales Order

order should be confirmed

4

Use the Excel file to check UEV and ELV attachments in KAU

The values should match the Output values calculated in the Excel file.

5

confirm the LAU
receive the products

DLI should be created

6

in DLI, click in transfer
input any serial number 12345
delivery date: "today"
confirm

The Inspection Vouchers can be auto generated or not, depending on the configuration. Check the Output values in the Excel file for more info.

7

If Inspection Vouchers are generated in step 6, continue step 7, 8, 9, 10, 11, and 12. Otherwise, the test ends here.
go to menu Sales > Inspection Vouchers > Inspection Vouchers
Use the Excel file to check the following information in the 3 vouchers

The Inspection Vouchers can be auto generated or not, depending on the configuration. Check the Output values in the Excel file for more info.

8

Click on Print

the PDF Voucher should be downloaded

9

check in the pdf the following dynamic fields: ABC, XYZ, Rahmen-Nr, Übernahme, Einlöse Zeitraum, Wert.

the fields should match the previous information

10

go to connectors module and force the first voucher from draft to valid.

voucher should be available for download in frontend detail view (Employee and Manager portal)

11

Go to the 1st voucher
click on redeem
select the supplier
Pay the LRE created

LRE should be attached to the voucher form at field ‘Liederantenrechnung’

12

go to the 1st voucher
go to "attachments" tab
upload the voucher signed by the supplier (choose file type: Inspection Voucher signiert)
click in save

voucher.pdf signed should be successfully uploaded.
in frontend detail view, the voucher available for download should change to the voucher.pdf signed by supplier.

TEST CASE 3: MANAGER DOES THE INSPECTION VOUCHER PROCESS

Testing Data Set:

Repeat the Test for each Testing Data record.
Use this Excel file to handle Input and Output values 

ID

Steps

Expected Values

1

click on "Neuer Auftrag" in dashboard

click on "Neuer Auftrag" in dashboard
user should be redirected to order form

2

- select any employee, any supplier, the bike type, brand as described in Testing Data Set
- add bike information like model, color, size...

all employee personal information like name, address, city, country ... should be filled automatically

3

add VK price, UVP price as described in Testing Data Sets

no error message should be shown

4

Use the Excel file to check these fields: AG-Zuschuss, Versicherungsrate_netto, Versicherungsrate_brutto, Leasingrate, Umwamdlungsrate, Geldwerter Vorteil, Tax - Geldwerter, Vorteil, Restwert, Restwert_netto

The values should match the Output values calculated in the Excel file.

5

- Confirm the order
- Upload documents
- Confirm the order again
- Release the order

Order status should be successfully confirmed

6

go to backend as a ABC Agent
Use the Excel file to check UEV and ELV attachments in KAU

The values should match the Output values calculated in the Excel file.

7

- confirm the KAU
- confirm the LAU
- receive the products

DLI is created

8

in DLI, click in transfer
input serial number: 12345
delivery date: "today"
confirm

The Inspection Vouchers can be auto generated or not, depending on the configuration. Check the Output values in the Excel file for more info.

9

If Inspection Vouchers are generated in step 8, continue step 9, 10, 11. Otherwise, the test ends here.
Go to menu Sales > Inspection Vouchers > Inspection Vouchers
Use the Excel file to check the following information in the 3 vouchers

The values should match the Output values calculated in the Excel file.

10

click on print the Inspection Voucher

the PDF Voucher should be downloaded

11

check in the pdf the following dynamic fields: ABC, XYZ, Rahmen-Nr, Übernahme, Einlöse Zeitraum, Wert.

the fields should match the previous information

CONCLUSION

That was how I reduced the number of my test cases from 4320 to only 3 (99,93%) using Pairwise Testing and Testing Data Sets while I still ensure 100% test coverage. I am new to the world of Software Testing. As some of my tester friends said, there are a lot of testing types and techniques used for different purposes. You can visit http://thetestingmap.org/ to see the big picture. With my limited knowledge I write this article after reading a lot of reference documents to solve a small problem: test cases for workflows with multiple parameters that I face at my current company. It is good to know Pairwise Testing as it is a popular technique I think. Hopefully I can write more articles about software testing in the future.