Meeting Numbers through Regression Testing

Software Testing Methodologies

Eliminating false-failures and improving repeatability is a must in high-volume manufacturing. Software regression testing methodologies and strategies can be an effective way of reducing occurrence of test failures. Consider this: A high-volume manufacturing line builds 1.5 million automotive cluster units in one year. An end-of-line functional test system on the line yields approximately 98% of the units, with 2% of the product dropping out due to false-failures. This hiccup correlates to 30,000 reject units from test system failure per year, translating into additional time at the rework stations, additional time for operators to retest, additional material costs if parts are scrapped and so on. All-in-all, it hurts the bottom line as deadlines may be missed due to inability to meet volumes. End result? Not a very happy plant manager, nor a pleased end-customer.

So what are some ways to reduce false-failures and bugs in your code? While it can be extremely difficult to release bug-free software, (think Windows barrage of updates and patches) there are methods we can use to improve the stability and repeatability of our code. Some of these ways include:

  • Looping sections of the code for extended periods.
  • Creating test-cases.
  • Failure injection (FMEA) analysis.
  • Interfacing your software on a variety of systems with complex configurations.

One case study during integration of a test system required logging data to a csv when the cycle count failed and analyzing it with our plotting software. Due to the nature of the failure (occurrence approximately 1 out of every 30 test cycles, with each test cycle being 30 minutes long) it was quite difficult to reproduce. Once plotted, the failure was obvious and the threshold for the algorithm had to be adjusted.

Creating test-cases for your software is a standard among majority of large-scale companies. In fact, several IDEs, such as Microsoft Visual Studio Test Professional or Eclipse’s JUnit incorporate a Test-Case Management platform to help facilitate software testing. Formal test-cases are tracked in traceability matrices to determine requirements are met without adverse effects.

A sample pre-screen interview a company like Microsoft may give to a potential Software Test Engineer candidate would be along the lines of asking you to:

  1. Create a function which determines the type of triangle based on its corresponding lengths (scalene, isosceles or equilateral)
  2. Create test cases for such function.

And of course, we will include our proposed solution below.

public class Shapes
     * public int GetTriangleType(int x, int y, int z)
     * Determines the type of triangle based on its corresponding lengths.
     * Input Type Description
     * x int length 1
     * y int length 2
     * z int length 3
     * Output
     * int Triangle Type (1=scalene, 2=isosceles, 3=equilateral, 4=error)
     public int GetTriangleType(int x, int y, int z)
         const int SCALENE = 1;
         const int ISOCELES = 2;
         const int EQUILATERAL = 3;
         const int ERROR = 4;

         if (x <= 0 || y <= 0 || z <= 0)
            return ERROR; //lengths must be greater than 0
         else if (x == y && x == z)
            return EQUILATERAL;
         else if (x == y || x == z || y == z)
            return ISOCELES;
         else if (x != y && x != z && y != z)
            return SCALENE;
            return ERROR; //unkown error

The corresponding test-cases may look like this:

public class Program
    static void Main(string[] args)
        Shapes shapeTest = new Shapes();
        int shapeType = -1;
        //Test Case 1 Group: Test for an equilateral triangle - Returns 3 (Equilateral)
        shapeType = shapeTest.GetTriangleType(10, 10, 10);
        //Test Case 2 Group: Test for an isosceles triangle - Returns 2 (Isosceles)
        shapeType = shapeTest.GetTriangleType(10, 10, 1);
        shapeType = shapeTest.GetTriangleType(10, 1, 10); 
        shapeType = shapeTest.GetTriangleType(1, 10, 10); 
        //Test Case 3 Group: Test for a scalene triangle - Returns 1 (Scalene)
        shapeType = shapeTest.GetTriangleType(1, 2, 3); 
        //Test Case 4 Group: Test for an error - Returns 4 (Error)
        shapeType = shapeTest.GetTriangleType(0, 2, 3);
        shapeType = shapeTest.GetTriangleType(1, 0, 3);
        shapeType = shapeTest.GetTriangleType(1, 2, 0);
        shapeType = shapeTest.GetTriangleType(0, 0, 0);
        shapeType = shapeTest.GetTriangleType(5, 0, 0);
        shapeType = shapeTest.GetTriangleType(0, 5, 0);
        shapeType = shapeTest.GetTriangleType(0, 0, 5); 

The list is not exhaustive, but I’m sure you get the idea. I mention Microsoft because – well – they gave me this very pre-screen test back in 2011 when I was contacted for a position by their Office group for software testing in Redmond. That’s a story I shall save for another day…

Although it’s extremely beneficial to eliminate bugs and false-failures, it can take extended amount of time and effort to detect and fix such issues. Additionally, we often worry that introducing software fixes could cause other consequences. It reminds me of the Pesticide Paradox:

Pesticide Paradox: “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.” – Beizer, Software Testing Techniques.

Meeting Numbers through Regression Testing was last modified: January 14th, 2014 by Paulo R

Leave a Reply

%d bloggers like this: