CSC207 Software Design
Lectures
Systematic Testing

Sample Problem

Find intersection of two rectangles

Used in windowing systems, urban planning, etc.

Rectangle is represented by lower left and upper right corners

Assume all coordinates are non-negative integers

And non-zero extent in both directions

Output is:

New rectangle (their intersection)

null if they don't overlap

Goal: build up a testing framework

Motivate/justify design of tool introduced in next lecture

Examples

Terminology

A fixture is a thing to be tested

E.g. a pair of rectangles

A test is something done to a fixture

Must know what outcome you expect

A result is one of:

Pass: test produced correct answer

Fail: test ran, but produced incorrect answer

Error: test failed to produce an answer (i.e. there's something wrong with the test itself)

Design

Must be able to run tests repeatedly

Measure progress while writing code

Prevent regression when fixing/modifying code

Tests that behave as expected should be quiet

If you give people screenfuls of output saying "Everything is OK," they will miss the two lines that say, "Except for this."

When something does go wrong, people will run the tests in a debugger with a breakpoint set in the broken test

How to Test Testing?

To show how testing works, need several different implementations of overlap

Some incorrect

Don't want to have to rewrite testing code each time

Solution:

Write testing code in terms of an abstract generic rectangle class

Use factories to create actual rectangles to test

Only test with legal rectangles

Generally not good to assume users can't create invalid objects

But let's test one thing at a time

Generic Rectangle

abstract class AbstractRect {
    
    public AbstractRect(int loX, int loY, int hiX, int hiY) {
        init(loX, loY, hiX, hiY);
    }
    
    public AbstractRect(int[] coords) {
        init(coords[0], coords[1], coords[2], coords[3]);
    }
    
    public abstract AbstractRect overlap(AbstractRect other);
    
    public boolean equals(Object other) {
        if ((other != null) && (other instanceof AbstractRect)) {
            AbstractRect obj = (AbstractRect)other;
            return (fLoX == obj.fLoX) && (fLoY == obj.fLoY)
                && (fHiX == obj.fHiX) && (fHiY == obj.fHiY);
        }
        return false;
    }
    
    protected void init(int loX, int loY, int hiX, int hiY) {
        fLoX = loX;
        fLoY = loY;
        fHiX = hiX;
        fHiY = hiY;
    }
    
    protected int       fLoX, fLoY, fHiX, fHiY;
}

Notes on AbstractRect

Constructors can take either:

Four discrete coordinates

An array of coordinates

Really should define in terms of Point

Make the overlap() method abstract

Users must implement it in derived classes

Always define equals() to take an Object

Make sure to handle null and objects of unexpected types

Actual class overrides hashCode()

Always do this when defining equals()

If equal objects do not have equal hash codes, sets and maps will fail to find them

Skeleton

class Test {
    public static void main(String[] args) {
        ...create a rectangle factory...
        ...run tests and accumulate results...
        ...report results...
    }

    public static int runTest(IRectFactory rectFactory, int testId, int[][] test) {
        int result;
        try {
            ...create input rectangles and expected result...
            ...run overlap...
            ...check result...
        }
        catch (Exception e) {
            ...result is error...
        }
        return result;
    }
}

Test Results

Define integer constants for pass, fail, and error

Use values to index 3-element array of result counts

public static final int PASS  = 0;
public static final int FAIL  = 1;
public static final int ERROR = 2;

Test Table

Using table-driven testing makes test cases easier to read and write

Store parameter in an int array

Convert to objects when needed

Easy to mix tests in code with tests read from files

public static int Tests[][][] = {
    // left      right       expect
    {{0,0,1,1},  {0,0,1,1},  {0,0,1,1}},
    {{0,0,1,1},  {2,2,3,3},  null},
    {{1,1,4,4},  {2,2,5,5},  {2,2,4,4}},
    {{1,2,3,4},  {4,4,5},    null},        // badly-formatted test
    ...etc...
};

Selecting Rectangle Factory

public static void main(String[] args) {
    IRectFactory rectFactory = null;
    if (args.length > 0) {
        if (args[0].equals("fixed")) {
            rectFactory = new FixedRectFactory();
        }
        else if (args[0].equals("almost")) {
            rectFactory = new AlmostRectFactory();
        }
        else if (args[0].equals("correct")) {
            rectFactory = new CorrectRectFactory();
        }
    }
    if (rectFactory == null) {
        System.err.println(USAGE);
        System.exit(1);
    }
    ...run test and report results...
}

Gathering Results

public static void main(String[] args) {
    ...create rectangle factory...

    int[] results = {0, 0, 0};
    for (int i=0; i<Tests.length; ++i) {
        int r = runTest(rectFactory, i, Tests[i]);
        results[r] += 1;
    }

    System.out.println("PASS  : " + results[0]);
    System.out.println("FAIL  : " + results[1]);
    System.out.println("ERROR : " + results[2]);
}

Running a Single Test

public static int runTest(IRectFactory rectFactory, int testId, int[][] test) {
    int result;
    try {
        AbstractRect left   = rectFactory.makeRect(test[0]);
        AbstractRect right  = rectFactory.makeRect(test[1]);
        AbstractRect expect = null;
        if (test[2] != null) {
            expect = rectFactory.makeRect(test[2]);
        }
        AbstractRect actual = left.overlap(right);
        if (expect == null) {
            result = (actual == null) ? PASS : FAIL;
        }
        else {
            result = expect.equals(actual) ? PASS : FAIL;
        }
    }
    catch (Exception e) {
        System.err.println(e + " (on test " + testId + ")");
        result = ERROR;
    }
    return result;
}

A Note on Syntax

C ? T : F is a ternary expression

If C is true, value is T

If C is false, value is F

Allows code like x * ((i >= 0) ? i : -i)

So result = (actual == null) ? PASS : FAIL is the same as:

if (actual == null) {
    result = PASS;
}
else {
    result = FAIL;
}

The Fixed Rectangle Class

"Fixed" because its overlap() method returns a fixed result, no matter what the input

class FixedRect extends AbstractRect {

    public FixedRect(int loX, int loY, int hiX, int hiY) {
        super(loX, loY, hiX, hiY);
    }

    public FixedRect(int[] coords) {
        super(coords);
    }

    public AbstractRect overlap(AbstractRect other) {
        return new FixedRect(0, 0, 1, 1);
    }
}

The Fixed Rectangle Factory

class FixedRectFactory implements IRectFactory {

    public AbstractRect makeRect(int loX, int loY, int hiX, int hiY) {
        return new FixedRect(loX, loY, hiX, hiY);
    }

    public AbstractRect makeRect(int[] coords) {
        return new FixedRect(coords);
    }
}

Fixed Rectangle Output

$ java ...args... Test fixed
java.lang.ArrayIndexOutOfBoundsException: 3 (on test 3)
PASS  : 1
FAIL  : 5
ERROR : 1

Note: test program never crashes

So that error in one test never stops another from running

Go back and fix the error (a badly-formatted test table entry)

Almost Correct

public AbstractRect overlap(AbstractRect other) {
    int fLoX = (this.fLoX > other.fLoX) ? this.fLoX : other.fLoX;
    int fLoY = (this.fLoY > other.fLoY) ? this.fLoY : other.fLoY;
    int fHiX = (this.fHiX < other.fHiX) ? this.fHiX : other.fHiY;
    int fHiY = (this.fHiY < other.fHiY) ? this.fHiY : other.fHiY;
    if ((fHiX <= fLoX) || (fHiY <= fLoY)) {
        return null;
    }
    return new AlmostRect(fLoX, fLoY, fHiX, fHiY);
}

Can you spot the error reading the code?

Almost Correct Output

$ java Test almost
PASS  : 6
FAIL  : 1
ERROR : 0

Should add a command-line flag to have the program print the IDs of failing tests

Then set conditional breakpoints in the main loop to step into just those tests

Limitations

Automated unit testing appeals to programmers because it is programming

Very easy to spend all of your time building test harness, rather than actually testing

Good at preventing regression

Not good at finding:

New types of bugs ("I didn't know that could break")

Design errors ("Isn't it supposed to do that?")

Unexpected interactions (you're only testing units)

$Id: systest.html,v 1.1.1.1 2004/01/04 05:02:31 reid Exp $