CSC444'06F Assignment 2

The purpose of this assignment is for you to better understand both release planning and regression testing. For this assignment, you may optionally work in teams of up to 3 people if you so choose.

You are to write a full automated regression test environment for the slide puzzle of assignment 1. You will treat this project as a release, coming up with a release plan in advance, and then comparing your actual to your estimated times and work factors.

The features your program are to contain shall be as follows. You may add other features if you choose, or divide the features out in different ways than are listed here.

• Write a test director (TD) that can repeatedly start the program under test (PUT), feed it data via stdin (hardcode the sequence of tests in the TD for now), and store the results reaped from stdout (assuming the PUT does not crash for now)
• Implement a list of tests to run stored externally to the TD.
• Implement a timeout in the TD that can kill the PUT if is not finished within a given time.
• Specify the appropriate timeout with each test.
• Make the TD understand and record the details of PUT crashes.
• Make the TD measure and store the total time taken to solve the puzzle.
• Make the TD compare the results of running a test to a previously established baseline.
• Make the TD automatically establish a baseline the first time a new test is run.
• Implement a puzzle solution tester that can automatically test if a proposed solution to a puzzle is an actual solution and integrate it into the TD.
• Implement a solvable puzzle generator that can generate repeatable random solvable puzzles of various dimensions, and integrate its use into the TD.
• Implement as a separate program a test results reporting (TRR) component that can summarize the number of tests run and passed over time (ideally a web application - though not necessary).
• Include into TRR a time history of average performance over all tests.
• Enhance the average time history to be comparable from day to day in the face of changing tests.

You will receive a letter grade for your efforts. Hand in the following:

• Before starting work on the system, go through a planning exercise with your teammates and put together a preliminary release plan formatted the same way as the Deterministic Release Plan in Appendix A of the book. Choose your phase dates. Divide features into A and B groups as you see fit. Estimate your w's during the coding period in advance, and estimate your feature coding ECDs.
• On the final due date, hand in the release plan above modified to also include actuals for feature sizings and work factors. Be clear about which features are working at the end and which are not.
• Detailed time logs for each team member.
• A written description and design for your system.
• The well-commented and nicely and neatly printed out program code for your system with report output samples.
• A report of your team's experiences with release planning.
• Part of your mark will be based upon how functional the system is (how many useful features are working within it) inversely weighted by team size.