csc407 students please note that this a slightly edited vesion of the marks scheme prepared for the teaching assistants. My hope is that you can get a sense of what I was looking for from what I asked the TAs to look for.
The main lesson was supposed to have been that small inconsistencies in a design are hard to avoid and relatively easy to work around. Several students, after complaining about how long the assignment took, admitted that the actual programming of the program took only a few hours. Several further admitted to agonizing over each inconsistency for a long time.
In fact, it appears that the writing of test code around a few core classes is the real take-away. Ironically students appear not to have realized that almost all their code to date has been test scaffold around a very few useful classes!
You have 15 minutes to figure out that the students have understood the design as given and implemented it more or less successfully. The test results document should be your starting point. Spend no more than five or ten minutes looking around in the code for stuff that doesn't appear in the document.
This guideline, borrowed from David Penny's mark scheme last year, should help us mark a little more consistently:
| 10 | Excellent | superior graps of the subject matter | no problems |
| 9 | few and minor problems | ||
| 8 | a few more problems, none major | ||
| 7 | good | grasps the ideas, but some conceptual problems | |
| 6 | adequate | grasps the idea at a primitive level | |
| 5 | marginal | some slight evidence that they understand. | |
| 0-4 | inadequate | little to no evidence of even a superficial understanding of what is required | |
Mark Scheme:
| marks | Comment |
|---|---|
| 10 | Understanding the entity design. No flagrant departures from design. if documented properly you may accept variations. The key is that they understood the design not that they followed it slavishly. |
| 10 | testing covers all use cases and exceptional courses. You should be able to discern this entirely from the document. If you have to fish around in the code you can penalize. |
| 10 | Quality of test scaffold design. If anyone thought to create a UML for their scaffold design reward it! Test scaffold must not disturb the entity class implementation whatsoever. Put another way, the testing must be done from outside the entity classes. I characterized this by telling them that their entity class implementations are to be "handed off" to the user interface team later in this hypothetical project. |
| 10 | Implementation quality. This is where the keeners who made great interactive programs or test scaffold should do a little better. If students did a really good job of documenting their implementation in a way that referred to the design you can reward them here also. Particularly messy implementations can be penalized. |
The thrust behind A1 is not to neatly implement some bookstore classes. Rather, it is to read and understand a UML design and figure out how to build the classes and some stand-alone infrastructure to test them.
The A1 handout is somewhat inconsistent, omits some classes but by-and-large is quite good by the standards of these things. The students were rather distracted by relatively small inconsistencies and likely invested a lot of effort documenting their assumptions.
We need not worry too much about the accuracy of their implementation except when it belies a lack of understanding of the design and/or UML. In marking the assignment you should not dock marks for small variations from the design as given -- but you should dock marks for big departures on the assumption that it is an indication of the student's failure to understand the design. If a significant departure is clearly identified as such and the reasoning is clearly articulated then let it pass. My position is that even though this is against my instructions they have indicated the required knowledge. We can let some future client or manager deal with their attitude.
The idea of test scaffold turned out to be totally new to the students and I ended up spending a half hour of lecture time describing various approaches. Though not my intention, I imagine that this might turn out to the be the most valuable take away of the assignment.
In any real application most developers are faced with the problem of how to test their own work before the entire system is finished. Let me explain myself with a story:
Once upon a time there was a young compiler developer. He had written a new register allocator for a new C compiler on a new CPU and all was not well. One day he received an email from a very senior developer telling him that regular expression searching failed in emacs if his register allocator was run but was fine if it was not. (Emacs has a very complex search facility that compiles a regular expression into a finite state machine.) Time was running out. Our young developer was somewhat worried but bit the bullet and started reading regex.c. Things started to look up when he found the following comment:
/* If DEBUG is defined, Regex prints many voluminous messages about what it is doing (if the variable `debug' is nonzero). If linked with the main program in `iregex.c', you can enter patterns and strings interactively. And if linked with the main program in `main.c' and the other test files, you can run the already-written tests. */
It turns out that using this test program the problem was found in a matter of a few hours. Richard Stallman was the author of the test mainline.
I told this story to the students. Their test scaffold should aim to achieve a similar purpose as the code Stallman left in place for me.
It turns out that building an interactive scaffold is quite a lot of code, even for this simple program. Therefore in office hours and in lecture I described a much less ambitious scheme that I explicitly told them would be acceptable also. I suggested that they write a static method for each test that they want to run.
I indicated to the students that there are two main ways of building their classes. Both must test each use case and exceptional course. The first approach will require some careful documentation for each test arguing that it shows the classes it test work correctly. The second approach is much harder to program and will require some general description of how it works. On the other hand the correct tests should be self evident. In both approaches students must show and discuss the test results to convince you that tests pass or fail.
This course is not on object persistence and hence I don't much care what approach the students use to creating objects for their testing. Some, no doubt, will write a lot of code to read from text files in a desire to make their program more "realistic". This is okay, but not required. In class and office hours I suggested that test data should be instantiated and "pushed into" their entity classes from the test scaffold. I suggested this because it is the simplest possible way of proceeding.
Read the A1 page
Read the handout, available from A1 handout
Read the A1 faq as it will help you understand where the students are coming from.
Finally, I faked up the parable on test scaffold a little. The comment I quoted is in fact from xemacs 21.4.9 and the test scaffold is now really fancy. It looks like regex.c has been rewritten and no longer contains Stallman's autograph.