How to use makefiles for automated testing

But before, a warning: I'm putting this up quick and dirty, to give you an idea of how to use makefiles for testing if you don't already know. It is probably not entirely clear -I barely proofread it-, but I preferred to put something up fast to help you if need be. This is not the best way to do automated testing, by far, but it works and gives you almost immediate results.

For this very short tutorial, I assume you already have an idea of how makefiles work. There are tons of tutorials on make --google "make tutorial"--, read and understand one of them before continuing! Also, I'm assuming you'll be using either linux or, if on a Windows machine, cygwin...

Alright, so you already know that makefiles are normally used for building a system, and that they consist of three basic elements:

target : source(s)
	commands

(Note that by some awful accident of fate, every command must be preceded by a tab. Not by spaces, it has to be a tab. Also, source(s) are not necessary.)

The idea, then, is to use targets as test cases, and commands as instructions for your OS to execute the test case. For example, the following makefile:

testSomething :
	program.exe

...will run program.exe if you type "make testSomething" on your command line.

You can, of course, run tests somewhat more complicated than that. For example, if you want to feed in some lines of text to your program, you can take advantage of the command line and try:

testSomething :
	program.exe < input.txt

And if you want to send the output from the program to yet another file...

testSomething :
	program.exe < input.txt > output.txt

This way, if you type "make testSomething", your program will be executed, whenever it asks for input it will receive it from input.txt, and will send all its output to output.txt.

You can still go a bit further than that. Using the diff command, you won't even need to open output.txt to make sure the output is correct. Assuming you have the correct expected output in a third file (let's call it correct.txt):

testSomething :
	program.exe < input.txt > output.txt
	diff correct.txt output.txt

"make testSomething" will run and keep quiet if it doesn't encounter any differences between the output and the expected output, but will warn you and stop if it does find differences. This is very helpful if you have tens or hundreds of test cases: you just run make and, if it keeps quiet, you know your tests passed!

By the way, if you have several test cases you can execute them all with one instruction. Let's say you have test1, test2, and test3 in a makefile. Add the following line at the end of the file:

test : test1 test2 test3

Now, if you type "make test", the makefile will see that to 'build' your target 'test' it will need to execute targets test1, 2 and 3 as well.

That's the gist of it. Try it out, it beats manual testing by far, and it's a good alternative when you can't do unit testing.