System testing is a vital part of the engineering process. It's the
last line of defense against issues that appear in a software system
as an integrated whole. In CS 32 we use extensive system tests in
evaluating your projects, and you should, too. We've written a testing
script to help you test your command line interfaces. The tester is
/course/cs0320/bin/cs32-test but you
can access it just by typing
cs32-test in your
terminal as long as your PATH is setup properly.
The tester takes one argument and some optional flags.
cs32-test [--help] [--executable <executable>] [--timeout <timeout>] [--ignore-whitespace] <test suite(s)>
The test suite argument is a path to your test suite(s). The --help flag will show all flag options and their descriptions. The --executable flag allows you to specify what program you'd like to test. Usually, this is the ./run script, so the system tester looks for a run script in the current directory by default. The --timeout flag allows you to specify a timeout (in seconds), after which the test will fail. The default timeout is 5 seconds, but you may need to change this value for certain tests (especially those with very large datasets or expensive computations). Finally, the --ignore-whitespace flag tells the system tester to remove any trailing whitespace from test files and your program's output. By default, whitespace is not ignored (that is, trailing whitespace is passed into the program and also considered when comparing expected and actual output).
Tip: All of these flags can be abbreviated to their first letter. That is, you may refer to them respectively as -h, -e, -t, and -i.
Structure of a test suite
The test suites that we use in CS 32 are modeled with each test having its own file containing three sections: ARGS, INPUT, and OUTPUT. The args section contains a single line with any command line arguments that should be passed into the program when it is executed. The input section contains lines passed directly to the standard input stream, and the output section defines what the test expects to read from the standard output stream. The end of the test is marked by a line with END on it. Our naming convention is to give each test file a .test file extension. Here is an example the contents of a test for the Node.js executable, node. Node is installed on all department machines by default. We call this file node.test:
ARGS --interactive INPUT 1+1 OUTPUT > 2 > END
This test can be run with
cs32-test --executable /local/bin/node node.test
on any department machine. If you have Node installed on your own computer, you should be
able to run the same command by substituting /local/bin/node with the path to
your installation of the Node executable (try
which node to find it). A shorter
way to write this command would be
cs32-test -e /local/bin/node node.test.
Each of the three sections are optional, meaning you can leave them out if you don't need any arguments or input. However, you'll almost always want an expected output section. Here's an example of a test for the standard command line utility, echo. Notice that there is no input section:
ARGS Stop repeating everything I say! OUTPUT Stop repeating everything I say! END
If this were saved to a file named echo.test, you could run this test with
cs32-test -e /bin/echo echo.test.
The system tester's job is to ensure that the contents of the output section match the contents of the input section. But there is a little room for variation, explained below.
Expecting an error
In CS 32 and in industry, we expect you not to make your internals visible to the users. One way in which this might happen would be an uncaught exception bubbling to the surface, showing a stack trace and killing your program. This would be bad user experience — your code should explain problems accurately to the user to make it obvious how the problem can be fixed. At the very least, if you have to display to the user that something went wrong, don't do it with a stack trace — print something informative to the user and continue running (if possible).
As a concession to simplify our testing of your code, you should
always preface error messages with the
ERROR:. This will help us tell apart your error
messages from the rest of your output, and will allow you to explain
the error however you think is best, without requiring a perfect
string match in the tests. Any line of output that begins
ERROR: will pass the tester in this case, because we
don't specify all of the error edge cases — it's your job to
find them. But we will look at your error and reward clearer
A sample error handling case is here:
INPUT wacky error-causing input!! OUTPUT ERROR: END
If a test has line that is
ERROR:, any number of consecutive
"ERROR: ..." lines output by the program will be covered by this single line
in the expected test output. That is, feel free to print multiple lines in
certain error cases where it's warranted. Your test file will only specify that
at least one error is expected.
Ignoring extra output
There may be some cases where you wish to log additional information
from your program, but do not wish to include it in the expected output
of your system tests. Any line prefixed with the string
will be ignored by the system tester. That being said, it is good practice
to keep debugging or extraneous information from cluttering your command
line interfaces, so we do not expect your handins to print out numerous
INFO messages while using the command line interface.
Give your tests clear, informative filenames. It should be easy to understand all the cases your tests are covering just by listing out the files in your test suite directory. We will deduct points if your tests are named something along the lines of case1.test or error.test.
If all or some of your tests are failing because the output is empty, then there is a high chance that the timeout is too small and it is cutting off your program before it gets a chance to return the answer. Try a larger timeout — the exact value can vary from project to project and dataset to dataset.
If you want to test your entire suite you can run all tests in your directory using the wildcard path ./tests/*.test. However you can also just test a single test case by specifying it directly (e.g. ./tests/no-database-provided.test).