System testing is a vital part of the engineering process. It's the last line of defense against issues that appear in a software system as an integrated whole. In CS 32 we use extensive system tests in evaluating your projects, and you should, too. We've written a testing script to help you test your command line interfaces. The tester is located at /course/cs0320/bin/cs32-test but you can access it just by typing cs32-test in your terminal as long as your PATH is setup properly.

The tester takes one argument and some optional flags.

cs32-test [--help] [--executable <executable>] [--timeout <timeout>] [--ignore-whitespace] <test suite(s)>

The test suite argument is a path to your test suite(s). The --help flag will show all flag options and their descriptions. The --executable flag allows you to specify what program you'd like to test. Usually, this is the ./run script, so the system tester looks for a run script in the current directory by default. The --timeout flag allows you to specify a timeout (in seconds), after which the test will fail. The default timeout is 5 seconds, but you may need to change this value for certain tests (especially those with very large datasets or expensive computations). Finally, the --ignore-whitespace flag tells the system tester to remove any trailing whitespace from test files and your program's output. By default, whitespace is not ignored (that is, trailing whitespace is passed into the program and also considered when comparing expected and actual output).

Tip: All of these flags can be abbreviated to their first letter. That is, you may refer to them respectively as -h, -e, -t, and -i.

Structure of a test suite

The test suites that we use in CS 32 are modeled with each test having its own file containing three sections: ARGS, INPUT, and OUTPUT. The args section contains a single line with any command line arguments that should be passed into the program when it is executed. The input section contains lines passed directly to the standard input stream, and the output section defines what the test expects to read from the standard output stream. The end of the test is marked by a line with END on it. Our naming convention is to give each test file a .test file extension. Here is an example the contents of a test for the Node.js executable, node. Node is installed on all department machines by default. We call this file node.test:

ARGS
--interactive
INPUT
1+1
OUTPUT
> 2
>
END

This test can be run with cs32-test --executable /local/bin/node node.test on any department machine. If you have Node installed on your own computer, you should be able to run the same command by substituting /local/bin/node with the path to your installation of the Node executable (try which node to find it). A shorter way to write this command would be cs32-test -e /local/bin/node node.test.

Each of the three sections are optional, meaning you can leave them out if you don't need any arguments or input. However, you'll almost always want an expected output section. Here's an example of a test for the standard command line utility, echo. Notice that there is no input section:

ARGS
Stop repeating everything I say!
OUTPUT
Stop repeating everything I say!
END

If this were saved to a file named echo.test, you could run this test with cs32-test -e /bin/echo echo.test.

The system tester's job is to ensure that the contents of the output section match the contents of the input section. But there is a little room for variation, explained below.

Expecting an error

In CS 32 and in industry, we expect you not to make your internals visible to the users. One way in which this might happen would be an uncaught exception bubbling to the surface, showing a stack trace and killing your program. This would be bad user experience — your code should explain problems accurately to the user to make it obvious how the problem can be fixed. At the very least, if you have to display to the user that something went wrong, don't do it with a stack trace — print something informative to the user and continue running (if possible).

As a concession to simplify our testing of your code, you should always preface error messages with the string ERROR:. This will help us tell apart your error messages from the rest of your output, and will allow you to explain the error however you think is best, without requiring a perfect string match in the tests. Any line of output that begins with ERROR: will pass the tester in this case, because we don't specify all of the error edge cases — it's your job to find them. But we will look at your error and reward clearer messages.

A sample error handling case is here:

INPUT
wacky error-causing input!!
OUTPUT
ERROR:
END

If a test has line that is ERROR:, any number of consecutive "ERROR: ..." lines output by the program will be covered by this single line in the expected test output. That is, feel free to print multiple lines in certain error cases where it's warranted. Your test file will only specify that at least one error is expected.

Ignoring extra output

There may be some cases where you wish to log additional information from your program, but do not wish to include it in the expected output of your system tests. Any line prefixed with the string INFO: will be ignored by the system tester. That being said, it is good practice to keep debugging or extraneous information from cluttering your command line interfaces, so we do not expect your handins to print out numerous INFO messages while using the command line interface.

Useful Tips

Give your tests clear, informative filenames. It should be easy to understand all the cases your tests are covering just by listing out the files in your test suite directory. We will deduct points if your tests are named something along the lines of case1.test or error.test.

If all or some of your tests are failing because the output is empty, then there is a high chance that the timeout is too small and it is cutting off your program before it gets a chance to return the answer. Try a larger timeout — the exact value can vary from project to project and dataset to dataset.

If you want to test your entire suite you can run all tests in your directory using the wildcard path ./tests/*.test. However you can also just test a single test case by specifying it directly (e.g. ./tests/no-database-provided.test).