Testing: General Principles

All functions must have tests that exercise non-trivial cases. We require this to encourage good software development skills, but, more importantly, to force you to show us that you know what your code is supposed to do and that you’re not getting a correct answer by guessing. We will not give full credit to untested functionality, even if it is correct!

Good testing:

TestFest

Important: We do not anticipate using TestFest this semester, but we are leaving these instructions here in case we do.

Test-fest is a collaborative testing system that we use for select assignments. You upload both programs and tests. Your tests test others' programs, and vice versa.

You upload your tests in the form of a test suite. A test suite is a listing of test and test/exn expressions (including any helper functions). For example: 

(test (calc (interp '{+ 2 3})) 5)
(test/exn (calc (interp '{with {x 6} y})) "unbound identifier")

When you upload a test suite, it is run against our TA-written solution. If it passes, we add it to a global pool of test suites. (If it fails we notify you that it is buggy; you may contest our claim, claiming that your test suite is correct and that the TAs have a buggy solution.)

When you upload a solution to the assignment, it runs against all test suites and reports results back to you.

What are the time/memory limits imposed by TestFest?

TestFest runs your submitted code in a sandbox with limited resources. Your submissions have 150MB of RAM and 50 seconds to run on each test suite. (We will adjust these if necessary.)

How often can I submit to TestFest?

You can submit both test suites and solutions as often as you like. However, the night before assignments are due, the system is usually overloaded and slow. So, start and finish your homework early!

When I submit a new test suite, what happens to my old test suite?

If you submit a new test suite that is accepted by the system, your old test suite is removed from the global pool of tests. If you submit a new test suite that has an error, it is rejected and the old test suite remains in the pool of tests.

We expect you to grow, refine, and organize your test suite over time. Periodically, you should submit your entire test suite to TestFest, without worrying about which test are new, which tests changed, if tests were reordered, etc.

How should I test helper functions?

You should test your own helper functions. However, since other groups might use different helper functions, you cannot submit them to TestFest. TestFest runs your test suite on a TA-written solution before accepting it. This TA-written solution does not have any helper functions, so tests that try to directly call them will fail (with an unbound identifier error).

What error messages should I use with test/exn?

Your assignments in CS173 do not specify the exact error messages your programs should report. Therefore, different groups may use different error messages for the same error:

(test/exn (interp (parse 'x)) "unbound identifier")
(test/exn (interp (parse 'x)) "unbound variable")

TestFest ignores error messages and simply checks for errors. You can use precise error messages to help debug your own code without worrying about compatiblity with others' error messages.

TestFest does not support Internet Explorer.

Yes, TestFest does not support Internet Explorer.

How does TestFest affect grades?

You are graded on your final solution and test suite, not on code submitted to TestFest. However, while grading, we will test your solution using others' tests in TestFest in addition to reviewing your code manually.

You will receive additional points for catching bugs in others' solutions via TestFest. However, you will not lose points for submitting bugging solutions to TestFest. Therefore, TestFest can only increase your grade, not lower it.