In industry, it is common to peer review code: that is, you present your code to other programmers, who critique it on design, correctness, structure, efficiency, and so on. In some organizations, peer-review is required before any code can be formally committed to the code base. These principles aren’t limited to industry: the benefits of peer-review are an argument for open source software, and for demands by security experts to make sources public. Finally, just about every programmer has had a friend come over and look at their code and provide feedback.
“Code”, in the sense of the executable part of the program, isn’t
the only artifact available to review. Many other parts, ranging from
design documents to data files, are also worthy of inspection. In this
class, we will emphasize the review of another important part of
programs: its examples (test cases for the interfaces defined in the
These examples can’t refer to implementation details, which may vary widely.
While people can reasonably disagree about implementation details, there ought to be much less difference of opinion about the expected behavior of the interfaces defined in the problem.
They are concrete. Therefore, they are often more accessible and easier to grasp.
They can usually be understood in isolation from each other. You can understand some without understanding them all, whereas different parts of source code are usually more tightly interlinked.
They might point to weaknesses in your own understanding of a problem!
As a reviewer, you will typically be given three submissions to review. We have found this number to strike a good balance between effort and exposure to a diversity of thinking about the problem.
Over time, especially when one assignment builds on another, the set of examples can grow quite large. Reviewing these becomes onerous to other students. They might also miss problems that you would have liked to hear about.
We have successfully tested a simple alternative. Instead of submitting an entire test suite, we will let you submit only a sweep (a term inspired by archeology and geology). The sweep is a set of 5 to at most 10 examples that are meant to be representative of the problem. That is, the sweep asks you to put your “best foot forward” rather than just dumping every example you can think of and forcing your reviewers to sort them out. Be pithy, not prolix.
Naturally, when limited to a sweep, you may no longer be able to cover the entire space of inputs and outputs. You should instead focus on a handful of standard ones and the ones that illustrate interesting corner cases. Sometimes it helps to add a comment to explain why you think a particular example is interesting (or present in your sweep).
Performing peer-review after an assignment has been submitted is a little like performing peer-review after a system has shipped: not only useless, but also frustrating when you do find a mistake or a way to make things better. Instead, therefore, we’re asking you to perform peer-review while an assignment is in progress.
To enable this, we want everyone to submit their examples within 48 hours of the assignment being published. Submissions will be assigned to reviewers as quickly as possible. Reviewers, in turn, must submit their reviews ideally within 24 hours. This way, both the reviewers and the authors will receive material with enough time to actually benefit from reading it.
Initially, we will have everyone serve as a reviewer and also receive reviews, to get you acquainted with the process.
Reviewers, who will write reviews (but not receive any).
Reviewees, who will receive reviews (but not write any).
Others, who will neither receive nor write reviews.
You will find out which group you’ve been assigned to after you submit your examples for review.
Once you’ve seen someone else’s excellent test case, you may find it
hard to resist copying it. That’s okay—
Be professional in your reviewing. Do we need to say more? Good.