Oracle

Sometimes, a where block just isn’t enough. When you are testing complex functions, or perhaps even relations (as in the case of this assignment), you will need to put time and effort into building testing oracles.

1 The Stable Marriage Problem

In this assignment, you will develop oracles to test purported solutions to the Stable Marriage Problem. You can learn more about the problem from Wikipedia, but a summary of the problem is as follows:

Assume you have two sets, A and B, each with n members. Every member in A wants to be matched with exactly one member of B and vice versa. Every member of each set ranks its preference for being matched with each member of the other set by assigning each one a unique number between 0 and n-1 (i.e., providing a total ordering of members of the other set).

As an application of this problem, imagine matching companies with candidates. Each company will keep an internal ranking of all the candidates, based on who they would prefer to hire (we assume that only one candidate will be hired per company). Each candidate also has an opinion about where they want to work and therefore ranks each company as well. A programmer is asked to design and implement a program that generates n pairings of n companies with n candidates. The program’s generated “hires” are stable if there are no two members, one from each set, that would prefer each other to their current match.

There are many problems of this nature. Consider assigning TAs to classes; matching residents with hospitals; pairing students for homeworks; and much more. Of course, some of these problems represent a slight variation on the theme (maybe the companies don’t rank every candidate, or are allowed to give some of them the same rank; maybe you have only partial information for making the assignment; etc.). Ultimately, however, this problem in its many guises has wide application.

2 The Assignment

Being confident that your software is correct involves more than just writing code you think is right. However, almost no software complex enough to be useful can be proved correct by hand in a reasonable amount of time. Naturally, a computer scientist’s solution to this problem is to write automated testing. Your job in this assignment is to build an automated testing oracle for a hypothetical solution to the stable marriage problem.

Your oracle’s job is to generate and feed test inputs to this solution, and test the correctness of the output. In the past, excluding Sortacle, you did this by comparing the output to a precomputed correct answer. This assumes two things: that there is only one right answer, and that it is easy for you to find it. In the real world either or both of these can and will be false. (How do you know what the right answer to an arbitrary instance of the problem is if the original problem was to write a program to find it?)

You’ve got your work cut out for you: we give you a correct solution to the stable marriage problem to help you write and test your oracle. You submit your oracle, and we unleash hell’s own fury of incorrect solutions at it, each with its own subtle flaw.It’s worth thinking about when we can conclude that we’re “done”. We ought to have multiple correct matchmakers because there are multiple correct outputs for each input. How many suffice?

You have access to Hire, hire, is-hire, and matchmaker from the support code, which is automatically imported in the template given below.

3 Input-Output Specification

Each purported solution will be a function like matchmaker that consumes two arguments, both of which are List<List<Number>> in which every list (both inner and outer, for both lists) is of the same size (call it n but n is naturally not fixed). The first list provides the companies’ preferences, the second one those for the candidates (though you should convince yourself it doesn’t matter which one is which). Each inner list corresponds to a specific candidate or company—identified by its index—and contains a list of n numbers, indexed from 0 to n-1, where each number refers to the index of members of the other group, and the list is sorted in order of preference from greatest to least. The function generates a set of hires, in which the numbers refer to the indices of the candidates and hires in the input lists.

To do well on this assignment, you will want to spend time considering all the different ways that output could be either invalid or inconsistent with the original problem statement. Be thorough! That’s the name of the game when testing.

Write a function named generate-input that generates input. Above, we have described only the shapes of the inputs; you will have to infer the constraints we’ve left out. Your function must generate a list of rankings for a specified number of candidates or companies.

fun generate-input(n :: Number) -> List<List<Number>>:

Using generate-input and is-valid, write a function named oracle that tests whether an algorithm is a valid solution to the stable marriage problem.

Remember, an algorithm may sometimes produce a correct solution even if it is an incorrect algorithm. Therefore, your oracle should try to only return true if an algorithm seems to always produce a stable set of hires (you will have to quantify “always” with some number of tests).

fun oracle(

a-matchmaker :: (List<List<Number>>, List<List<Number>>

-> Set<Hire>)

)

-> Boolean:

4 Built-Ins

Please see the Programming Language Use guide.

Even though, by this point in the course, you should be able to implement sets efficiently on your own (and you should do it; it’s a good exercise!), we are allowing you to use Pyret’s built-in Set library. Feel free to use any Set functions or methods, including list-to-list-set and .to-list.

5 Testing and Review

This assignment has an extra initial submission step, for the sweep.

You will submit 5-10 interesting test cases for is-valid by the sweep deadline. After you submit, you will be given three other students’ tests to review, which you should do within a day. (Note that you can submit both the tests and your reviews earlier than this, but the deadline makes sure that everyone gets feedback in time to use it before the assignment is due.)

To write the tests, start by copying the two template files below, and write your tests in oracle-sweep.arr. To submit them, first download just your tests by clicking "More / Download this file" from the menu at code.pyret.org, and save it as oracle-tests.arr. Then, visit:

https://www.captain-teach.org/brown-cs019/assignments/

and click on the oracle assignment. This will direct you to upload a file for tests. Select the oracle-sweep.arr file you just saved. You will have just one chance to submit your tests, because when you do they will be sent out for review by other students.

Right after submitting, the page should say that the submission was successful, and give you a “Continue” link. Click “Continue” and complete the three reviews of other students’ code. You can complete reviews in any order (and view them all at the same time), but you must complete all three reviews before moving on to the next step.

When you receive reviews from your classmates, you will also be given the ability to submit feedback on the review. This feedback is only for the course staff and has no effect on your grade. We’re interested in hearing if the review was particularly helpful or unhelpful, or if you think it was wrong. Also, if you feel the review you receive is in any way inappropriate, you can flag it to bring the problem to our attention.

6 Template Files

Initial Tests

Implementation Code

Final Tests

Note: Implementation-dependent testing should be in the implementation file. The final tests file should contain your tests for the three procedures we had you implement.

7 Handing In

Remember that this assignment requires you to hand in sweeps.

Note that there are additional instructions for handing in your initial tests above.

To submit your implementation, return to the Captain Teach assignments page:

https://www.captain-teach.org/brown-cs019/assignments/

and click “Next Step” again.

This time, first save and upload oracle-tests.arr, and then in the next step, do the same for oracle-code.arr.

In oracle-tests.arr, you can include as many tests as you want (beyond 10), and you can include tests you saw while reviewing (but copy with care!—and please do attribute the ones you copied to to peer-reviewing, even though you won’t be able to name the author).