Measuring the Effectiveness of Error Messages Designed for Novice Programmers
Guillaume Marceau, Kathi Fisler, Shriram Krishnamurthi
Scheme and Functional Programming, 2010
Abstract
Good error messages are critical for novice programmers. Many projects attempt to rewrite expert-level error messages in terms suitable for novices. DrScheme’s language levels provide a powerful alternative through which error messages are customized to pedagogically-inspired language subsets. Despite this, many novices still struggle to work effectively with DrScheme’s error messages. To better understand why, we have begun using human-factors research methods to explore the effectiveness of DrScheme’s error messages. Unlike existing work in this area, we study messages at a fine-grained level by analyzing the edits students make in response to various classes of errors. Our results point to several shortcomings in DrScheme’s current treatment of errors; many of these should apply to other languages. This paper describes our methodology, presents initial findings, and recommends new approaches to presenting errors to novices.
Comment
Compared to our SIGCSE 2011 paper, this one describes a broader set of experiments, but much more preliminary data in section 5. For more on the experiment in section 5, see the other paper.
Paper
These papers may differ in formatting from the versions that appear in print. They are made available only to support the rapid dissemination of results; the printed versions, not these, should be considered definitive. The copyrights belong to their respective owners.