Iterative Student Program Planning using Transformer-Driven Feedback
Elijah Rivera, Alexander Steinmaurer, Kathi Fisler, Shriram Krishnamurthi
Conference on Innovation and Technology in Computer Science Education, 2024
Abstract
Problem planning is a fundamental programming skill, and aids students in decomposing tasks into manageable subtasks. While feedback on plans is beneficial for beginners, providing this in a scalable and timely way is an enormous challenge in large courses.Recent advances in LLMs raise the prospect of helping here. We utilize LLMs to generate code based on students’ plans, and evaluate the code against expert-defined test suites. Students receive feedback on their plans and can refine them.
In this report, we share our experience with the design and implementation of this workflow. This tool was used by 544 students in a CS1 course at an Austrian university. We developed a codebook to evaluate their plans and manually applied it to a sample. We show that LLMs can play a valuable role here. However, we also highlight numerous cautionary aspects of using LLMs in this context, many of which will not be addressed merely by having more powerful models (and indeed may be exacerbated by it).
Paper
These papers may differ in formatting from the versions that appear in print. They are made available only to support the rapid dissemination of results; the printed versions, not these, should be considered definitive. The copyrights belong to their respective owners.