Optimal Allocation of Reviewers for Peer Feedback

TY - CONF

T1 - Optimal Allocation of Reviewers for Peer Feedback

AU - Aslak, Ulf

AU - Wind, David Kofoed

AU - Jørgensen, Rasmus Malthe

AU - Winther, Ole

PY - 2017/10/1

Y1 - 2017/10/1

N2 - Peer feedback is the act of letting students give feedback to each other on submitted work. There are multiple reasons to use peer feedback, including students getting more feedback, time saving for teachers and increased learning by letting students reflect on work by others. In order for peer feedback to be effective students should give and receive useful feedback. A key challenge in peer feedback is allocating the feedback givers in a good way. It is important that reviewers are allocated to submissions such that the feedback distribution is fair -meaning that all students receive good feedback. In this paper we present a novel way to intelligently allocate reviewers for peer feedback. We train a statistical model to infer the quality of feedback based on a dataset of feedback quality evaluations. This dataset contains more than 20,000 reviews where the receiver of the feedback has indicated the quality of the feedback. Using this model together with historical data we calculate the feedback-giving skill of each student and uses that as input to an allocation algorithm that assigns submissions to reviewers, in order to optimize the feedback quality for all students. We test the performance of our allocation strategy using real data from over 600 peer feedback sessions and simulate the effects of different allocation strategies. By comparing our method with a random allocation algorithm and a "super-informed oracle" algorithm we demonstrate that we are able to allocate reviewers to submissions in such a way that all submissions receive feedback of similar quality and that we are able to significantly outperform simple random allocation of reviewers. Additionally we investigate the effect of pre-allocating reviews in comparison to allocating reviewers live during the review process and show that live-allocation leads to better results. Our method is robust to reviews not being completed and other real-life quirks and improves as more feedback data is collected.

AB - Peer feedback is the act of letting students give feedback to each other on submitted work. There are multiple reasons to use peer feedback, including students getting more feedback, time saving for teachers and increased learning by letting students reflect on work by others. In order for peer feedback to be effective students should give and receive useful feedback. A key challenge in peer feedback is allocating the feedback givers in a good way. It is important that reviewers are allocated to submissions such that the feedback distribution is fair -meaning that all students receive good feedback. In this paper we present a novel way to intelligently allocate reviewers for peer feedback. We train a statistical model to infer the quality of feedback based on a dataset of feedback quality evaluations. This dataset contains more than 20,000 reviews where the receiver of the feedback has indicated the quality of the feedback. Using this model together with historical data we calculate the feedback-giving skill of each student and uses that as input to an allocation algorithm that assigns submissions to reviewers, in order to optimize the feedback quality for all students. We test the performance of our allocation strategy using real data from over 600 peer feedback sessions and simulate the effects of different allocation strategies. By comparing our method with a random allocation algorithm and a "super-informed oracle" algorithm we demonstrate that we are able to allocate reviewers to submissions in such a way that all submissions receive feedback of similar quality and that we are able to significantly outperform simple random allocation of reviewers. Additionally we investigate the effect of pre-allocating reviews in comparison to allocating reviewers live during the review process and show that live-allocation leads to better results. Our method is robust to reviews not being completed and other real-life quirks and improves as more feedback data is collected.

M3 - Paper

ER -