Best student paper award at the 2023 ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO’23) was “30 Million Canvas Records Reveal Widespread Sequential Bias and System-design Induced Surname Initial Disparity in Grading” authored by Jiaxin Pei, Zhihan Wang, and Jun Li.
The widespread adoption of learning management systems in educational institutions has yielded numerous benefits for teaching staff but also introduced the risk of unequal treatment towards students. We present an analysis of over 30 million Canvas grading records from a large public university, revealing a significant bias in sequential grading tasks. We find that assignments graded later in the sequence tend to
(1) receive lower grades,
(2) receive comments that are notably more negative and less polite, and
(3) exhibit lower grading quality measured by post-grade complaints from students.
Despite all the benefits and conveniences it brings about, Learning Management Systems platforms can create a serious discrimination against students with later initials because of the seemingly harmless surname sorting.
Furthermore, we show that the system design of Canvas, which pre-orders submissions by student surnames, transforms the sequential bias into a significant disadvantage for students with alphabetically lower-ranked surname initials. These students consistently receive lower grades, more negative and impolite comments, and raise more post-grade complaints as a result of their disadvantaged position in the grading sequence. This surname initial disparity is observed across a wide range of subjects, and is more prominent in social science and humanities as compared to engineering, science and medicine. The assignment-level surname disparity aggregates to a course-level surname disparity of students’ GPA and can potentially lead to inequitable job opportunities. For platforms and education institutions, the system-induced surname grading disparity can be mitigated by randomizing student submissions in grading tasks. Education institutions should keep the workload of graders at a reasonable level to reduce fatigue and/or have multiple graders as a cross validation to enhance grading quality.
There are multiple interesting papers presented at the conference and available at the EAAMO ’23: Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization.
However, the student paper is under review by the journal Management Science and
currently available as a working paper, to be found at SSRN.
Surtil : 