Balancing Student Success and Inferring Personalized Effects in Dynamic Experiments

Abstract

Randomized controlled trials (RCTs) can be embedded in educational technologies to evaluate how interventions affect student outcomes and how effectiveness varies with characteristics like prior knowledge. But RCTs often assign many students to ineffective conditions. Adaptive algorithms like contextual multi-armed bandits (MABs) could change how students are assigned to conditions over time, offering the potential to both evaluate effectiveness for subgroups of students and direct more students to interventions that are effective for them. We use simulations to compare contextual MABs to traditional RCTs and non-contextual MABs. Contextual MABs improve outcomes for each subgroup; in contrast, non-contextual MABs may help one group of students, such as those with high prior knowledge, while hurting another. Because both MAB algorithms adaptively assign conditions based on prior students’ results, both recover biased estimates of condition effectiveness. However, data collected from a contextual MAB is still nearly as good for inferring the optimal assignment policy as from an RCT.

Publication
Proceedings of the 12th International Conference on Educational Data Mining
Arghavan Modiri
Arghavan Modiri
Graduate Student

My research interests include Artificial Intelligence, Machine Learning and Human-Computer Interaction.