When Offline Selectors Cannot Beat the Best Single Model
We had five pre-trained dropout predictors, and an oracle that picked the best one for each student would gain 9.7 accuracy points. Learned selectors ought to recover some of that gap. On 84.5M edX clickstream events, none did: behavior cloning, offline DQN, and CQL all failed to beat the best single model. The paper’s contribution is a three-stage diagnostic protocol (k-NN consistency, paired algorithm comparisons, and state ablation with TV-shift analysis) that explains why. The bottleneck is representational ambiguity: the state features don’t carry enough signal to tell which model will be right, so no selection algorithm can fix it.
I was first author and led the experimental design, evaluation infrastructure, and analysis. The work came out of graduate research at Georgia Tech, with Alan Nadelsticher Ruvalcaba, Dustin Khang LeDuc, Thomas Trask, Nicholas Lytle, and David Joyner.
Accepted at the ICML 2026 DEMO Workshop.