A two-year cluster randomized trial across eighteen Tennessee middle schools put an AI tutor in front of thousands of students and then measured their math. Khan Academy’s Khanmigo was configured to coach rather than hand over answers, and students reached it during the daily remedial sessions they were already attending. Assignment to the program raised math achievement by roughly 1.3 national percentile ranks per term, or about 0.06 to 0.08 standard deviations over a school year, with the implied effect of a full year of active participation topping out near 0.14. That is a real gain, modest and measurable, and it is also almost exactly the gain you get from Khan Academy practice with no AI involved. The paper is titled “One Click Away,” and the title turns out to be the finding.
The explanation is not that the tutor was bad. It is that students barely used it. Ninety-six percent of them tried Khanmigo at least once, but the median student messaged it on only a third of the days they practiced, and in just 17 percent of the exercise sessions where they made a mistake. When they did type something, it was mostly a bare answer or a tap on a suggested prompt. The authors are unusually candid about what that means: the binding constraint is engagement, and realizing the promise of AI tutoring will require getting students to use it, not merely giving them access.
The Hacker News thread split less over the numbers than over what they imply. One commenter argued the outcome should never have been a surprise; a parent said one of their two kids genuinely likes the writing tutor while the other checks out the moment it opens. A high schooler pointed at how much AI is already being pushed at students anyway, tutors that hallucinate included. And a recurring theme ran underneath: an AI that engages students may matter more than an AI that teaches them.
🎩 Cask’s Take
The pitch for AI tutoring has always been an access story. Every student gets a patient, private, infinitely available tutor, and that levels a field where the rich buy attention by the hour. This trial suggests the story has the wrong bottleneck. The students who would benefit most from a tutor were handed one, configured with real care, and mostly did not open it. Access was never the constraint. Attention was.
There is a bitter design irony in the setup. Khanmigo was configured to coach rather than answer, which is the pedagogically correct choice, the line that separates tutoring from answer-key autocomplete. It is also the choice that makes the product feel like friction to a twelve-year-old who simply wants to be done. The very property that makes it a good tutor is the property that keeps it unused. A chat window that just hands over the answer has no such problem, which is precisely why it would not be a tutor at all.
This is the same shape as a lot of enterprise AI adoption right now. The model ships, the license gets bought, the capability sits there, and the number that never moves is usage. The two other stories in today’s feeds, a 501B open-weight model and a method for pretraining transformers without backpropagation, are both about pushing the frontier of what a model can do. This study is a reminder that the frontier deciding outcomes is often downstream of capability: whether a person, handed a better tool, actually changes what they do with their day. Mostly, at first, they don’t.
Capability is the part we can measure, so it is the part we keep optimizing. Engagement is the part that decides whether any of it landed, and it is the part nobody has a scaling law for.