The Volokh Conspiracy

Mostly law professors | Sometimes contrarian | Often libertarian | Always independent

Politics

"Can AI Ace Your Exam? A Penn Law Experiment," by Prof. R. Polk Wagner

"On two exams, both scored on an open, non-saturated scale, it outscored every enrolled student."

|

A very interesting preprint; here's the Abstract:

I report a controlled experiment that ran a frontier large language model through eleven real University of Pennsylvania Carey Law School final exams in Spring 2026 and sent the machine's answers out for blind grading alongside the work of enrolled students. Each exam was answered twice: once with no course materials in the model's context (the "no-materials" arm, relying on the model's training alone) and once with the course syllabus and an outline supplied (the "with-materials" arm). Three findings emerge.

First, the model is broadly competitive: across the ten exams returned with grades, it passed every one, sat at or above the class median on every one, and reached the top decile on seven. On two exams, both scored on an open, non-saturated scale, it outscored every enrolled student.

Second, the benefit of supplying course materials is real but concentrated and uneven. It is large on multiple-choice components, small to absent on essays, and on one exam the materials helped the machine's multiple-choice score while hurting its essay—so the gain vanishes when the two are averaged into a single number. The subject of the exam matters as much as its format.

Third, faculty graders detected the AI submissions at a high rate, but mostly through signals that are easy to remove: formatting artifacts, identification-number placement, even a reversed quotation-mark glyph. The detection that survives those fixes is content-based—two independently generated answers that read as though written by the same hand, and a voice that does not match time-pressured student work. One grader was confident enough in detection to decline to grade the AI answers at all.

I read these results less as a verdict than as a measurement problem: when a model that never took the course can top the curve, it is no longer clear that the time-limited, high-stakes final is measuring what we have long assumed it measures—mastery of this course—rather than fluency in a genre the machine commands. That genre was always a contrivance built for grading, not a faithful model of practice, so the machine's success says less about lawyering than the headline suggests; but it says a good deal about how much weight the instrument can still bear.

The urgent ask the experiment surfaces is intentionality: that faculty be clear about what each assessment is meant to measure, and recognize that AI is widening an already-real gap between the law-school exam and the practice it stands in for. The responses will range—from rethinking assessment top to bottom, to narrower adjustments, to simply naming the mismatch and grading with it in view—and reasonable colleagues will differ