The LSAT Is Not What It Used to Be

Derek Muller explores recent changes in the test, how it is administered, and how it affects US News rankings.

The Law School Admission Test (LSAT) has undergone significant changes over the past two decades--changes in its composition, administration, and score reporting--that may be affecting its reliability as a predictor of law school performance. In addition, US News has changed its weighting and treatment of LSAT scores, altering the incentives for law schools to prioritize the LSAT scores of applicants, though it is not clear many law schools have altered their admissions practice accordingly (at least not yet).

Derek Muller explores these changes and their implications in a long and highly informative post at Excess of Democracy. It begins:

The LSAT is an important predictor of law school success. It does a very good job of predicting who will perform well in law school. The higher your LSAT score, the higher your law school grades are likely to be. It is not perfectly correlated, but it is well correlated. When combined with your undergraduate grade point average (UGPA)—yes, regardless of your major, grade inflation, school disparities, and all that—it can even further predict law school success.

But the LSAT has changed over the years. As has its weight in the USNWR rankings. Many law school admissions practices, however, look at the LSAT like it's 2005—like the test scores resemble what they did back then, and like the USNWR rankings care about them like they did back then. A lot has changed in a generation.

Muller summarizes some of these changes, the aggregate effect of which may be to reduce the LSAT's predictive value.

The LSAT, as a raw score, is less predictive of ability than it was 20 years ago. That is, a 170 or a 160 means less than it did 20 years ago. It may still be predictive in the aggregate. That is, a 170 means a higher likelihood of success than a 160. But there are error rates in that 170 that were unknown 20 years ago—the 170 likely overstates the "true" value compared to 20 years ago. Relatively speaking, and in terms of its validity as a statistical matter, it's still valuable—it just has a different value than before.

Likewise, schools have continued to rely on the LSAT but used it in a way that makes it less predictive than it is designed to be—by relying on the highest score, for instance, or by refusing to use the index score. This is exacerbated by the fact that LSAC allows more retakes than it did a generation ago, and it allows cancellation of scores in mechanisms unknown a generation ago.

More recent developments, including the acceleration of extra time test-takers and the dropping of logic games from the LSAT, promise to further dilute the predictive validity of the LSAT in yet-unknown ways.

For more detail, and a discussion of how law schools have (and perhaps should) respond to some of these changes, read the whole thing.