US News Makes Beneficial Changes to its Law School Rankings System

Downgrading reputational surveys and abolishing points for per-student expenditures are steps in the right direction.

As Jonathan Adler points out, the Wall Street Journal has an article on how US News and World Report is planning to make changes to its influential law school rankings system:

In a letter sent Monday to deans of the 188 law schools it currently ranks, U.S. News said it would give less weight in its next release to reputational surveys completed by deans, faculty, lawyers and judges and won't take into account per-student expenditures that favor the wealthiest schools. The new ranking also will count graduates with school-funded public-interest legal fellowships or who go on to additional graduate programs the same as they would other employed graduates.

The motive for the changes may well be the revolt against US News undertaken by numerous high-ranking schools which have announced they will no longer cooperate with the rankings by providing information used in the rankings. The elements downgraded or eliminated are mainly ones US News depends on schools to provide. Faculty and deans at non-participating schools may now be less likely to fill out the reputational surveys (though, as far as I can tell, no law school has actually forbidden such participation by its faculty). And US News relies on schools to provide expenditure data.

But, whatever the motive, these are changes for the better. I have long criticized the use of expenditure data because it effectively rewards wasteful spending. As I explained way back in 2006:

[I]f School A and School B are exactly equal in the quality of their students, faculty, facilities, etc., but School A spends twice as much money per student to get this result as School B, then A will come out well ahead in the US News rankings. A is actually rewarded for being far less efficient in getting educational value for its money than B! Thus, the US News system gives schools an incentive to engage in wasteful expenditures. This is particularly unfortunate in the case of public law schools, where some of the funds expended are taxpayer money. And even private law schools receive many direct and indirect government subsidies as well.

There is no reason to believe that including expenditures provides useful information to applicants or others interested in the school's quality. To the extent that the money the school spends translates into real improvements in quality, these can be measured directly by including ratings for the quality of the faculty, student body, and facilities. Many of these factors are already included in the US News formula and the rest certainly can and should be….

Good riddance to this perverse incentive!

The reputational surveys—which are currently the most heavily weighted components of the US News system (25% for the law school faculty survey and 15% for the survey of lawyers and judges) are also badly flawed. The problem is that most of those surveyed have little idea of what goes on at most schools.

There are currently some 190 American Bar Association-accredited law schools in the US. Most legal scholars don't have time to keep close track of more than a small fraction of them. The same is even more true of most lawyers and judges.

I'm a law professor myself. If I tried to keep track of various indicators of student and faculty quality at more than a small fraction of all the law schools in the United States, it would be almost a full-time job. At the very least, it would  divert much valuable time and energy away from teaching and scholarship. That would obviously be a poor trade-off for my career prospects. But it would also be bad for my institution (which is why schools don't usually reward faculty for developing expertise about other law schools). I keep track of scholarship in my fields, and therefore have a fairly good idea of which schools have faculty doing important work on issues I write about myself. But I don't know much about what happens at most other schools beyond that.

In effect, I am "rationally ignorant" about most of what goes on at the vast majority of law schools. That is also true of most other legal scholars, and also most lawyers and judges. They too have competing demands on their time that (in most cases) preclude keeping close track of the strengths and weaknesses of 190 law schools. At best, lawyers and judges have incentives to have some knowledge of a few top national schools, plus a few in their region whose graduates they regularly encounter.

Because most US News survey respondents are rationally ignorant about most of the schools they are supposed to rate, the rankings they give are heavily influenced by crude shortcuts and stereotypes, such as rating schools based on the general prestige of the universities they are attached to. The system also discriminates against relatively new schools and those whose quality has improved rapidly in ways poorly informed observers are likely to miss. By contrast, it benefits declining institutions who are living off of old reputational capital.

My only potential reservation about this change is that it's not clear how much the reputational ratings will be downgraded. Ideally, they should be abolished entirely. But even a moderate reduction in their significance will be an improvement.

I am a bit more tentative in my assessment of the third change US News plans to make: counting graduates with public-interest fellowships as employed, and doing the same for those who go on to additional graduate education. On balance, this seems a good change. However, during the 2008-09 recession and its aftermath, some schools tried to game the system by giving graduates fellowships of questionable career value, in order to boost their employment numbers. This seems like a self-limiting problem, because most schools can't afford to hand out more than a relatively small number of such Potemkin Village fellowships. But it's still an issue US News should consider.

Whether the changes made by US News placate any of the rebelling schools remains to be seen. There is only modest overlap between the changes and the complaints raised by Yale Law School Dean Heather Gerken in her letter that kicked off the revolt against the US News rankings.

In addition to raising the fellowship issue (on which US News has now  moved in her direction), she also urged downgrading of the role of LSAT scores. US News' decision to instead downgrade reputational surveys and eliminate per-student expenditures is likely to actually make LSAT scores a bigger component of the ranking system.

While I am far from the biggest fan of the LSAT, on balance I think it plays a useful role in rankings and law school admissions. Schools need some objective metric to compare students from hundreds of different undergraduate institutions. Absent the LSAT (or some other standardized test), this is extremely difficult to do.

Moreover, removing standardized tests from the equation (or greatly downgrading them) will actually favor graduates of more elite undergraduate institutions. How can an applicant from, say, the University of Idaho, prove he or she is as good as one from Harvard? Even if she has a similar or better GPA, many admissions officers will be skeptical of that metric, because it's hard to say whether the classes she took were as rigorous. Plus, she might have been competing against weaker classmates than the Harvard grad. The LSAT helps level that playing field. And it is a strong predictor of academic success in law school (though admittedly still far from perfect).

In principle, the LSAT can be replaced with some other, better, standardized test (I don't doubt it has significant flaws). Alternatively, law schools could develop better ways to compare GPAs across undergraduate schools that take  account of the differences between them. But the latter is likely to require costly research and data collection that—so far, at least—no one has been willing to do.

In sum, US News' changes are a step in the right direction. That's true regardless of their motives, and regardless of whether they succeed in placating rebel law schools. Ultimately, US News (and other ranking producers) don't need law schools' endorsement to do rankings based on publicly available data. And applicants don't need that endorsement to use the rankings to help make their decisions.

And that's a good thing! The law school industry (myself included!) should no more enjoy veto power over law school rankings systems than the auto industry should have veto power over institutions that rank the quality of various types of cars. In both cases, there is an obvious conflict of interest.

Of course, that doesn't mean applicants should rely solely on US News. Other rankings—most notably the Leiter ranking of student and faculty quality—also have useful information (faculty quality continues to be a notable omission in the US News system). And there are some factors relevant to applicant decision-making that no existing ranking system covers well, such as the ways in which going to a particular school can advance your career in a particular region (as opposed to nationally). If you really want to practice law in Montana, it may well make sense to choose the University of Montana—with its many connections to influential lawyers and judges in the state—over many schools with higher national ratings.

The US News system still has flaws and limitations. But the changes revealed today will likely make it better than before.

NOTE: As legal scholar Derek Muller points out in his analysis of the US News changes, my own institution (George Mason University, currently ranked no. 30), is likely to be among the biggest beneficiaries of the downgrading of reputation scores relative to other metrics. I think we will also benefit, at the margin, from the exclusion of per-student expenditures. If readers believe this makes me a biased observer, I can understand that. Ultimately, few if any law professors can be completely unbiased about law school rankings systems. I will only say that the merit (or lack thereof) of my arguments can and should be assessed independently of my motives for making them.