Malcolm Gladwell Took the LSAT. But What Did We Learn?

Thoughts on Gladwell's recent podcast on legal education.

Malcolm Gladwell recently released a two-part podcast series on legal education that is focused on the LSAT.  Gladwell's basic argument, inspired by a 2004 law review article by Bill Henderson, is that LSATs (and many law school exams) unfairly penalize those who are tortoises instead of hares.  Hares work very quickly, while tortoises need more time.  Law schools need to give applicants more time, Gladwell suggests, to have a more fair system.

I found Gladwell's podcast entertaining.  But I also found it pretty frustrating.  I thought I would explain my frustrations in this post.

I. What is Gladwell's Argument?

My first problem was trying to figure out exactly what Gladwell finds objectionable. I tried to summarize Gladwell's take above.  But it was no easy task, as Gladwell's concerns seem to have an accordion-like quality. At times he objects just to the LSAT, and his concerns seemed narrow.  For example, Gladwell was very worked up about the fact that the LSAT is broken into sections and test-takers are not permitted to work on other sections during the test window.  His big proposed reform at another point was giving test-takers an additional 25% of time to take the LSAT.

And at other times, Gladwell's objections seemed quite broad. At some points he seemed concerned with time-pressured in-class law school exams, and at times he seems to object to timed exams more broadly.  Near the end, his main objection expands to the hierarchical nature of legal education generally. Gladwell is offended by the idea that the top-ranked law schools are very selective, and that therefore there must be some means to distinguish students in a competitive admissions process.  It's unfair to have an admissions hierarchy based on schools, Gladwell suggests. We shouldn't have a system where a test determines where you go to law school and therefore what job prospects you have.

I'm open to any or all of these arguments.  But they're different arguments, I think, and they need to be treated separately.  I found it frustrating that these arguments were too often mixed together.

II.  The Pieces Didn't Quite Fit

A second problem I had with the podcast was that the pieces didn't seem to fit together well.

Consider the question of Gladwell's own LSAT performance.  (Spoiler alert: Skip the next three paragraphs if you don't want to know more.) The hook in the first episode is that Gladwell and his assistant Camille both take the LSAT at the same time to see how they do.  The chief difference between them, we hear, is that Camille is a young hare and Gladwell is an old tortoise.  Camille does very well under quick time pressure, while Gladwell flails about under pressure and desperately needs more time.  The idea is that Gladwell's poor performance relative to Camille demonstrates just how unfair the LSAT is.

I appreciate the idea of humanizing the argument by having Gladwell struggle with the biased test himself. It makes for good listening.

But then it runs into a problem: At the end of the second episode, Camille and Gladwell get their scores back and they had identical scores.  Wait, if they had the same score, doesn't that suggest that Gladwell's personal complaints might have been misplaced?  The podcast doesn't go there, though.  Instead, it ends on Gladwell's happy reaction that he didn't do worse than Camille.

I was also puzzled by the discussion in the second episode about law firm performance.  Gladwell discusses an apparent study in which a consultant found that, among lawyers at a firm, the law school the lawyer attended had no influence on the chances that the lawyer would be a "rainmaker" --  someone who brought in a lot of business.  Gladwell takes this as proof that it's irrelevant where someone went to law school.  He suggests that law firms should ignore the school a person attended, as it's just irrelevant to how good that lawyer might be.

But I was confused by the study, assuming I understood it correctly.  Even assuming that firms are looking primarily to identify future rainmakers, it's problematic to use the performance of the lawyers a firm actually hired to draw lessons to figure out who the firm should have hired. If a law firm hires from the top 50% of the class at top school A, but only the top 10% of the class at regional school B, the ones who are hired aren't random selections from schools A and B.

I was also not sure what to make of Jeff Sutton's example.  This was the most fun part of the podcast for me: Judge Jeff Sutton makes an appearance!  Judge Sutton is used as an example of a tortoise that the law sadly overlooked.  We learn in the podcast that Sutton applied to Michigan and Ohio State for law school.  Sutton wanted to go to Michigan, but they turned him down.  So Sutton went to Ohio State, graduated first in his class, and then became one of the legal world's star advocates, a top judge, and a leading figure in the law.  Gladwell takes the lesson to be that the LSAT is a failure.  If Sutton didn't get into Michigan, it must have been because his LSAT score didn't recognize his talents.  The LSAT therefore needs reform.

Maybe, but maybe not.  A single example can help make a broader point.  But here we don't know the details of the example.  Maybe Judge Sutton just had a bad day.  Every test is flawed.  Is the existing LSAT more flawed than the alternatives?  And we also don't know why Sutton did less well on the LSAT (assuming that was the case) when he then did extremely well on other tests that require quick thinking like law school exams (remember, he graduated 1st in his class) and Supreme Court arguments (where he was a distinguished advocate).  Maybe the speed of the LSAT was the problem.  But there are a lot of other possible explanations.

To be clear, I'm not defending the LSAT.  I think there are big problems with it.  As I have written before, I think the "analytical reasoning" section (aka "games") should be eliminated. And I think law schools rely on the LSAT in a problematic way, fueled in large part by the U.S. News rankings.

But some proxies are needed, and no proxies are perfect.  The U.S. legal market is massive.  Every law firm can't hire every law student in the country for a spell, test them out, and rank order them to see who the firms want to hire.  Instead employers rely on proxies like law school and law school grades, and the schools in term rely on proxies like LSAT scores and college grades. The proxies are imperfect, and we always need better proxies. But there are complex choices and dynamics, with no ideal answer.  Gladwell's impressionistic approach seemed to glide over the difficulties in a way that wasn't as illuminating as I hoped.

III.  The Broader Question of Exam Timing

A final thought is about the broader question of law school test timing.   I think it's a difficult question for a few reasons.

First, there's the question of what set of skills best signal the quality of a top lawyer.  What schools test for should match the skills the best lawyers have, right?  Gladwell discusses this very briefly, but not nearly enough, and I was underwhelmed with his take on it.  He interviews a former Supreme Court law clerk who says that when she read briefs as a clerk, she read them very slowly and carefully.  Voila!, Gladwell says, working slowly and carefully is critical to being a great lawyer.  But there's so much more here, and it seemed a lost opportunity to reflect on what mix of skills lawyers need and how to test for them.

Second, there's the interesting question of whether having a long time to study for a short exam is testing long-term skills, short-term skills, or both.  Consider a law school in-class exam.  Yes, it is very time-pressured.  It rewards speed.  But the weeks and months of time before the exam also reward long-term effort.  You need to spend weeks of learning to be able to quickly see the legal significance of a problem.  I think that's testing both being a tortoise and being a hare. Maybe it's the wrong mix, see point one above.  But I don't think you can only look to the time it takes to complete the exam to see what skills are being tested by it.

One last thought about one specific timing objection Gladwell makes.  Gladwell strongly objects to the rule that LSAT takers can't work on other sections.  That's bizarre and unrealistic, he points out.  In the real world, you can distribute your allotted time as best you see fit.  I can see that, and at first that seemed like a very good argument.  But if I heard the podcast correctly, I think Gladwell may have cut off the LSAC employee as she started on a plausible explanation of why they structure the LSAT that way.

If I heard the employee's argument correctly -- a big "if," as the explanation was garbled and cut off early on --  the problem is that the LSAT has one experimental section that is not graded and is being used to test the quality of questions for future exams.   Applicants don't know which section is experimental.  But different test-takers are taking different experimental sections with different questions.  Some of the experimental questions are easy.  Some are hard. And some are problematic and will never be used.  (That's why they're just experimental.)

This means that different students are taking different exams, and giving the students the full exam at once might be problematic.  To see why, imagine the full exam were given at one shot and say you have two applicants to consider.  One applicant has a particularly easy experimental section, while the second has a particularly hard one.  In a time-pressured exam environment, the applicant with the easy experimental section may finish quickly and have more time to spend on other sections.  The applicant with the hard experimental section may take a lot of time and have less time for the rest of the exam.  That would be pretty unfair.

In contrast, if each section is given individually, with no room to work on other sections, it can lessen the unfairness.  The ease or difficulty of applicants' experimental sections may give their confidence a boost or hit, and that's no small matter.  But it won't impact the time they spend on the parts of the LSAT that are actually graded.

I don't know if that justifies the current system.  But it's a plausible argument, and I would have like to see it considered.