Malcolm Gladwell Took the LSAT. But What Did We Learn?

Thoughts on Gladwell's recent podcast on legal education.

Malcolm Gladwell recently released a two-part podcast series on legal education that is focused on the LSAT.  Gladwell's basic argument, inspired by a 2004 law review article by Bill Henderson, is that LSATs (and many law school exams) unfairly penalize those who are tortoises instead of hares.  Hares work very quickly, while tortoises need more time.  Law schools need to give applicants more time, Gladwell suggests, to have a more fair system.

I found Gladwell's podcast entertaining.  But I also found it pretty frustrating.  I thought I would explain my frustrations in this post.

I. What is Gladwell's Argument?

My first problem was trying to figure out exactly what Gladwell finds objectionable. I tried to summarize Gladwell's take above.  But it was no easy task, as Gladwell's concerns seem to have an accordion-like quality. At times he objects just to the LSAT, and his concerns seemed narrow.  For example, Gladwell was very worked up about the fact that the LSAT is broken into sections and test-takers are not permitted to work on other sections during the test window.  His big proposed reform at another point was giving test-takers an additional 25% of time to take the LSAT.

And at other times, Gladwell's objections seemed quite broad. At some points he seemed concerned with time-pressured in-class law school exams, and at times he seems to object to timed exams more broadly.  Near the end, his main objection expands to the hierarchical nature of legal education generally. Gladwell is offended by the idea that the top-ranked law schools are very selective, and that therefore there must be some means to distinguish students in a competitive admissions process.  It's unfair to have an admissions hierarchy based on schools, Gladwell suggests. We shouldn't have a system where a test determines where you go to law school and therefore what job prospects you have.

I'm open to any or all of these arguments.  But they're different arguments, I think, and they need to be treated separately.  I found it frustrating that these arguments were too often mixed together.

II.  The Pieces Didn't Quite Fit

A second problem I had with the podcast was that the pieces didn't seem to fit together well.

Consider the question of Gladwell's own LSAT performance.  (Spoiler alert: Skip the next three paragraphs if you don't want to know more.) The hook in the first episode is that Gladwell and his assistant Camille both take the LSAT at the same time to see how they do.  The chief difference between them, we hear, is that Camille is a young hare and Gladwell is an old tortoise.  Camille does very well under quick time pressure, while Gladwell flails about under pressure and desperately needs more time.  The idea is that Gladwell's poor performance relative to Camille demonstrates just how unfair the LSAT is.

I appreciate the idea of humanizing the argument by having Gladwell struggle with the biased test himself. It makes for good listening.

But then it runs into a problem: At the end of the second episode, Camille and Gladwell get their scores back and they had identical scores.  Wait, if they had the same score, doesn't that suggest that Gladwell's personal complaints might have been misplaced?  The podcast doesn't go there, though.  Instead, it ends on Gladwell's happy reaction that he didn't do worse than Camille.

I was also puzzled by the discussion in the second episode about law firm performance.  Gladwell discusses an apparent study in which a consultant found that, among lawyers at a firm, the law school the lawyer attended had no influence on the chances that the lawyer would be a "rainmaker" --  someone who brought in a lot of business.  Gladwell takes this as proof that it's irrelevant where someone went to law school.  He suggests that law firms should ignore the school a person attended, as it's just irrelevant to how good that lawyer might be.

But I was confused by the study, assuming I understood it correctly.  Even assuming that firms are looking primarily to identify future rainmakers, it's problematic to use the performance of the lawyers a firm actually hired to draw lessons to figure out who the firm should have hired. If a law firm hires from the top 50% of the class at top school A, but only the top 10% of the class at regional school B, the ones who are hired aren't random selections from schools A and B.

I was also not sure what to make of Jeff Sutton's example.  This was the most fun part of the podcast for me: Judge Jeff Sutton makes an appearance!  Judge Sutton is used as an example of a tortoise that the law sadly overlooked.  We learn in the podcast that Sutton applied to Michigan and Ohio State for law school.  Sutton wanted to go to Michigan, but they turned him down.  So Sutton went to Ohio State, graduated first in his class, and then became one of the legal world's star advocates, a top judge, and a leading figure in the law.  Gladwell takes the lesson to be that the LSAT is a failure.  If Sutton didn't get into Michigan, it must have been because his LSAT score didn't recognize his talents.  The LSAT therefore needs reform.

Maybe, but maybe not.  A single example can help make a broader point.  But here we don't know the details of the example.  Maybe Judge Sutton just had a bad day.  Every test is flawed.  Is the existing LSAT more flawed than the alternatives?  And we also don't know why Sutton did less well on the LSAT (assuming that was the case) when he then did extremely well on other tests that require quick thinking like law school exams (remember, he graduated 1st in his class) and Supreme Court arguments (where he was a distinguished advocate).  Maybe the speed of the LSAT was the problem.  But there are a lot of other possible explanations.

To be clear, I'm not defending the LSAT.  I think there are big problems with it.  As I have written before, I think the "analytical reasoning" section (aka "games") should be eliminated. And I think law schools rely on the LSAT in a problematic way, fueled in large part by the U.S. News rankings.

But some proxies are needed, and no proxies are perfect.  The U.S. legal market is massive.  Every law firm can't hire every law student in the country for a spell, test them out, and rank order them to see who the firms want to hire.  Instead employers rely on proxies like law school and law school grades, and the schools in term rely on proxies like LSAT scores and college grades. The proxies are imperfect, and we always need better proxies. But there are complex choices and dynamics, with no ideal answer.  Gladwell's impressionistic approach seemed to glide over the difficulties in a way that wasn't as illuminating as I hoped.

III.  The Broader Question of Exam Timing

A final thought is about the broader question of law school test timing.   I think it's a difficult question for a few reasons.

First, there's the question of what set of skills best signal the quality of a top lawyer.  What schools test for should match the skills the best lawyers have, right?  Gladwell discusses this very briefly, but not nearly enough, and I was underwhelmed with his take on it.  He interviews a former Supreme Court law clerk who says that when she read briefs as a clerk, she read them very slowly and carefully.  Voila!, Gladwell says, working slowly and carefully is critical to being a great lawyer.  But there's so much more here, and it seemed a lost opportunity to reflect on what mix of skills lawyers need and how to test for them.

Second, there's the interesting question of whether having a long time to study for a short exam is testing long-term skills, short-term skills, or both.  Consider a law school in-class exam.  Yes, it is very time-pressured.  It rewards speed.  But the weeks and months of time before the exam also reward long-term effort.  You need to spend weeks of learning to be able to quickly see the legal significance of a problem.  I think that's testing both being a tortoise and being a hare. Maybe it's the wrong mix, see point one above.  But I don't think you can only look to the time it takes to complete the exam to see what skills are being tested by it.

One last thought about one specific timing objection Gladwell makes.  Gladwell strongly objects to the rule that LSAT takers can't work on other sections.  That's bizarre and unrealistic, he points out.  In the real world, you can distribute your allotted time as best you see fit.  I can see that, and at first that seemed like a very good argument.  But if I heard the podcast correctly, I think Gladwell may have cut off the LSAC employee as she started on a plausible explanation of why they structure the LSAT that way.

If I heard the employee's argument correctly -- a big "if," as the explanation was garbled and cut off early on --  the problem is that the LSAT has one experimental section that is not graded and is being used to test the quality of questions for future exams.   Applicants don't know which section is experimental.  But different test-takers are taking different experimental sections with different questions.  Some of the experimental questions are easy.  Some are hard. And some are problematic and will never be used.  (That's why they're just experimental.)

This means that different students are taking different exams, and giving the students the full exam at once might be problematic.  To see why, imagine the full exam were given at one shot and say you have two applicants to consider.  One applicant has a particularly easy experimental section, while the second has a particularly hard one.  In a time-pressured exam environment, the applicant with the easy experimental section may finish quickly and have more time to spend on other sections.  The applicant with the hard experimental section may take a lot of time and have less time for the rest of the exam.  That would be pretty unfair.

In contrast, if each section is given individually, with no room to work on other sections, it can lessen the unfairness.  The ease or difficulty of applicants' experimental sections may give their confidence a boost or hit, and that's no small matter.  But it won't impact the time they spend on the parts of the LSAT that are actually graded.

I don't know if that justifies the current system.  But it's a plausible argument, and I would have like to see it considered.

Comments

[bernard11]

My limited understanding of testing in general is that there is a tradeoff between testing for speed and testing for power.

A speed test, presumably like the LSAT, wants you to answer a lot of not wildly difficult questions in a limited time. Naturally, it requires a time limit.

Time limits are less appropriate for power tests, which pose fewer quite difficult questions. Here you are testing for the ability to solve the problem, and don't much care how long it takes. Getting the solution is paramount, and there will be great variation in how long it takes different people to get it.

This makes sense to me. I was an undergraduate math major. I recall times when I saw the way on a question immediately, and other times when I stumbled around filling sheets of paper with nonsense before the light came. I suspect that others in related subjects had similar experiences.

[James Pollock]

The LSAT is a computer-scored exam. This makes it easy to score (just feed it into the machine) but limits its effectiveness.
There are a number of ways to attack the test IN ADDITION TO knowing what the right answer is.

A better, more accurate testing method is to ask the question and provide empty white space for the candidate to answer the question. Either the candidate knows the answer, or they don't. However, these questions must be scored by a human being, and calibration is a possible problem.

[Michael]

In fairness to Gladwell, I don't think he could have covered everything in the detail you'd have wanted to see in 60 minutes of podcasting.

I think his basic argument is something like this:

Part I:
1. The LSAT is close to deterministic of what caliber of law school a student can attend.
2. The caliber of law school a student ends is close to deterministic regarding what type of legal career is possible. (If you didn;t go to Harvard, forget clerking for Scalia).
3. Thus, how one does on the LSAT decides what kind of legal career is possible. (This is my oversimplification of his oversimplified argument).

Part II:
1. Because it is a timed test that many people struggle to complete, and because part of preparing for it is developing test taking strategies for dealing with the limited time, the LSAT as it is evaluates a somewhat different set of abilities than it would do if time were not a factor. People would approach it differently, the relative performance of the test takers would be somewhat different, and the people admitted to the top law schools would be somewhat different.
2. One could argue that maybe the current LSAT *is* the best way to identify the top lawyers. Gladwell tries to rebut this with various arguments: the analogy to chess, the excellent performance (even relative to his fellow students) of a seemingly mediocre law student once the time constraint is removed, examples of legal work that is thorough and detail oriented rather than speed-oriented, rainmakers tend not to be from the top schools but they are certainly valuable, example of a lawyer who rose to the top of his profession without going to the top school/performing at an elite level on the LSAT, Scalia's favorite law clerk even though Scalia himself would never even have considered him.

Part III: I won't say much about his proposed solutions. He was throwing spaghetti at the wall.

If you look at it less as a proof of something, and more as generating a hypothesis that could be evaluated empirically, maybe you would find it less frustrating?

[Steven]

I appreciate your thoughtful assessment and I plan to listen to the podcast. I'm very alert to the tortoise/hare distinction -- I went to law school as a (much) older adult. I sought out classes with "tortoise" exams (e.g. 8-hour or longer essay exams), and my grades clearly reflected the difference: a full grade higher than in the classes with "hare" exams. One of my very young classmates had the opposite strategy, convinced that he could score near the top of the curve in any high pressure multiple choice exam. We were both right.

[James Pollock]

"I appreciate your thoughtful assessment and I plan to listen to the podcast. I’m very alert to the tortoise/hare distinction — I went to law school as a (much) older adult."

I did, too, but I am a "hare"... I do extremely well on standardized exams. It's why I don't place much weight on their accuracy. I was a top-ten-percent LSAT, top-third-of-my-class graduate. Then again, I was in the 4th percentile for undergraduate grades, so the LSAT was a better indicator of my ability to complete a JD program than my undergraduate grades were.

[BadLib]

Does the Supreme Court offer you extra time to respond to Justices' questions during oral arguments if you are a tortoise rather than a hare? Is there some sort of stamp on one's bar card to identify you as a ???? or a ?????

Oh, wait, that's not how it works in the real world...

[BadLib]

...and now we know that Reason comment section doesn't do Unicode emojis...

[Lee Moore]

1. Success in most fields of endeavor which have a significant cognitive component - which would include performance at law school, and performance as a lawyer, is correlated with (a) IQ and (b) conscientiousness. See thousands of psychometrc studies
2. For any given batch of questions, ability to get the answers right quickler is correlated with higher IQ
3. There are, obviously, more direct, valid and reliable tests of IQ than tests requiring a lot of detailed background knowledge
4. However no one has yet managed to devise a valid and reliable test for conscientiousness
5. If you can successfully devise one, you will make out like a bandit in the recruitment business
6. Thus we know that tortoises, on average, have lower IQs than hares
7. But some tortoises may be more conscientious than some hares and so may perhaps turn out to be better lawyers. But we have no way of predicting which ones, in advance
8. So until you come up with your winner under 5 above, the rational thing to do is to test for hares.

[James Pollock]

"2. For any given batch of questions, ability to get the answers right quickler is correlated with higher IQ"

This is open to question.

"7. But some tortoises may be more conscientious than some hares and so may perhaps turn out to be better lawyers."

Technically, the LSAT doesn't, and doesn't purport to, say anything about how candidates will perform as lawyers. It purports to evaluate whether candidates will successfully complete a JD program.

[Lee Moore]

This is open to question

Everything is open to question. Which is why we do experiments.
And so far the experiments are consistent with speed and IQ being correlated, and inconsistent with them not being correlated.

After another fifty years of testing, perhaps those who don't appreciate that will have caught up 🙂

[James Pollock]

Your extensive expertise on education standards and practices allow you to sum up your conclusion without actually offering any evidence that supports your claim.

Is that what you mean by being "caught up"?

[OldCurmudgeon]

"Technically, the LSAT doesn’t, and doesn’t purport to, say anything about how candidates will perform as lawyers. It purports to evaluate whether candidates will successfully complete a JD program."

That just begs the question, why do law schools exams rely so heavily on time pressure to differentiate students? Is there any evidence that lawyers who can write/type quickly make better lawyers?

My personal guess is that it's because it's hard to write/grade exams that actually test knowledge/skill, and professors are motivated to put forth that effort.

[OldCurmudgeon]

are not motivated

[Allutz]

Because without time pressures it often becomes difficult to distinguish between good students and great students, and also difficult to distinguish bad students from average students.

Also it overly rewards memorization and under-rewards systemization, which is what most people actually want to test for. Given lots of time all your students will be able to coherently tie together all their points into a 2 page answer. But with less time you reward people who have a structure of, say, property law already in their head.

[James Pollock]

" why do law schools exams rely so heavily on time pressure to differentiate students?"

The easy answers are "because the profs want it that way" and "because that's the way it's always been done".

Note that about a third of my classes either allowed a time limit that was generous enough with the time limit to not put pressure on, or didn't use an exam at all. (YMMV)

[M L]

I read his book Tipping Point and my personal opinion conclusion was that Malcom Gladwell is frankly a moron, well-meaning perhaps, but with very little real insight about anything. The one thing I liked and came away with was the bit about how birth date mattered so much for hockey players in Canada.

I'm a big fan of the LSAT. Maybe I'm biased because my score was above 175 (but below 180). But statistically, there's no metric that comes close to predictive value for law school performance (it blows UGPA out of the water). It seems like a very good test of analytic capability and general intelligence.

Anyway, this is nothing new. It's just part of the general push for lower standards toward political ends, which ends include a very destructive antipathy toward competition, objective achievement and American institutions themselves. Read this and click through to the lengthy critique of lowered standards for military special forces. Exclusive: Alleged Retaliation by Army General Against Green Berets Led a Military Family into a Moldy Home

[jubulent]

my personal opinion conclusion was that Malcom Gladwell is frankly a moron

His background is as a journalist, right? In my experience journalists tend to overestimate their expertise in a subject.

You shouldn't expect academic rigor from someone whose background and training is simplifying a complex subject into an easy-to-understand summary that can be understood by lay people in the industry.

Consider, for example, Nina Totenberg at NPR. She is a journalist covering the law but frequently gets some of the fine (or not so fine) points incorrect. Yet she is considered the resident expert on legal subjects, simply because she's been on the "law beat" for the last 40 years.

[James Pollock]

" In my experience journalists tend to overestimate their expertise in a subject."

Far from unique to journalists. Close to EVERYBODY overestimates their expertise.

[bernard11]

In my experience journalists tend to overestimate their expertise in a subject.

No doubt. Are you a lawyer?

[jubulent]

Yes, and I am happy to admit my many limitations when it comes to legal practice.

[phoqueue]

[The LSAT] seems like a very good test of analytic capability and general intelligence.

You know what else is even better, an IQ test. But that's taboo.

[James Pollock]

" this is nothing new. It’s just part of the general push for lower standards toward political ends, which ends include a very destructive antipathy toward competition"

Your telepathic sense allows you to state with authority what other people are thinking?

The LSAT, as you correctly note, is only partially accurate as a predictor of ability to complete a law school program. It overvalues the ability of some candidates and undervalues the ability of some. Pointing out who is unfairly disadvantaged by the present law-school admissions regime isn't necessarily part of some grand scheme to undermine law and society.

It's a fact that a week or two of prep time and coaching can raise an LSAT score substantially, which shows that it is NOT a very good analogue for general intelligence... it is at least partially a means test, since some candidates can afford prep and coaching and some cannot.

[RestoreWesternHegemony]

In my experience, only dumb people say that standardized tests are not good measures of anything.

[KenveeB]

I think that the LSAT is a good summation of law practice generally -- you spent large amounts of time preparing, but you have to be able to work quickly and get the product out at the deadline in the end. I might do hours and hours of prep on a case (like studying in law school classes), but ultimately I have a small amount of time to actually make my oral argument or trial (like taking the LSAT). Every lawyer has deadlines that they have to make, and sometimes that means moving more quickly than you would otherwise prefer -- or organizing your prep time better so that you can get it out quickly when it's crunch time.