LSAT to Drop the "Logic Games" Section

Good riddance.

Back in 2007, I wrote a Volokh post, Should the LSAT Have A "Logic Games" Section?. arguing that the Law School Admissions Test (LSAT) should drop the logic games section because it tested abilities that didn't relate to work as a lawyer:

I confess I don't understand why the LSAT has a "games" section (aka "analytical reasoning"). This section tests an ability to understand relationships among a handful of variables and to see the different ways that different combinations of those variables can fit different criteria. The skill set seems to be keeping a lot of variables in mind and working with how a change in the boundaries of a problem changes how the different pieces can relate to each other. That is an important skill set in many professions, to be sure; it's something that I did all the time when I was in engineering graduate school. But I wonder, how important is that skill to either the study or the practice of law? What kinds of legal tasks rely heavily on that skill?

The Law School Admission Council's report on the history of different LSAT questions explains that the purpose of these questions is "to understand the structure of a relationship," and claims that they "represent the kind of detailed analyses necessary in solving legal problems."(p.8) But I don't see why. (The report cites a 1993 study, but I couldn't find it online.) It's not clear to me that this particular kind of reasoning is directly relevant to either the study or practice of law.

Some Volokh Conspiracy posts change the world immediately, while others simmer for a while. This one took sixteen years, apparently, as the people who administer the LSAT just announced the following, via Reuters:

The Law School Admission Test will ditch the so-called "logic games" section of the exam in 2024, according to the organization that creates the test, marking a major change to the exam's content.

The change means that perplexing questions such as who gets which meal at a dinner party if Mary has a fish allergy, Devin doesn't eat gluten and Jamal prefers organic will no longer be part of the test.

The Law School Admission Council (LSAC), which develops and administers the test, sent an email on Wednesday to U.S. law schools, which was reviewed by Reuters, notifying them of the change.

Seriously, glad to see this.  I don't think the games section should have ever been part of the test, and it ended up a silly barrier to entry to many that only distorted the admissions process.  Good riddance.

Comments

[Bob from Ohio]

Wait til you see what replaces it. Bound to be worse.

[Bored Lawyer]

The change means that perplexing questions such as who gets which meal at a dinner party if Mary has a fish allergy, Devin doesn't eat gluten and Jamal prefers organic will no longer be part of the test.

OK. But who gets which meal?

[Á àß äẞç ãþÇđ âÞ¢Đæ ǎB€Ðëf ảhf]

Mary has a fish allergy, Devin doesn’t eat gluten and Jamal prefers organic

Assuming no liars ... Mary's claim is definitive and a priority. Devin's choice is his alone, so he can go hungry without affecting anybody else. Jamal has a preference which he can presumably forego for one meal.

[Bored Lawyer]

You could just serve organic steak and potatoes to all of them.

[santamonica811]

Slight quibble: D's rationale is not given to us. If he doesn't eat gluten due to, say, an allergy, then he's in the same boat as Mary, of course. If due to religious reasons, then he's lesser to Mary, but (perhaps) superior to Jamal. If due to mere preference, then he's similar to Jamal.

[Trollificus]

You (and others) are overlooking that the test question completely ignores intersectional factors such as "What race are each of the dinner guests?" and "Are any of the guests gay or trans?"

And the answer should be: "Doesn't matter, you're never going to invite any of these people to a dinner party again."

[Á àß äẞç ãþÇđ âÞ¢Đæ ǎB€Ðëf ảhf]

A better answer than mine, which would have been "D is an adult and has agency, and if he'd meant for his choice to matter, he would have said do."

[Life of Brian]

If due to religious reasons, then he’s lesser to Mary

Under the new and improved system of things, isn't one who merely believes they're going to die if they eat something on fully equal footing with one who actually will? *ducks*

[Nige]

All this stuff is so prevalent you have to keep inserting it yourselves to get mad at.

[Doug Heffernan]

I don’t think the reuters piece includes all of the premises for the question.

Personally, I find analytical reasoning and natural deduction immensely valuable and exceedingly enjoyable. The games section has (had) only relatively simple questions that need only propositional/sentential logic. There is no predicate logic or need for godel’s theorem. Even though most pre-law undergraduate programs include an upper level philosophy class that includes godel. Maybe all of that should be dispensed with too and in its place a defensive driving course could be added, with emphasis on driving while chasing an ambulance.

[Social Justice is neither]

I'm sure they've loaded up the questions with critical theory dogma because who needs logic when you have the progressive stack to guide you.

[Dr. Ed 2]

No one gets ANY meal, they can all go hungry and remember to bring their own food next time.

The one that drove me crazy was the painting the rooms question, and my attitude was mix all the paint together to form "mill end grey" and then paint all the rooms the same color -- grey.

(For those too young to remember it, back when paint factories were smaller and not controlled by computers, they'd have a bunch of various colors (of the same paint) left over at the end of the day so they'd mix them all together and sell it at a discount to people who needed to paint sheds and such and didn't care what color the paint was.)

[WuzYoungOnceToo]

Anybody who has ever seen Airplane! knows that you never eat the fish.

[Jaypd]

That's interesting, I was just working with my wife on that section as she plans on going to law school. I didn't even remember it. I thought it was fun and it did remind me of trying to explain legal principles and how they relate to real world happenings. I don't see it as being as totally off-base, but hey, the LSAT in total is kind of a weird test in terms of its relation to being a lawyer. Anyway my wife will be thrilled.

[Life of Brian]

I thought it was fun and it did remind me of trying to explain legal principles and how they relate to real world happenings.

Yes, agree. Real-world legal scenarios are chock-full of interdependent moving parts, and I'm not at all convinced that the remaining logic section (read a passage and draw a logical conclusion about it) tests your ability to track that sort of dynamic much if at all.

[Life of Brian]

This arose from two blind applicants suing LSAC after it refused to accommodate them by dropping logic games from their tests and substituting it with something else because they couldn't effectively diagram the problems. Instead, as part of a settlement, they're now... dropping it for everyone and substituting it with something else.

Unfortunately, I did way too well on the logic games section for least-common-denominator decisions like that to make sense.

[M L]

If that's the reason for this change then it's very unfortunate.

I thought law schools were trying to start shunning the LSAT altogether? Or was that the rankings.

Don't people already get extra time for ADHD? That's already a sham.

[Life of Brian]

I thought law schools were trying to start shunning the LSAT altogether? Or was that the rankings.

Yes and yes. The latter just seems to me like the infamous "Top 13" law schools cutting off their nose to spite their face, but otherwise neither here nor there vis-a-vis the profession. The former I think is more of a race to the bottom.

[Dr. Ed 2]

Well if you don't know what colors *are*, how on earth could you ever answer that color question?

[KenveeB]

You could tell me there are categories crwet, ytwep, 3sdg, and spp*(#, and I can classify by them. I don't have to know what they are. Similarly, you don't have to know what the colors are to classify based on "Bob hates green, Jim likes blue or green, and Jan loves yellow."

[SRG2]

As far as a barrier to entry is concerned, Oxford University’s entrance exam set consisted of a few papers, some compulsory and some optional and one irrelevant (you had to do a Latin translation paper but it didn’t count towards your assessment…). If you went in on the General Studies exams you could choose General Paper III which consisted of logic problems and brain teasers, and it was well known that if you scored highly enough on that paper, you’d be admitted almost regardless of how well or badly you did in the other papers. That was how I got in

[M L]

About barriers to entry, law schools have traditionally pumped out a number of graduates far exceeding the number of legal jobs available, such that a huge proportion never work as a lawyer or only do so for a short time. So in practice law school admissions haven't really limited the supply of legal services or ensured gainful employment, unlike dental and medical schools.

It's worth noting that the financial interest of law professors, like other academics, is to have more people going to school. Each matriculant may represent up to $200,000 in revenue. As a lawyer I should like to have the federal government fund my clients' consumption of legal services to the tune of trillions of dollars, like they do for higher education.

[Roger S]

It is an IQ test. IQ is highly correlated with competence in a lot of areas, including legal reasoning. Is there any reason to believe that another type of question will do a better job of screening potential lawyers? I doubt it.

[Lee Moore]

Yup. I am confident that the decision to drop it has diddley squat to do with Prof Kerr’s ancient advice to get rid of it, and everything to do with someone having calculated that if it were to be dropped, that would knock a couple of points off the average differences between racial groups.

Look for something using fast twitch fibers to replace it. Maybe a short sprint ? Certainly useful to lawyers - for keeping the ambulance in sight.

[BrotherMovesOn]

You just recently made such a good post.
Now this 'keeping the ambulance in sight' ...

[Lee Moore]

You just recently made such a good post.

Sounds most unlikely. What was it about ?

Now this ‘keeping the ambulance in sight’ …

If thousand year old lawyer jokes can get to you, maybe you should be in a different line of work. Seriously though, I don't hate lawyers. Though millions do 🙂

[Krychek_2]

It's been said that if John Marshall came back from the dead and attended an ABA convention, the only thing he would still recognize would be the lawyer jokes.

[Sarcastr0]

Of course this guy is super into IQ.

IQ is more strongly correlated with socioeconomic class than competence.
https://www.cambridge.org/core/books/abs/in-the-know/iq-only-reflects-a-persons-socioeconomic-status/02476B81A2DF0B958E780D90C27CDFAA

[Michael P]

Affiliation of the author who conflated correlation and causation: Some place that only got promoted from state college to university 15 years ago.

[Sarcastr0]

The author doesn't claim causation.

[Artifex]

You do realize that this book is titled "In the Know: Debunking 35 Myths about Human Intelligence" and you have chosen the 11th myth he is debunking and are trying to claim that this is his opinion right ?

I realize that talking points might be more important to you than an actual understanding, but if you want a rebuttal of your opinion, you might want to actually read the book you are quoting.

[Carl_N_Brown]

Book blurb for "In The Know":
"Emotional intelligence is an important trait for success at work. IQ tests are biased against minorities. Every child is gifted. Preschool makes children smarter. Western understandings of intelligence are inappropriate for other cultures. These are some of the statements about intelligence that are common in the media and in popular culture. But none of them are true. In the Know is a tour of the most common incorrect beliefs about intelligence and IQ. Written in a fantastically engaging way, each chapter is dedicated to correcting a misconception and explains the real science behind intelligence. Controversies related to IQ will wither away in the face of the facts, leaving readers with a clear understanding about the truth of intelligence."

[Roger S]

Soon all the LSAT, law schools, and Bar exams will be declared racist. Not all groups pass in the same proportions. Competency will be considered a White supremacist ideal.

[Sarcastr0]

The evidence is clear: wealthier individuals tend to score higher on intelligence and academic tests. This is true, both in adulthood (Herrnstein & Murray, 1994) and childhood (Zwick, 2002). As a result, some people – like those quoted above – have argued that the tests of g are actually little more than tests of someone’s socioeconomic status. Others have argued that differences in wealth or socioeconomic status cause differences in performance on tests of g.
In other words, they believe that money makes people smarter, or that it can buy higher scores through test preparation classes, better schools, or home life advantages (e.g., L. Brody, 2018; Zwick, 2002). As a result, some skeptics of intelligence testing believe that these tests reflect economic advantages more than any cognitive abilities.

Doesn't seem I'm taking the myth as truth at all.

[r]

If you had just said: IQ is strongly correlated with socioeconomic class, you'd be fine. But you made a much stronger claim: IQ is more strongly correlated with socioeconomic class than competence. This claim is not at all supported by your citation.

Take the L and move on.

[WuzYoungOnceToo]

You do realize that this book is titled “In the Know: Debunking 35 Myths about Human Intelligence” and you have chosen the 11th myth he is debunking and are trying to claim that this is his opinion right ?

Lying about what others have said is the only arrow in Sarcastr0's rhetorical quiver.

[phoqueue]

rofl
Good work, Sarcastr0.

[M L]

The practice of law absolutely involves that skill. For example, when working on a dense 60 page purchase agreement excluding exhibits and schedules, where 7.4(b)(3)(iii) is integrally related to 1.1(x) and 9.2(c) and six other provisions with trap doors all over the place. A good lawyer will be able to hold all of it in the forefront of their mind concurrently and fluidly explain to the client the implications of any facts or changes in terms on the fly. Same goes for other complicated agreements, as well as just any legal analysis involving complicated and variable fact patterns. I can definitely see how it’s more of a “law practice” (client-facing) thing than a “law study” or academic sort of thing.

Furthermore, the logic games section is built on the same concepts as the other two sections, logical reasoning and reading comprehension. In some ways it just a different twist on testing the same things. Where LR and RC tend to go “deeper” and present a more difficult single problem, LG goes “broader” along the surface, testing the ability to hold a number of relatively simpler logical relationships in mind at one time while globally identifying gaps and deductions and determining the impact of changes in variables.

With that said, logic games is widely considered, and is in my view, the most learnable or “gameable,” if you will, section of the LSAT, and therefore may be the least useful or effective at testing ability/skill (and probably least correlated with IQ). I suspect LSAC’s data supports this in various ways. Therefore removal of the logic games section in favor of only utilizing LR and RC may improve the quality of the test.

[Sarcastr0]

You have not described the practice of law generally in your purchase agreement scenario.

I'm also unconvinced pattern matching in logic games is completely analogous to contract practice.

[phoqueue]

His description would also apply to something like pretty normal CGL policies, which are awfully common.

[Sarcastr0]

Ability to handle Commercial General Liability practice is not something we should be gatekeeping entrance into lawschool by.

I had to look the term up; none of my peers from law school work that side of the street.

[Dan Schiavetta]

You must not know anyone who does transactional work either. Or anyone handling a case involving the interplay of statutes and regulations and related case law.

[M L]

Right. It is broadly applicable to practical legal analysis. Sitting down with a client for an estate planning conference. Analyzing tax issues. Regulations, case law, etc.

No surprise that someone who "had to look up" the most common type of business liability insurance in existence knows nothing about the practical side of practicing law.

[YepThatGuy]

Agreed. I am actually fairly regularly reminded that not all lawyers aced the LG section, usually when one fumbles the most-tested rule of logic in the games: the converse.

[Area Man]

I hated the "logic games" section, and probably didn't do too well on it. That said, as a practicing attorney--not, with respect to Prof. Kerr, an academic--I've come to see the value of testing such skills. If it were the entire, or a significant portion of the, exam, then that would need to change. But as just one part of the test, I think this is another bad idea that's just dumbing down the legal profession.

[Randal]

If I recall, the other sections are just all various forms of reading comprehension, yeah?

I guess that makes sense although it does seem like a dumbing-down.

What kinds of legal tasks rely heavily on that skill?

I always thought it was about being able to develop and compare hypotheticals, analogies, and counterfactuals. You have to understand the significance of each connection in the web of relationships linking the various elements of a scenario together in order to be able to do that well.

[Dan Schiavetta]

We lawyers are familiar with the phrase "thinking like a lawyer". It's actually our profession's pompous phrase for clear thinking. It's important in any profession. In auto mechanics it allows to you isolate the cause of that strange knocking sound, by testing and eliminating other causes. In law it allows you to focus on what will work, given the parties, the issues, the layout of the likely litigation, the applicable law, and the judge involved. Yes, you can try the fish strategy (e.g., conducting extensive discovery) but this client is allergic to fish (it doesn't have the $$ to finance it).

I don't understand why the professor thinks the "logic games" section was silly. It seemed like the best way to screen out foggy thinkers.

[Á àß äẞç ãþÇđ âÞ¢Đæ ǎB€Ðëf ảhf]

Clear thinking and lawyers do not go well together. Quibbling is a lawyer's primary quality.

[Sarcastr0]

Pattern matching and clarity of thought have overlap but I do not believe they are the same thing.

[SRG2]

The one area where I am fairly sure that "thinking like a lawyer" means "fucking up" is in the field of probability. There's a reason that a well-known Bayesian trap is called the "prosecutor's fallacy"

[Gozer the Gozarian]

I always hated that part of the LSATs and did poorly on it. I too never understood what it had to do with law other than vetting people who play the JUMBLE puzzle in the Sunday newspaper.

[Randal]

Based on your posts 'round these parts, I'm guessing you didn't do much better on the other sections.

[Gozer the Gozarian]

I didn’t bother having it scored. I had a cold and made the brilliant decision to take Robitussin before taking it.

And I don’t post to impress people. It’s not worth the effort.

Do you have any favorite posts of mine? I had no idea you were an admirer and had an immediate and thorough recall of my posts.

Was your post supposed to be a dig? Because I'm still laughing at it...

[Randal]

Mostly thinking about how we need to kill all the Palestinians.

[Ridgeway]

"Some Volokh Conspiracy posts change the world immediately, while others simmer for a while. This one took sixteen years, apparently...."

I bet if you had gotten Blackman to sign on, it would have been done in a week!

[mydisplayname]

One comment picked up on "Some Volokh Conspiracy posts change the world immediately, while others simmer for a while."

[NellieBly]

Too hard! Too hard! I can't be forced to do anything that is too hard! I do not need logic to be a social justice lawyer! They never use logic!

[Davy C]

It seems to me that there should be studies done on how the section score correlates with graduating. Otherwise you're just guessing.

[BrotherMovesOn]

Bang the gavel! It is settled. The summer camp logic games were fun but of dubious probative value.
Next!

[David Welker]

Orin Kerr is probably right.

But I thought logic games were the only fun part of the LSAT...

[loki13]

A few thoughts after reading the article and the comments-

1. For those of you who missed it, Prof. Kerr was joking and not taking actual credit for the change.

2. According to the LSAC, the revised test (with a second logical reasoning section) has no impact on scoring, and performs the same in terms of predicting 1L grades. Obviously, we don't know how rigorous their testing was, but they did look into it before the change.

3. I always thought that logic games was a good test for a subset of "lawyer-like" skills, or, at least, "law student-like" skills. That said, as someone who did well on the LSAT back in the day ... well enough to earn some nice money in law school as a private tutor for people taking the LSAT, it was my experience that the logic games section was also the easiest section to teach students in terms of improving their scores. Sure, people had limits, but it was generally the one section that you could always get a student from bad to mediocre, or mediocre to good, in terms of scoring if you just worked with them a little- thus making me question the ability of it to test purely for natural aptitude. In other words, students who could afford to pay for prep courses and instruction could often see the greatest benefit in that section. Again, IME.

[CommentMonkey]

I think your third point is a good one. The games section seems to be the one most susceptible to improvement through practice/repetition. If the goal is to assess innate ability, it is thus the least valuable. However, if the goal is to predict performance in law school, I'm not sure that critique applies. Performance in law school is heavily dependent on the ability to stick with what would strike many as tedious study routines, even when success is not always immediate. A student who grinds out LSAT prep is likely to grind out the case briefing requirements so important in the first year of law school. A student who doesn't bother with LSAT prep is less likely to have the fortitude to endure the tedium of case briefing, and thus less likely to excel as a law student.

I think the games section tests skills essential in the practice of law. Whether in the context of contract drafting, litigation strategy, or whatever, lawyers are often put to the task of devising a course of action that balances a variety of short-term and long-term goals, sacrificing some for achievement of others. Being able to properly organize priorities and understand how a change impacts different variables is fundamental to providing competent legal representation. This is exactly what the games section tests.

Unfortunately, the legal profession tends to attract type A personalities who are susceptible to pursuing a strategy with such zeal that collateral consequences are sometimes ignored. The games section counters that tendency, forcing the prospective lawyer to focus on the larger picture and think through consequences carefully. I think Prof. Kerr's description of the games section as a "silly barrier to entry" badly misses the mark.

[loki13]

I agree with all of your points.

However (and isn’t there always a however?), I do have to wonder about the issues surrounding the teachability of the games section, given that there are definite disparities in the abilities of different students to be able to prepare for it. If the LSAC was more open about the ability to perform much better by “grinding” LSAT prep, especially w/r/t that section, I think it would be less of an issue.

But if they were to do this, then it would be less of an aptitude test, and more become more of something that they would have to acknowledge that people can study for.

As it is, IMO, it’s a little bit of one, a little bit of the other. You can’t get someone that naturally scores a 144 to a 180. But it is possible to get a 153 to a 167*, and that makes a massive difference.

*Not always, not everyone, but I’ve seen it done.

[Life of Brian]

I do have to wonder about the issues surrounding the teachability of the games section, given that there are definite disparities in the abilities of different students to be able to prepare for it.

Some are teachable and just need to be kick-started into that way of thinking; others apparently can't wrap their brains around it. Since nearly all of us seem to agree the thought process tested by logic games is valuable for legal practice, binning LSAT takers into "get it"/"don't get it" ranges seems useful.

(And yes, that doesn't distinguish well between "don't get it" and "didn't try very hard to get it" -- but that's true with many testable topics and so doesn't strike me as a good reason to yank this one.)

[KenveeB]

That's a shame, I always thought that was a good section. Being able to keep track of large categories of information and determine how they relate to each other depending on new information seems a very important lawyer skill to me. Maybe it's less important in some of the purely academic areas. But it's certainly useful when dealing with clients, trying to work out a solution that takes into account everyone's needs, or coming up with a response that ticks as many of the positives and as few of the negatives as possible.

[David Nieporent]

The revised format was also as effective as the current one in predicting first-year law school grades

Another way to say this is "The revised format was no better than the current one in predicting first-year law school grades." Which would seem to be an argument against the change.