A word of caution before offering the bar exam online

Remember the Iowa Caucus app and HealthCare.gov?

California and Massachusetts have now committed to offering the bar exam online. Let me offer a few words of caution: Technology is hard, especially for lawyers. Transitioning a multi-day exam to an online platform is a significant undertaking, and it cannot be done in a few months.

Does anyone remember the Iowa Caucus App? (Did we ever actually receive the final results, or is it a moot point now?). Does anyone remember HealthCare.gov? The ACA almost failed on arrival due to the botched website.

Recent graduates are stuck in a very, very stressful situation. Rushing through an online testing system, without adequate planning, will only increase the stress. Even worse, it is conceivable that errors in the testing process could alter results. Imagine if a student sits for the bar, and some of the questions are not uploaded; or data is corrupted; or a computer crashes and there is no way to retake the exam till six months later. One of the reasons my faculty opted for a credit/no-credit system is that we lacked confidence in the integrity of online examinations.

My colleague Derek Muller highlights additional problems.

Let's put aside the security issue for a moment and simply focus on reliability of software. Six years ago, ExamSoft had an issue during the July 2014 bar exam where thousands test-takers were unable to upload their answers for hours. Some (I think, wrongly) even blamed this debacle on a decline in bar passage rates that cycle. Exam software is not sufficiently reliable even in the best of times. Add to that the remote (and secure) delivery of materials that have previously been printed, and the collection of those materials after the exam.

In-room security is a huge problem, too. Bar exams are notorious for picayune requirements, like a small clear plastic bag containing limited personal effects, sign-in sheets to use the restroom during the exam, and so on. Remote proctoring software purports to watch the eye movement of exam test-takers during the exam, to scan the room before and after to make sure no one else is present, and other rather theatrical promises. Let's face it—those probably work in much lower stakes tests.

Given enough time, these sorts of contingencies can be planned around. But is impossible to create a quality product under these circumstances.

Comments

[lucia_l]

The high school advance placement exams are going to be online in May. They are lower stakes than the bar, but are very important to the students taking them. I'm waiting to read how these go.

Yes, the Iowa Caucus experience comes to mind. It's something where problems have serious consequences for test takers. It's a first time try. The College Board is evidently confident. But OTOH, what else are they going to say? Admitting even any lack of confidence would just stress the kids taking tests out more.

We'll see how that goes in two to three weeks.

[Noscitur a sociis]

I agree that there are significant concerns about administering an online bar exam.

What would you propose as an alternative?

[NToJ]

Do away with it entirely. It doesn't test the vast majority of substantive law in any jurisdiction. The entire thing is a colossal waste of time and money.

[santamonica811]

Okay, I will rephrase the question you kinda avoided answering: Given that states will NOT do away with bar exams, and given that SOME exam must therefore be given; what would you propose as an alternative?

[NToJ]

"Given that states will NOT do away with bar exams..."

I don't take this as a given at all. Wisconsin and New Hampshire already have diploma privilege for admission to their state bars. The Utah Supreme Court is considering it right now, because "the coronavirus pandemic makes the July bar exam seem unlikely at this point." There's a chance Iowa will go for it as well. I'm proposing that all the states adopt and broaden the approach accepted in Wisconsin and New Hampshire, and under consideration in Utah.

[MatthewSlyfield]

Diploma Privilege only works for people who attended a law school in the state they intend to practice in.

[NToJ]

Diploma Privilege works exactly how states want it to work. As an example, the proposal from Utah would grant licensure to anyone who graduated from an ABA-approved law school (I assume that means accredited) from May 1, 2019 to June 30, 2020, so long as that school had a first-time bar passage rate at or above 86%. (I assume they mean for Utah's bar, but I don't know.) It throws in a 360 hour apprenticeship requirement, which is probably how all lawyers in the country should be licensed, since bar exams are stupid as shit.

Several law professors have advocated abandoning the "pure form" diploma privilege you reference because it "would hamper efforts to serve clients, and would be particularly problematic for graduates (and their employers) who have already accepted jobs in other states." Their proposed solution is to broaden diploma privilege to all accredited law schools, so long as the other jurisdictions promise reciprocity on this basis as well.

Since a majority of states are going to the UBE effective 2021 anyway, state-specific bar exams are already on the way out. Of course states could do whatever they want, including among others:

1) Selecting specific schools (not just ABA-accredited ones) to extend diploma privilege to;
2) Additional certifications like proof of summer clerkships, affidavits from employers or supervisors, etc.; and
3) Pre-licensure CLE requirements, specific to the candidates' actual area of future practice.

Supervised Practice is another proposed alternative.

[MatthewSlyfield]

Your own words:

"I’m proposing that all the states adopt and broaden the approach accepted in Wisconsin and New Hampshire,"

Both Wisconsin and New Hampshire limit the diploma privilege to in state schools.

[Noscitur a sociis]

Which is, presumably, why NToJ said "adopt and broaden" rather than "adopt".

[Dr. Ed]

When you have law schools where 4 out of 5 fail to pass. who or what will protect the public?????

[James Pollock]

" Given that states will NOT do away with bar exams, and given that SOME exam must therefore be given; what would you propose as an alternative?"

Apprenticeship in the trade, AKA "Articling"

[KOB]

The purpose is not to expect a memorization test for a majority of law in any jurisdiction, but it is a fairly low bar to see if a person is capable of learning and applying the law in a consistent way.

[NToJ]

The bar exam I took did not test consistent application of law.

[Rossami]

Your complaint starts with the hidden premise that the bar exam ought to be a high stakes test. I would push back and suggest that the bar exam is nothing more than naked protectionism and that the right alternative is to simply eliminate it.

[DRM]

Requiring degrees from ABA-accredited law schools (ever actually look into the standards for ABA accreditation?) is much more naked protectionism than the bar exam.

[Dr. Ed]

EXACTLY! -- And you typed it first...

[Kenneth Almquist]

There was a lawsuit discussed here (which I unfortunately cannot locate) filed by a prospective lawyer who had failed bar exam and was suing to get the licensing system overturned. The brief he wrote listed a number of precedents, all of which rejected his legal theory. Or did they? The Supreme Court "holding" consists of a sentence in the opinion which notes that states have required licenses to practice certain professions and lists some examples, lawyers being one example. That's not a holding, it's a passing reference to existing practice in a case that has nothing to do with the merits of requiring a license to practice law.

There are a variety of ways to deal with adverse precedents; the approach used in this brief was what I will call argument by exclamation point. Rather than simply stating (falsely, let's not forget) that your legal theory is contrary to Supreme Court precedent, you state in a sentence ending in an exclamation point. Surely, the theory appears to be, if someone who can't even pass the bar exam expresses surprise that his pet legal theory is contradicted by precedent, the precedent must be wrong.

Reading the brief was like watching a football coach who thinks he's coaching a chess team. It's not merely a case of lacking talent; the writer doesn't understand the rules of the game. If it weren't for the bar exam, this person could be offering legal services to the public. In this one instance, at least, the bar exam served its professed objective of protecting the public.

[Rossami]

A. How did that guy pass his law school program?
B. A functioning loser-pays system would be far more effective at rapidly weeding out the incompetent lawyers .

[James Pollock]

" Given that states will NOT do away with bar exams, and given that SOME exam must therefore be given; what would you propose as an alternative?"

An attack on the American system that doesn't even bother to address the concerns why the American system is different from the English system? Why not just go directly to a third-world system? Whoever offers the judge the best bribe wins the case.

[Armchair Lawyer]

That seems like a horrible idea. The Bar exam is one of the highest stakes exams in common usage. The concept that people won't cheat on it, if offered the ability, is nuts. How remote proctoring would really work...whoops, the person helping you isn't in the camera view, our bad...is nuts

[NToJ]

"The concept that people won’t cheat on it..."

Who cares if they do? I "cheat" during the practice of law all the time. If I don't know the answer to something, I open a book and learn or remind myself what the law is. Not once in my entire career has my practice depended on immediately recalling some minutiae I memorized for the bar exam. If I attempted to practice exclusively from what I can immediately recall in the same way that I would during a proctored exam, I'd commit malpractice.

[Armchair Lawyer]

Of course you do. But what's important is that you know to look, know where to look and how to look.

The largest flaw in knowledge-based professions isn't not knowing something. It's not realizing that you don't know something and not knowing how to acquire the knowledge you need (ideally without asking someone else).

The Bar (and other exams) ideally tests those skills. Some specific knowledge is needed. But the application of the knowledge, and using it to acquire the details you need is what is critical.

[NToJ]

The bar exam does not test whether lawyers know “where to look and how to look.” (For what?) Some law schools teach that.

[Armchair Lawyer]

"Some law schools teach that" Apparently not very well.

Bar passage rates are about 60%. Which is really low for such a critical pass-fail exam. USMLE licensing exams, by contrast, are in excess of 95% generally. Moreover, when adjusted for grades, bar prep classes don't appear to be helping students

The Bar exam is acting to prevent unqualified lawyers from practicing. The law schools are not failing out people like they should be, apparently.

[NToJ]

“ Apparently not very well.”

Slow down. The bar exam does not test X. Some law schools teach X. You respond by saying apparently law schools don’t teach X well, since many law students fail the bar exam. But recall that the bar exam doesn’t test X.

If the bar exam tested people with coin flips, it would be even more effective at weeding out students.that, like the bar exam, would have nothing to do with qualification to practice law, though.

[James Pollock]

15% of the bar exam is the part that's actually based on the way licensed professionals do it in the real world.

An effective bar exam would be open-book, and offered in a law library, not unlike the exam offered to get onto the law review. (Note: I only know how one law review operated their write-on competition, so YMMV.)

[KOB]

The problem is about having some minimal ability to learn and apply the law.

[regexp]

And yet cheating happens all the time on Bar Exams based on what I've heard from lawyers. Perhaps you should try something new.

[Armchair Lawyer]

Perhaps you should forward these names to certain people in certain associations. There is an overabundance of lawyers. Perhaps these ranks should be thinned a little, based on ethical concerns.

[Bob from Ohio]

Its been a long time since I took the bar but I think time restraints will limit the effectiveness of cheating in any event. Most people who took the bar in my day used up most of the allotted time, looking up things adds a lot of time.

[santamonica811]

Perhaps. If you have a lawyer friend who is willing to be a participant, then it would be easy. Remember, you don't need to know the law well to ace the bar. You have to be able to spot the issues, and know the rules that apply. And then methodically apply them. Bar graders spend an average of 2-3 minutes grading each short essay (in Calif, anyway). They are looking for keywords, and that's about it. A friend who can write a note: "Don't forget burglary. Breaking, entering, dwelling, of another, at night, with prior intent" will allow you to get full marks for that entire question. With this conspirator (1) She can allow you to study and memorize far fewer things, and (2) She can do research on Question 2 while you're working on Question 1, write up a short outline, and allow you to get a very strong score in half the time...giving you more time for the other questions. A huge advantage.

But for cheating by looking up stuff on your own?...yeah, I agree with you. Might help in limited circumstances. But generally, the time you'd spend cheating would not be worth it, since it would swallow so much time that would be better spent gobbling up easy points answering questions and issues you already understand.

[Armchair Lawyer]

But with online exams, you can imagine someone off camera, as you read the question out loud, and the person off camera responds with what you need.

I mean, it would cost a pretty dime. But, compared to the cost of law school, for the marginal students, it would be worth it.

[Life of Brian]

for the marginal students, it would be worth it.

And I think that's the key. Well-prepared candidates aren't going to mess with this -- they've prepared and drilled to the point where they're comfortable they'll land in a passing range. Candidates more on the margins (and particularly retakers, where the stakes are already higher for multiple reasons) are much more inclined to turn to measures like this, and much more likely to get a benefit that, though likely small in the absolute, is enough to get them over the line.

[Armchair Lawyer]

Yep. Keep in mind, failure rates on the bar are on the order of 40%. Which is pretty high, all things considered. 40% of people taking a pass fail exam will fail, and failing means the loss of a lucrative career and years of schooling and paying for that.

[santamonica811]

Sure. Actually, if there were cheating, I would assume that most of it would be with friends who are lawyers, and not with "professional" bar takers, as the risk of being exposed and blackmailed would (I am imagining) be higher with non-friends.

On the other hand, I could totally see a boyfriend/girlfriend helping you cheat on the Bar, you break up, and they immediately call the Bar and report you for cheating.

Moral: don't cheat. (But, as you say, students on the margin--who feel quite justifiably that they cannot pass on their own merit--might cheat regardless of the risk, since they'll not become lawyers without cheating.)

[James Pollock]

The thing is, already being a licensed practicing attorney is no guarantee of passing the bar exam. In fact the longer a person has been practicing increases their knowledge and skill in their preferred practice area but greatly diminishes their competency in all the others. That's why lawyers seeking to be admitted in additional states take bar prep classes along with the recent JD graduates.

[James Pollock]

"Its been a long time since I took the bar but I think time restraints will limit the effectiveness of cheating in any event. Most people who took the bar in my day used up most of the allotted time, looking up things adds a lot of time."

One thing you can do in the simulated practice section is give people the same question, but different materials. So student A is given a snippet of the criminal code that says the alleged crime is a So, and student B get s a snippet of the criminal code that says it's a felony, and student C get s a snippet of criminal code that suggests not all the elements of the crime are present. So if you question includes the fact that the prosecutor has offered a plea bargain of one year of prison time, and asks whether you should advise the client to take it, you would have different "correct" answers based on the given statutes you're asking the candidates to examine.

[AtR]

I have looked into the technologies, and I have not found *any* system that provides reasonable security against cheating. Indeed, for a high stakes exam, there are companies who will gladly provide you the means to cheat, even if you yourself don't have the technical know-how.

The only silver lining is that, perhaps, for many lawyers it doesn't matter.

[regexp]

I have not found *any* system that provides reasonable security against cheating.

Live proctoring via computer webcam which is used for several information security certifications.

So I've shown you one.

[Noscitur a sociis]

Can you elaborate? I have never heard of cheating on a bar exam, and it seems like it would be a pretty big deal if it did happen. I also can't imagine how being s lawyer would help me find out about it.

[Noscitur a sociis]

NB: this was intended as a response to the question above claiming lawyers know that there is rampant cheating on the bar exam.

[James Pollock]

"I have never heard of cheating on a bar exam"

Not surprising. Discussing cheating on a bar exam would provoke undesirable thoughts in the public regarding the actual qualifications of the candidates for bar admission, including those previously admitted. If people are hearing about candidates cheating to pass the exam, it suggests that candidates think they can get away with it, and if candidates think they can get away with it, you have to wonder how many actually did. This is also why the bar associations promote the notion that the bar exam is difficult, when it is not. It is designed so that most people can pass it, if they spend a couple of thousand dollars and a few weeks on a bar review course.

[Rossami]

I took one of those when I stood for my CISM. There were easily still a half-dozen ways I could have cheated.

Interestingly, for an information security exam it's not clear that successful cheating should be a reason to fail the exam. Cheating and getting away with it generally requires a thorough understanding the security principles that you're trying to compromise. Getting caught, however, indicates that you don't really understand the system you're trying to hack. Counter-intuitively, it's an incentive for the weaker students to stay honest and take their chances with the test questions.

[James Pollock]

Pretty much any IT certification exam at a testing center is proctored via surveillance cameras if it isn't observed by live proctors, and sometimes even if it is.
I would tend to agree that it's usually easier to learn the subject matter sufficiently to be able to answer the questions than it is to work out an undetectable cheating method. And that working out a successful, undetected cheat shows a mastery of information technology security methods and practices.

[James Pollock]

"I have looked into the technologies, and I have not found *any* system that provides reasonable security against cheating."

Keylogging would do a pretty good job for monitoring someone who only has access to a single computer. I imagine a C&F panel handed a keylogging transcript which contained "www.westlaw.com" would have no trouble concluding that cheating had occurred. "www.google.com" would be a pretty clear smoking gun, too.

[James Pollock]

"I have looked into the technologies, and I have not found *any* system that provides reasonable security against cheating."

Faraday cage around your testing center.

[christophersw]

Make them handwrite it. And no, I'm not kidding - I hand wrote the bar exam less than a decade ago (by choice). We offer handwriting options as a backstop for times when the technology solutions won't work. This is one of those times. If an applicant is not comfortable writing the exam - then they can wait. I don't mean to down play the pain to applicants, this is terrible, and handwriting is a major burden to them, but we need a realistic, proven path to barring attorneys this summer. I'd guess that a significant percent of currently practicing attorneys hand wrote their bar exams, and I fail to see why that isn't a viable option for this pandemic cohort.

[NToJ]

You serious, Clark?

[Noscitur a sociis]

And how would that "solution" address any of the issues raised by Prof. Blackman?

[James Pollock]

I handwrote the exam back in 2010, because I looked at the exam-security software with some degree of suspicion and literally nobody could explain how the software actually added any security. And it wasn't available in open-source form, so I couldn't even be sure that it actually did any of things its proponents claimed. AND they wanted me to pay for it! It made a difference, I can touch-type a lot faster than I can write.

[James Pollock]

I took the CISSP exam, which is a six-hour multiple-choice exam, entirely by filling in bubbles with a number-two pencil.

[Peter Gerdes]

And this helps how? Writing out the answers does nothing to stop confederates from feeding you answers if you aren’t in a proctored exam room.

As for this security software this is one of the few times it’s a good thing to be closed source. In fact a closed source cheating detection program that did absolutely nothing would work better than open source cheating detection software. With the later you can just rest your cheating strategy beforehand to your heart’s content and verify it won’t trigger the security program but even a placebo closed source cheating detection program has substantial deterrence value since test takers have no way of knowing what it might detect.

[James Pollock]

One of the key assumptions of computer security is that if the bad guy can get you to run his software on your computer, then it isn't your computer any more. One of the mechanisms to prevent accidentally running a bad guy's software is to only run software you create yourself. That's something a lot of people don't want to take on, but open-source software lets you check to see what, exactly it is that someone else's software is actually doing. It also allows for crowd-sourcing of bug fixes. In ye olden times, a lot of people trusted Microsoft, then they released office-productivity software with a capability of running malware built right into it.

[Peter Gerdes]

Fundamentally proctoring software is hostile software meant to limit your use of your machine .

[James Pollock]

It's spyware. It only limits your use of the machine if you care if someone else knows what you do and how you do it. Which a cheater cares about, but a non-cheater doesn't have to.

[The Turtle Dove]

I did the same -- for both the bar exam after graduation, and the multistate version after the Ph.D. It was my method of notetaking in the JD, so it worked well. And in this situation, it would free up the laptop to shoot a whole-room feed -- show test-taking surface and all four walls at the beginning, then put the laptop against the wall.

Mr. D.

[christophersw]

This is was what I was thinking.... But as many have pointed out above, handwriting doesn't solve a lot of the issues spotted in the article.

[James Pollock]

It takes a method of communicating with outside sources out of the mix. It also helps with avoiding letting the questions out, if you care about that.

[regexp]

Technology is hard, especially for lawyers.

This is the truest statement ever said on this blog. Honest question - why are most lawyers luddites?

Does anyone remember the Iowa Caucus App?

Voting is a relatively rare event. And they must go smoothly the one time because you can't redo voting in most cases. So it's a poor use case for technology. Especially since pencil, paper, and optical scanner do the job extremely well.

Does anyone remember HealthCare.gov? The ACA almost failed on arrival due to the botched website.

So did Medicare when it was introduced. Here is the thing - they got over it, fixed the issues, and moved on. And online exams have been a thing for a decade now. It's a good use case for technology. The issues you highlighted have largely been resolved. It incrementally gets better until you realize that there were issues in the first place.

[James Pollock]

"Voting is a relatively rare event. And they must go smoothly the one time because you can’t redo"

It's got secrecy baked into it, which limits the ability to forensically reconstruct after a data-loss incident. In exams, the secrecy is different in polarity. It's the exam-giver who has all the secrets, not the exam-taker.

This makes a difference.

[James Pollock]

"This is the truest statement ever said on this blog. Honest question – why are most lawyers luddites? "

The human brain has a limited capacity for specialized knowledge. Lawyers use most of theirs to remember important legal principles and precedents. This doesn't leave them much vacancy for other complex specialized knowledge, such as details about how information technology works. The same principle explains why other kinds of specialists struggle to incorporate legal reasoning into their repertoire.

[Jimmy the Dane]

I'm sure management here will reach the same conclusion management does every time it is confronted with that annoying phrase - planning.

"We will figure it out as we pull the trigger," is what the CEO says thinking that makes him look bright and agile.

"Plan and project management are for bureaucrats," says the CEO thinking that makes him look like a good leader.

"We need to keep our eye and prize and make this happen," says the CEO thinking that makes him look goal oriented while everyone else wants to get waded down in the process.

And when it all inevitably fails and it comes back to the CEO...

"It was the fault of (insert person thrown under bus)," says the CEO thinking that no one will pay attention to the fact he set everyone up for failure from the beginning.

[James Pollock]

Our country's CEO doesn't take the blame for anything. And his apologists buy all the excuses at full mark-up. "It's Obama's fault" is a favorite

[Jimmy the Dane]

Hate the game, not the player.

[James Pollock]

I remember a time when Republicans believed in personal responsibility. Their leader currently mocks the concept.

[ThePublius]

So, online exams are not good enough for the "high stakes" bar exam, but vote by mail is fine?

Voting is much higher stakes. There is absolutely no safeguard to ensure that the person eligible to vote is actually the one voting.

Bar associations and bar exams are naked protectionism. A diploma from an accredited institution, or reading the law and then passing some kind of exam, should be sufficient.

Down with professional licensing!

[Sarcastr0]

If the OP has ever taken a position on mail-in voting, I don't know of it.

[David Nieporent]

So, online exams are not good enough for the “high stakes” bar exam, but vote by mail is fine?

Because anchovies taste good, everyone should take a train when traveling to Disneyworld.

(See, I can do non sequiturs too!)

[James Pollock]

If your premise is false, then nothing that follows is valid.

[James Pollock]

"Voting is much higher stakes. There is absolutely no safeguard to ensure that the person eligible to vote is actually the one voting."

Proctor.

[Peter Gerdes]

Yes, but it’s literally incoherent to cheat at voting. Getting advice from a friend or looking up facts while voting doesn’t taint the vote.

Vote fraud requires you submit ballots on behalf of others. This is easily deterred because any attempt to do so at scale will almost surely result in a double vote (the chances none of the 10k people whose vote you cast also vote themselves is essentially 0) and once the election supervisors see a bunch of double votes it will likely get the result thrown out and the FBI called. The FBI can scour surveillance tapes to figure out who is mailing thousands of fake ballots so you are unlikely to shift the election but very likely to spend a decade in prison.

[James Pollock]

" This is easily deterred because any attempt to do so at scale will almost surely result in a double vote"

There's a case from Oregon where an election worker filled in undervoted ballots. In other words, if a ballot came in where the voter didn't mark a choice, the worker filled in the bubbles for her party's candidates in those races. Which was detected because each party has observers to watch the election workers, and each other, while the ballots are processed.

[James Pollock]

"Voting is much higher stakes. There is absolutely no safeguard to ensure that the person eligible to vote is actually the one voting."

Unless you're in Oregon, where the voter signs and mismatched signatures result in ballots not going into the ballot-counting machine. You could do the same thing cryptographically if you were sure that nobody along the line would screw it up, and then you remember that literacy isn't a requirement for voting, so no matter how clear the directions you give might be, you're going to have problems, with partisans of at least two different types looking for ways to get the other guys' ballots not counted.

[James Pollock]

"Imagine if a student sits for the bar, and some of the questions are not uploaded; or data is corrupted; or a computer crashes and there is no way to retake the exam till six months later. "

One of the reasons to go to CBT is that you can reload and restart without having to wait six months. IT certification exams are computerized in large point for this reason. Note that "computerized" does not imply "unproctored" The advantage of computer-scored multiple-choice testing is that you get immediate results, too.

[Peter Gerdes]

Yes but the whole point of this post is how do you give exams when you can’t ask people to go to a central location to be proctored.

[James Pollock]

You don't send them all to the same place at the same time to be proctored.

[Peter Gerdes]

However, there is a solution that might work which avoids all the problems with technology. Simply pair bar takers randomly with attorneys in their area willing to do 1-1 proctoring of the exam. That’s a minimal transmission risk and probably more secure than the current system.

[James Pollock]

Or, you could assign each new graduate or out-of-state attorney a licensed practicing attorney to monitor their performance actually attorneying for the first six months they're licensed. Call it "apprenticeship" or "residency". That would be much more accurate in weeding out those unfit to practice and improving the skills of marginal performers.

[Burkhard]

my take on this: do what is done in the real life situation that corresponds to the test. There is no real life situation that corresponds to the test? Then you use the wrong sort of test to begin with

[James Pollock]

Switch from a knowlede test to articling. Use medical licensing as a model.

[Peter Gerdes]

The analogous real life situation is knowing if you should object in a courtroom, answering a question from a judge (eg in an appellate court) or whether or not to advise your client not to answer a question asked in a deposition.

The way that’s handled is a combo of proctoring and financial incentives. The justices of the supreme court would notice if you whipped out your phone to answer their questions. Sure you can get help from another lawyer on your team but they won’t carry you for long if you can’t pull your weight.

[iowantwo]

Now I know how China feels about having a virus named after them.

It is not the "Iowa" caucus app.
It is the DNC (Democrat National Committee) App, deployed for the Iowa Caucus.

[James Pollock]

You guys are the ones who insist on caucusing instead of primarying. That's an Iowa thing, not a partisan thing.

[Peter Gerdes]

The real problem is assistance from other individuals. I mean there is absolutely no good reason not to let students google go their heart’s content. Google won’t issue spot for you and I’d much rather hire a lawyer whose underlying analysis skills were so good they could pass the bar by googling despite being generally unprepared.

But it’s just not technically possible to stop people from receiving verbal assistance...at least if people are allowed to use desktops. I mean nothing stops the test taker’s accomplice from simply unplugging the mic or using a long cord and placing it in another room where a second conspirator can supply any sounds the program asks for to verify the mic is operating. A simple nanny can with a view of the screen can give conspirators an undetectable way to view the problems.

Compared to high school exams the huge monetary rewards involved and the fact that law students have more resources (and are less likely to live with disapproving parents) than high school test takers does raise the risk of cheating but at same time increases penalties for those caught helping.

[Peter Gerdes]

I mean if you were really serious about it and had a few years to plan you could just ship special purpose hardware that bundles 360 degree camera view and microphones in tamper resistant design and which test takers literally use to answer the questions via built in keyboard.

But this isn’t doable without years of planning ahead.

[James Pollock]

And then what do you do if your hardware vendor decides there's not enough money in it to continue building to your specifications.

[Peter Gerdes]

And I’d add the cheating and reliability issues are in tension. It’s straightforward and reliable to just send test takers to an online test in browser but, for security software to function usefully, it must be designed exactly to break the abstractions well behaved programs can’t detect.

I mean if you can just run the security software in a VM the game’s over.

[James Pollock]

Your software security vendor will have to integrate with or replace the stock OS. You're going to have to impound their smartphone, too.