Belated Caballes Thoughts

Julian Sanchez | February 2, 2005

After reading Jacob's piece from last week about the Supreme Court ruling that a drug dog's sniff isn't a search, I got to thinking about the problem of error rates. Jacob cites Souter's dissent, which references cases involving dogs in the field giving false positives 8 percent of the time or more and studies where in "artificial testing situations" they gave false positives between 12 and a mindboggling 60 percent of the time.

A few fairly obvious problems occur to me belatedly. The first is that to the extent that law enforcement officers now feel increased license to do indiscriminate sweeps, the conditions under which prior accuracy rates were ascertained in the field no longer apply. Presumably, dogs are typically brought in when there's already some independent reason to think the person thus searched is holding drugs. If police begin to feel they don't need that reason, we can expect the proportion of false alerts to rise.

The possibility of a shift to indiscriminate sweeps presents a related problem from another direction as well. Let's grant that the dog is 95 percent accurate. Now, you might think that sounds pretty good—95 percent certainty would surely count as probable cause, right? The problem is making the error—and I wonder whether maybe the justices did this—of inferring from a 95 percent accuracy rate that you're only going to end up physically searching one innocent person for every 19 who really do have drugs. But if searches are indiscriminate, that's wrong, because the vast majority of motorists won't have drugs.

Assume, and I suspect this is a big overestimate, that one in 100 motorists are driving around with drugs. If you're sweeping them randomly, a 95 percent accurate dog is going to "alert" for five innocent people for every one it catches with actual drugs. And those five people are going to get their cars torn apart. If the dogs are less accurate, it'll be many more than that. Anyone think that passes Fourth Amendment muster?

Addendum: Commenter Shannon Love turns up a handy-dandy java applet illustrating the problem mentioned above. Just substitute "drug possession" for "infection" and you can test the likely results for different levels of dog accuracy and motorist drug possession. On the extremely generous assumptions above (1 percent of motorists have drugs; dogs are 95 percent accurate), the chances that someone IDed in a random sweep (and therefore subject to an extensive and intrusive vehicle search) actually has drugs are about 16 percent. Relax those assumptions and the picture quickly gets far worse.

Comments

|2.2.05 @ 2:32PM|#

Good points all. There are two issue and not one. The value of the mechanism of a dog's nose is a separate issue from the indiscriminate use of any arbitrary search mechanism.

As a side note, I wonder if it helps to drive through running water. That always works in the movies ...

|2.2.05 @ 2:37PM|#

Anyone think that passes Fourth Amendment muster?

Probably doesn't matter.

The police think that the Fourth Amendment, like the Geneva Conventions, is a "quaint relic".

|2.2.05 @ 3:28PM|#

If you don't mind a humorous, related story...About 10 years ago a friend and I attended a big annual car show in downtown Boston. On the way in we enjoyed a large joint. Of course we were oblivious to the fact that we reeked of pot. As we're walking around we stumbled on a display put on by the Mass. State Police, featuring of course, drug-sniffing dogs. They went crazy as they walked by the area we were standing, and several of the officers gave us knowing looks. We beat a hasty retreat.

|2.2.05 @ 3:34PM|#

And those five people are going to get their cars torn apart. If the dogs are less accurate, it'll be many more than that.

Good point. But thats all it is for us. The Drug War Chronicles have done well keeping tabs on the innocents killed during a mistaken drug raid/bust and yet the feds just take it in stride as no big deal. The insanity continues...

Kevin Carson|2.2.05 @ 3:48PM|#

But as Radley Balko pointed out, a cop can make his dog "alert" a comic book, if he wants to. And if the cop "knows" you're a no-goodnik, he can plant any evidence he needs to prove it. Unless you're lucky enough to have video backup, what judge is going to take the word of a scraggly, anti-social druggie like you over that of a fine, upstanding blue knight?

Kevin Carson|2.2.05 @ 3:49PM|#

Oops. I forgot the term "blue knight" is out of date. Most local police forces now wear black uniforms, but only because the supply is more dependable--no other reason at all!

|2.2.05 @ 3:50PM|#

I wonder if, in the dog world, there's a certain negative stigma associated with being a drug sniffing dog: "Let's have a pack party, but for heaven's sake, don't invite Shep; he's a narc."

|2.2.05 @ 4:08PM|#

I've seen this argument in the excellent Innumeracy by John Allen Paulos. He notes that if a test with a 95% accuracy reports that you have a terminal disease, your response should not be despair, because the odds are that you do not have that disease.

The math is similar, and it's interesting to see that the lessons that *could* have been learned from Paulos' book... weren't.

|2.2.05 @ 4:08PM|#

Or perhaps they make sure to invite Shep, because he can score.

|2.2.05 @ 4:09PM|#

"Better that a hundred guilty go free than a single innocent convicted [paraphrased]."

|2.2.05 @ 4:26PM|#

Does shep get made fun of for his cheesy mustache?

|2.2.05 @ 4:31PM|#

Wouldn't the dogs learn the body language of the cops when the cop really wants to detain a particular person and narc the person out?

What was the counting horse's name, Big Hans or something?

Can I call in Dr. Pavlov as a witness for the defase at my trial?

|2.2.05 @ 4:33PM|#

dafase=defense

duuuuuude.......

Shannon Love|2.2.05 @ 4:48PM|#

Julian Sanchez,

The situation you described is in covered in statistics under the name of Baye's Rule. In short, even highly accurate test will give overwhelming numbers of false positives if the size of the population tested is far larger than the size of the possible true positives.

Here is a java applet that illustrates the rule very nicely.

The same problem applies in other areas like screening airline passengers for possible terrorist. Any system actually sensitive enough to flag real terrorist will flag many times more perfectly harmless people than actual terrorist.

|2.2.05 @ 4:52PM|#

I've seen this argument in the excellent Innumeracy by John Allen Paulos. He notes that if a test with a 95% accuracy reports that you have a terminal disease, your response should not be despair, because the odds are that you do not have that disease.

Could someone give a brief explanation to this non-mathematician, as to why there isn't a five percent chance of having the disease in this case? Does it have anything to do with the disease itself being relatively rare?

|2.2.05 @ 4:53PM|#

Never mind. I just saw Shannnon's post.

|2.2.05 @ 4:55PM|#

Question.

Does anybody know if there are dogs that are trained to find any drugs besides reefer?

Michele|2.2.05 @ 5:00PM|#

they can find just about all of them; coke, H, meth.
Re: failure rates; the difference between screening for terrorists and planes and possesion of drugs is huge. I'm not hugely in favor of being felt up, my irritation with the screening process is it could be more advanced if the TSA would be willing to buy non-US-made screening automation products.

Finding drugs on a person in a car who is otherwise not driving unsafely due to those drugs is a completely different thing. Since drugs by themselves should not be criminal and I should be able to transport them from point a to point be safely.

A terrorist with a bomb must never be allowed on the plane.

|2.2.05 @ 5:02PM|#

Does anybody know if there are dogs that are trained to find any drugs besides reefer?

Coke, heroin, etc... The big ones.

|2.2.05 @ 5:17PM|#

OK, do they have to use different dogs for each, or is there a one type fits all dog? If the same dog does them all, do they react differently to each drug?

I don't have to be a dog to be smelling a bunch of shit here.

Shannon Love|2.2.05 @ 5:30PM|#

This problem will only exacerbated when dogs are replaced by hand held chemical detectors. I expect to see the use of such devices within a decade.

False positives could be a real problem if the detectors can sense drug residue on money or clothing unrelated to the owners use.

|2.2.05 @ 6:17PM|#

What is the accuracy rate in finding public servants who can understand game theory?

|2.2.05 @ 6:21PM|#

What is the accuracy rate in finding public servants who can understand game theory?

Apparently 1 in 4 at the Supreme Court level.

|2.2.05 @ 6:23PM|#

Makes you stop and think when you realize the accuracy rate of breathylizers is lower than 95%.

|2.2.05 @ 6:51PM|#

"On the extremely generous assumptions above (1 percent of motorists have drugs"

HAHAHA 1% is a generous assumption? Here in Flinttown it's more like 1 percent DON'T have drugs on them at any given time!

No, really, where do you live, Utah?

|2.2.05 @ 7:29PM|#

Last numbers I've seen peg "current" (i.e. last 30 days) drug usage rates in the U.S. at a little over 8 percent of the adult population. Even assuming that's a couple points low, how many of those people --and remember we're talking about a lot of "couple tokes at a party now and then" folks along with serious users--do you expect usually go driving around with their drugs? More than a tenth?

|2.2.05 @ 9:44PM|#

This is called "ignoring base rates." Very common error.

|2.2.05 @ 11:56PM|#

Assuming this type of sniff'n-search(TM) becomes more widespread, I can envision some enterprising stoners having endless fun with some leftover shake, water, and a spray bottle.

Xmas|2.3.05 @ 5:36AM|#

But breathalyzers are used against a population with a high likelyhood of intoxication. To be subjected to a breathalyzer, you've already been pulled over for driving erratically. (Or, your coming out of a bar and the police are sitting near by waiting for drunk patrons.)

|2.3.05 @ 8:40AM|#

"Assuming this type of sniff'n-search(TM) becomes more widespread, I can envision some enterprising stoners having endless fun with some leftover shake, water, and a spray bottle."

No shake is ever leftover. You never know when the next batch will roll in.

Someone needs to develop some vaccine against the "if you're not breaking the law, you have nothing to worry about" logic. It's even infesting our Supreme Court.

Seriously, do some people think the whole notion of presumption of innocence something to wipe their asses with?