Junk science

The FBI's Forensics Disaster

This latest failure of criminal science again highlights the need for massive reforms at law enforcement agencies.


Credit: Byelikova | Dreamstime.com

The FBI has a new headline-grabbing—and hair-raising—scandal connected to faulty forensics practices. This is not the first time the science prosecutors have used to put people behind bars has proven to be deeply, seriously faulty, and it is another reason to push for reforms and better oversight.

The FBI has admitted that its hair examiners have been dishing out clap-trap in court and in their reports prior to 2000 when describing hair analysis in hundreds of cases—though the flawed science involved could affect as many as 3,000 cases. The FBI itself quotes Peter Neufeld, co-founder of the Innocence Project: "These findings confirm that FBI microscopic hair analysts committed widespread, systematic error, grossly exaggerating the significance of their data under oath with the consequence of unfairly bolstering the prosecutions' case." 

The FBI hair comparison experts were found to have made "erroneous statements" in about 96 percent of the studied cases in which "examiners provided testimony used to inculpate a defendant at trial." In 33 cases, errors were found in the analysis of defendants who were subsequently sentenced to death. Of those defendants, nine have already been executed, and five others died while on death row.

Worse still, this bad hair science is just the latest example of phony forensics. Another whole field of forensic science, compositional bullet lead analysis, was shown to be bogus in a 2004 National Academy of Sciences study. The FBI had been testifying that the chemical composition of a bullet could identify it down to the maker, or even the batch, or even, in some cases, the box. No, said the study: "The available data do not support any statement that a crime bullet came from a particular box of ammunition." 

And then there's the heavily questioned science of bite-mark analysis, heavily reported by former Reason editor Radley Balko. A 2002 study found a "false positive" error rate of 64 percent in bite-mark analysis. The Chicago Tribune reported that the study's author "figured that on average, they falsely identified an innocent person as the biter nearly two-thirds of the time." Fortunately, the FBI does not do bite-mark analysis. Unfortunately, other labs do. 

A 1992 study showed that many traditional arson investigation techniques were bogus. And yet Texas convicted Cameron Todd Willingham of murder mostly on the basis of those very techniques. In 2004, 12 years after the release of the report discrediting the crucial techniques used in Willingham's case, he was executed for his supposed crime.

Even fingerprints and DNA can go wrong. Fingerprints are pretty reliable when both the "known" and "unknown" images are clear and distinct. But the "unknown" image is often far from clear and distinct. The unknown image might be smudged, a small partial print, overlain by other possibly smudged prints, or deposited on an irregular surface like wood grain. In those cases errors become more likely. 

In 2004, the FBI made a "100 percent match" of a print from the deadly Madrid train bombing to Portland area lawyer Brandon Mayfield. They turned out to be 100 percent wrong, however. The FBI later apologized to Mayfield, who claimed to have been profiled because he was a convert to Islam, and paid out $2 million to settle a suit he had filed against them. In another famous misidentification, that of Shirley McKie, a Scottish police agency was found to have mistaken wood grain for fingerprint ridges!

In ideal conditions, DNA is our most reliable forensic technique. Conditions are less than ideal if the crime-scene sample is small or corrupted or if it has the DNA of more than one person mixed together in it. And we have seen mistakes there, too. Josiah Sutton was convicted of rape largely on DNA evidence that was later shown to be bogus. He was convicted and imprisoned at the age of 16 and released more than four years later.

What in the world is going on here? It's partly bad science, partly bad organization, and wholly unacceptable.

On television shows, forensic evidence is super-scientific and infallible. And yet we have seen over and over again in the real world decidedly unscientific techniques being used. A 2009 study by the National Academy of Sciences was blunt and plainspoken: "The bottom line is simple: in a number of forensic science disciplines, forensic science professionals have yet to establish either the validity of their approach or the accuracy of their conclusions, and the courts have been utterly ineffective in addressing this problem." 

Subjective judgment is a big part of the problem. In hair microscopy, bullet-lead analysis, and fingerprints the forensic scientist is asked to make a subjective judgment of similarity. Even many cases of DNA analysis require subjective judgment. That doesn't seem very scientific. 

And in most cases, public crime labs, such as the FBI lab, are a part of the law enforcement agency, not independent reviewers. This is where bad organization factors in. The forensic scientists are supposed to be neutral, but they're working for the cops. A 2013 study (that I co-authored) shows that many public crime labs are funded in part per conviction. And in 14 states, such a financing system is required by state law.

If you work for the cops, you're bound to end up seeing things from their angle no matter the desire for objectivity. That might be okay if forensic analysis actually was objective. It is not okay, however, when forensic analysis is subjective, which it often is. Add in a specific financial incentive to get convictions and you are pretty much asking crime labs to interpret everything as incriminating. 

The organizational problems of forensic science are compounded by monopoly. Once evidence goes to a given crime lab, it is unlikely to be examined or interpreted by any other lab. Thus, we do not usually get a reality check on the work a crime lab does. That lets errors slip through undetected. And errors can go undetected for years or even, perhaps, forever. 

With the right reforms, we can create a working system of checks and balances. Here are a few ideas Reason has promoted in the past. Maybe their time has finally come:

  • Cross-lab redundancy. A jurisdiction should contain several competing forensic labs. Some evidence should be chosen at random for multiple testing at other labs. This creates checks and balances.
  • Independence. Put crime labs under the department of health, not the cops.
  • Statistical review. Compare the results of different labs and look for statistical anomalies. An investigation may reveal bad practices to be eliminated or good practices to be emulated.  
  • Sequential unmasking. Forensic scientists can be biased by scientifically irrelevant information such as the criminal history of the suspect. Sequential unmasking is an administrative control process similar to that used in double-blind research studies. It prevents forensic scientists from learning potentially biasing information until after they have made their determinations. 
  • Forensic counsel for the indigent. Most criminal defendants cannot afford their own forensic experts. Basic fairness says that they should have a right to their own experts just like they have a right to counsel. A voucher system is the best way to provide defense experts.

Measures like these could help turn forensic science from a scandal pit into a source of improved criminal justice. That seems like a worthy goal in the context of today's troubled criminal justice system.

NEXT: Did Rand Paul Betray His Criminal Justice Reform Outreach on Baltimore?

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  1. The history of the F.B.I. has been one of lawlessness.

    1. I make up to $90 an hour working from my home. My story is that I quit working at Walmart to work online and with a little effort I easily bring in around $40h to $86h? Someone was good to me by sharing this link with me, so now i am hoping i could help someone else out there by sharing this link... Try it, you won't regret it!......

  2. The history of the F.B.I. has been one of lawlessness.

    1. The history of this comment is one of repetitiveness...

      1. Hey, at least Adans was consistent! He didn't try to walk that statement back in his second post.

        You have to admire him for staying true to his initial statement.

        1. Thank you,they truly suck,and all agents are reported to be so well schooled,collage and all.

  3. Look, do you want people put away quickly or do you want to waste tax dollars on lengthy, competent investigations?

    1. neither?

      1. no competent investigations for you, eh?

    2. The right to a speedy conviction?

  4. Is it forensic science or alchemy?

      1. Oh, Edmund... can it be true? That I hold here, in my mortal hand, a nugget of purest green?

        1. Shouldn't discussion of the Baby-Eating Bishop of Bath and Wells be in the thread about pork?

          1. By the time that district gets around to it, it's more like the teenager eating Bishop of Bath and Wells.

            1. Ahem.

              Baldrick: What about this priest?
              Blackadder: Tell him to get his sacred backside out of here! And what's more, if he comes begging again tell him I shall report him to the Bishop of Bath and Wells, who drowns babies during christenings and eats them in the vestry afterwards.
              [Baldrick leaves and then suddenly is thrown through the bedroom door with a loud crash.]
              Blackadder: What is it, Baldrick?
              Baldrick: It's that priest, he still wants to speak to you.
              Blackadder: And did you tell him about the baby-eating Bishop of Bath and Wells?
              Baldrick: I did, my Lord.
              Blackadder: And what did he say?
              Bishop of Bath and Wells: [entering] He said I AM THE BABY-EATING BISHOP OF BATH AND WELLS!
              Blackadder: Good Lord!
              Bishop of Bath and Wells: You haven't any children, Blackadder?
              Blackadder: No, I'm not married.
              Bishop of Bath and Wells: In that case, I'll skip breakfast and get straight down to business!

        2. Is that a pot reference?

            1. Thanks. Haven't watched that show.

    1. "The tire tracks are identical."

      *claps hands*

      1. Ha!

      2. + Marisa Tomei

      3. Florida Man, are you on druugs?

        1. *tosses glass pipe into sewer drain*

      4. Utes!

    2. If she weighs the same as a duck....

    3. I understand you can get lots of evidence from the bumps on a persons head .

      1. They reveal the shape of their brain, and thus the shape of their mind!

  5. I'm sure that forensic science is as rigorously scrupulous about objective peer review and transparency of methodology as the science of climate...

    1. 99% of cops agree!

  6. So - the science is settled

  7. But they're "experts"' That means everything they say is gospel. I can't read chicken bones or tea leaves therefore I have no choice but to believe those that can. Especially if they work for the authorities.

    1. They just need more zoom, and resolution enhancement, and cleaning up the image; a flashing target icon accompanied by a swooshing sound also improves stern focus on the evidence.
      And they need an intense soundtrack. Then most of the crime mysteries will get solved. They need to increase their budget to protect the innocent women and children from monsters.

      1. and a computer will fill in the missing images,because,science

      2. Shut up Slammer!

        I have relatives who aren't so wise in the ways of science who have sent me old pics they took on crappy digital cameras and really want me to use my computer-fu to enhance them just like on TV.

        1. Yeag, I love when they depixalate some microscopic distant image to be full size with perfect resolution

    2. So - the science is settled

      As surely as witches are made out of wood.

  8. Dear ISIS,

    Please, please, send more "operatives" to attack places like Texas. While, considering the "success" of the Brothers Tsnarnaev, you might be considering places like Boston, let's be honest; attacking Boston is like picking on the retarded kid in class. We're talking about a city that shut down because of a Light Bright toy. Texans, on the other hand, will shoot back. And their aim is true. So please, if you're going to martyr your young men, allow Texas to martyr them effectively and with as little disruption as possible.

    As an aside, I have yet to come across a mainstream media outlet that will deign to show images of the Geller event that include the older Rasta-looking Black guy in the audience, despite him being quite prominent in videos of the event. Narrative-alicious!

    1. Texas art gallery 2, assholes 0

    2. Boone county WV would be a good place to start. May not get far over the county line.They're raised on shotguns,lol

    3. I am going to go out on a limb here and say that even black guys with dreadlocks don't feel enough solidarity with their fellow oppressed brown people to want to be murdered either.

      1. That may be true, but CNN, at least, hasn't been showing the same concern for the safety of elderly, Hawaiian shirt wearing White men. So, I don't know why they would go out of their way to protect Mr. I and I.

        1. Because Hawaiian shirt wearing old white guys are by virtue of their existence just asking to be murdered.

          1. Well, yes, the man who tucked his shirt into his jeans does deserve death for his crimes against style.

          2. Would it help any if they were Balanese Batik?

    4. I see that the Daily Mail blacked out pics of the comics in their story about the shooting.


    5. Kinda reminds me of a great alien-invasion-of-Texas-gone-wrong short story by Howard Waldrop called, "Night Of The Cooters".

    6. If you really want to meet Allah we can arrange it. Please consider the rifling on the outer surface of the invitation you receive to be engraved just for you.

      1. Well done.

      2. Something I just felt like sharing...

        When a sniper shoots a round at you it definitely has your name on it. When a machinegunner or a mortarman fires a round at you the addressee is "to whom it may concern".

  9. Cops lie in court? I'm shocked! Shocked! Perjury prosecutions should be mandatory when testimony or evidence suggests perjury.


  10. This of course will never happen, but in a sane world the lab would be shut down. This is not one or two mistakes or one or two bad employees. This is a pervasive culture of dishonesty and fraud. You can't fix an organization this broken. The only way to solve the problem is to shut the lab down altogether.

    1. Well if I'm a defence attorney and that lab is presenting evidence, you better believe I'm bringing this up.

      1. Every single case. And if I am on a jury, no way am I trusting anything this lab says. And you make a good point, the lab is officially worthless now. Even if it fixes itself and does good work, no one will believe it.

        1. CSI fans disagree.

          1. That show is so awful.

            1. Show? Most fans believe it's a documentary.

          2. "Let's give that Coke can the full CSI treatment...that's it, a lot of flashy special effects that accomplish nothing."

            Chief Wiggum

            1. I have a friend who is a no kidding forensic pathologist. I can't tell you how much she hates that show. Everything in it is utter bullshit.

              1. I really don't know anything about the science but I figured the speedy turnarounds were ridiculous.

                I watched them for a few years but the plot lines were just so stupid. Basically every week it was: crime occurs, we investigate one angle; despite having a magical laboratory and multiple investigators we won't run down any leads; we'll accuse someone then in the interview find out he has an alibi THEN we'll go and do more research.

              2. What?! Even when the computer scans through a bunch of images and pops up "MATCH"?!

  11. Government employees should receive double penalties upon conviction of crimes.

    1. You'd have to get them charged first.

    2. Most of them get double-secret probation. With pay. Because FYTW.

  12. In addition to those steps listed, send some known test samples through one out of a hundred times -- two hairs from the same person, from identical twins, and complete strangers. The results would be fascinating; I bet they would show it for the pseudo science much of it is.

  13. Something I don't understand: Libertarians tend to favor reducing the power of law enforcement agencies and curbing law-enforcement abuses through reforms. But Libertarians also tend to support Republicans, who have always been the "tough on crime", pro-law-enforcement, pro-war-on-drugs (Ronald Reagan was the biggest champion of the war on drugs of all presidents in my voting lifetime) party. When the Dems have supported tough-on-crime policies and used tough-on-crime rhetoric, it was always in a half-hearted, "me-too" sort of way--Bill Clinton tried to make himself appear tough on crime, but it was an obvious attempt to steal the issue from the Republicans.

    Isn't there a contradiction here? Why do Libertarians support a party which is so bad on this issue?

    Now I'm going off-topic from the article, but as long as I'm talking about contradictions in Libertarian support for Republicans, here's another one: Libertarians tend strongly to be pro-choice on abortion. With just two aberrant exceptions, the Libertarian platform has always been pro-choice (as it is now--it calls for government to stay out of the matter entirely, which is a pro-choice position.) (The two exceptions were when Pat Buchanan's people temporarily took over the party and when Ron Paul ran on a right-to-lifist platform.) But again, Libertarians tend to support Republicans, who are anti-freedom on this issue, not just by rhetoric but by actual policy, especially now.



    Even on taxes, a strong argument can be made that Republicans have an anti-libertarian policy, at least as anti-libertarian as the Democrats. Republicans favor tax cuts for the top thousand or so richest families, whose wealth was accumulated in large part, by means of anti-libertarian activities--taking advantage of government-enforced monopoly and oligopoly power (which is how Ross Perot made his fortune); getting money from contracts with the government (like the owner of Blackwater/Xe and all the members of the military-industrial complex), and, once ones fortune reaches a certain level, buying politicians who pass laws favorable to their patrons, enabling them to become richer, and even going into politics and using ones office to enrich oneself and ones friends (like the Bush dynasty).



    Even natural-science-based high-tech multi-millionaires and billionaires, scientists who enter the private sector and found companies like Genentech, all got their advanced scientific training on the public dime and started their companies with government startup money. I used to work for a small biotech firm inventing drugs, and what enabled them to succeed and get rich was largely their ability to interact successfully with Federal regulatory agencies like the FDA, using personal connections which the president and VP of the company made while they were post-docs in NIH labs.

    Why should LIbertarians support tax-cuts for the upper-bracket, the overwhelming majority of whom accumulated the majority of their wealth using government, which means essentially, by theft???

    1. My take is that when faced with a choice between a party that gives lip-service to economic liberty, and a party that is openly hostile to any kind of liberty other than abortion, libertarians tend to support the former.

      1. You say the Dems are "openly hostile to any kind of liberty other than abortion"??? Rubbish. The Dems (mostly) support liberty to love and marry whom you want; the Republicans (mostly) oppose this. The Dems tend to support freedom of religion; the Republicans have become the party of Christian theocrats who call for mandatory church attendance and argue that the First Amendment should not apply to Jews and Muslims,. A Democrat president (J. Carter) granted amnesty to draft-resistors, something no Republican would ever have done.

        1. Is......is this a troll?

    2. There is a word in your handle that does not belong there and it's neither "Mr." nor "Toad."

      1. What, you mean left wing CT isn't the mark of a true genius?

      2. You never read THE WIND IN THE WILLOWS? Who brought you up???
        Anyway, I don't like to brag, but I did graduate cum laude from an Ivy League school, and I do hold a masters degree in physical chemistry from a university where they have special parking places marked "NL" which are reserved for exclusive use by Nobel laureates.....

    3. Let me guess, your solution to government monopolies is more government? Also more theft, more laws, more regulations, more of what created a system where cronies get rich. Here's a crazy idea. How about less to no government. No hands in your pocket, no thumbs on the scale, people interacting freely. I know this is scary because you are lazy and want people to steal from others and give it to you because you deserve it for drawing breath somehow. Please go join your socialist country of preference and leave the rest of us alone to try and revert the U.S. to an experiment in liberty.

    4. Libertarians tend to support Republicans? Based on what evidence exactly? I'm a libertarian and I haven't voted for a Republican in over 10 years. Saying that you agree with Republicans on certain points doesn't equate to supporting them. I support Democrats when they actually happen to propose actions that enhance individual rights and freedoms, which, admittedly, is rather seldom these days. I was certainly supporting their anti-war protests against the Bush administration, which amazingly vanished into thin air as soon as the current POTUS came into office, despite the fact that the wars continued.

    5. the overwhelming majority of whom accumulated the majority of their wealth using government

      [citation needed]

      1. You didn't build that!

  16. Google pay 97$ per hour my last pay check was $8500 working 1o hours a week online. My younger brother friend has been averaging 12k for months now and he works about 22 hours a week. I cant believe how easy it was once I tried it out.
    This is wha- I do...... ?????? http://www.netjob80.com

  17. In 33 cases, errors were found in the analysis of defendants who were subsequently sentenced to death.

    Whoa. Talk about having a bad hair day.

    1. Dammit! My lack of refresh skills has really been biting me in the ass lately.

      1. Happens to us all. Well, except Fist.

  18. The FBI: having bad hair days since 2000.

  19. Government has been very comfortable for many years using bogus "science" to destroy the lives of their prey. Originally it was used (abused?) mostly in the law enforcement fields, however much more so in recent decades it's been widely used for destruction on a much grander scale; EISs, bogus climate theories and so forth.

  20. Any industrial company that maintains its own laboratories for quality control will perform gauge r&r studies to determine the variability of their labs' data. They will continuously do round robin studies to compare each lab against all the others, and to ensure consistency and accuracy in the results. They will identify deficiencies and correct them.

    It doesn't sound like these government forensic labs are managed in any sort of appropriate way.

    1. Industrial companies rely upon customers voluntarily purchasing their goods and services, so they have an incentive to do things right.

      Governments will lock people up in a cage if they refuse to pay for goods and services that they neither want nor need, so government has no incentive at all to do anything right.

    2. Appropriate?!? You be plenty naive, kemo sabe.

    3. When industrial entities use shoddy and bogus science the result is very expensive failure. When law enforcement use shoddy and bogus science the result is big promotions, launching of high paying powerful political careers and so forth.

      Why don't we just go back to making suspects hold onto red hot iron bars and such? You know, if it burns them then they've been "scientifically" proven to be guilty. If it doesn't burn them "science" has proven them innocent.

  21. In 33 cases, errors were found in the analysis of defendants who were subsequently sentenced to death.

    "Hair today; gone tomorrow."

    Seriously, this shit is shit.

  22. I agree in general with this post, but must quibble on one point;

    "Subjective judgment is a big part of the problem. In hair microscopy, bullet-lead analysis, and fingerprints the forensic scientist is asked to make a subjective judgment of similarity. Even many cases of DNA analysis require subjective judgment. That doesn't seem very scientific.

    Actually, all science involves some judgement, and judgement is necessarily subjective. So, subjective judgement is part of the problem, but a bigger one is the worship of Science as an accepted substitute for an infallible God.

    This cuts to the core of the old (and tiresome) battle between ignorant Christians and ignorant Atheists. The Atheists are entirely too prone to claim that some aspect of Science (such as Evolutionary theory) is proven and settled, completely ignoring that, as it is understood and taught in the public schools (for example) it is an assertion of faith not a matter of demonstrated fact.

    People put faith in "Science" that science does not deserve.

  23. My sister makes $75 every hour on the laptop . She has been laid off for seven months but last month her pay check was $18875 just working on the laptop for a few hours.
    Look At This. ????????? http://www.jobsfish.com

  24. dishing out clap-trap

    Re-sist the im-pulse to in-sert hy-phens in words of two syllables.

  25. Google pay 97$ per hour my last pay check was $8500 working 1o hours a week online. My younger brother friend has been averaging 12k for months now and he works about 22 hours a week. I cant believe how easy it was once I tried it out.
    This is wha- I do...... ?????? http://www.netjob80.com

  26. Google pay 97$ per hour my last pay check was $8500 working 1o hours a week online. My younger brother friend has been averaging 12k for months now and he works about 22 hours a week. I cant believe how easy it was once I tried it out.
    This is wha- I do...... ?????? http://www.netjob80.com

Please to post comments

Comments are closed.