Breaking Up the Forensics Monopoly

Eight ways to fix a broken system

Roger Koppl from the November 2007 issue

America’s forensics system, the part of our criminal justice system responsible for scientific examinations of crime-scene evidence like fingerprints and DNA, is rife with errors. Some mistakes, like botched tests or erroneously interpreted results, are inevitable. But current error rates are needlessly high. The most recent comprehensive study of crime lab proficiency, published by the Journal of Forensic Sciences in 1995, analyzed the tests administered by the Forensic Sciences Foundation and Collaborative Testing Services as a part of the accreditation process. For many forensic disciplines, including the analysis of fibers, paints, glass, and body fluid mixtures, the rate of incorrect matches between recovered evidence and a reference sample exceeded 10 percent.

The best-performing group of disciplines, which included “finger and palm prints, metals, firearms, and footwear,” had error rates at or above 2 percent. The first item on that list is especially important: False fingerprint identification usually leads to a false conviction, because of the prestige of fingerprint evidence and its undeserved reputation for infallibility. With 238,135 requests for latent fingerprint comparisons in 2002 alone, a false positive error rate of 2 percent implies up to 4,800 false convictions or guilty pleas made in hopes of a lighter sentence each year in the U.S., 1,700 of them in felony cases. (The number of improperly matched fingerprints is not completely clear. A 2005 study of fingerprint analysis suggests that the false positive rate may now be as low as 0.8 percent. But another recent study suggests it could exceed 4 percent.)

Confronted with such statistics, policy makers usually call for greater oversight—that is, finding a governmental body to watch over forensics and make sure everyone does his or her job right. In the current climate, that certainly would help. But the core problem with modern forensics isn’t an absence of oversight. It’s monopoly. Once evidence goes to a given lab or facility, it is unlikely to be examined by any other lab or facility. That increases the chances that a mistake will slip through undetected.

With the right reforms, we can break down that monopoly and create a working system of checks and balances. Here are eight steps to a better system:

1. Rivalrous redundancy. A jurisdiction should contain several competing forensic labs. To the extent that it’s feasible, some evidence should be chosen at random for multiple testing at other labs. The same DNA evidence, for example, might be sent to three labs for analysis. The forensic scientist would not know when a given piece of evidence was being examined by another lab.

For fingerprints, multiple examinations should be routine. If the rate of false positive fingerprint error is 2 percent, triplicate examinations would eliminate 96 percent of false felony convictions due to misidentified fingerprints.

2. Independence. Coroners and forensic scientists often have a pro-police bias, thinking of themselves as a part of the prosecution team. To establish their independence from police and prosecutors, crime labs should be organized by the courts, not the cops.

3. Statistical review. Statistical review would lead to improved quality control. For example, if a given lab produces an unusually large number of inconclusive findings, its procedures and practices should be examined by an officer of the court.

4. Masking. When conducting forensic analyses, coroners and forensic scientists should be shielded from what psychologists call “domain-irrelevant information.” Knowing whether the case at hand is, for example, a murder or a burglary exposes a fingerprint examiner to a powerful unconscious bias: The emotional nature of a murder case tends to make the scientist eager to get a killer off the streets and more likely to declare a match. In a 2006 study by researchers at the University of Southampton, domain-irrelevant information doubled the error rate of experienced fingerprint examiners.

5. Forensic counsel for the indigent. Although forensic science decides many criminal cases, we do not have a right to forensic counsel similar to our right to legal counsel. Just as an indigent defendant has a right to the help of a qualified attorney, an indigent defendant should have the right to the help of a scientist qualified to interpret forensic analyses.

6. Forensic vouchers. An indigent suspect on trial should also have the right to select his own forensic counsel and use a government-issued voucher to pay for it. The forensic scientist who accepts the case would later redeem the voucher at the courthouse, receiving his paycheck from an officer of the court. Such a system would give forensic counselors to the poor an incentive to provide high-quality services.

7. Division of labor between forensic analysis and interpretation. A forensic scientist who conducts a blood test, for example, should not say whether the test excludes the suspect. The interpretation of the test should be made by other forensic scientists. When a lab report comes back, it should be transmitted to two forensic scientists—one representing the prosecution and one representing the defense—for interpretation. Combined with public funding of forensic experts for defendants who cannot afford them, this will make it less likely that errors of interpretation will go unchallenged.

8. Privatization. Private labs are subject to civil liability and administrative fines for poor performance. They therefore have stronger financial incentives than publicly owned enterprises to provide good and reliable work.

While those eight reforms would establish a system of competitive self-regulation, they are not an exhaustive description of good forensic practice. For example, labs should be accredited by groups such as the American Society of Crime Laboratory Directors’ Laboratory Accreditation Board, and they should have routine procedures to measure variables related to quality and a planned system of review to allow those procedures to be updated regularly.

These eight steps would also reduce the costs of the criminal justice system. The extra cost of multiple forensic tests is dwarfed by the savings associated with reduced jail time for the wrongly convicted. For example, the $100 cost of a fingerprint examination is one one-thousandth of the cost of incarcerating a wrongly convicted felon who has been given the average sentence of almost five years.

In Federalist No. 51, James Madison endorsed a “policy of supplying, by opposite and rival interests, the defect of better motives,” in order “to divide and arrange the several offices in such a manner as that each may be a check on the other; that the private interest of every individual may be a sentinel over the public rights.” It is time to make the private interest of every coroner, every medical examiner, and every forensic scientist a sentinel over the public rights. r

Roger Koppl, a professor of economics and finance at Fairleigh Dickinson University, is the founder and director of the university’s Institute for Forensic Science Administration (IFSA) and author of “CSI for Real: How to Improve Forensic Science,” a forthcoming study from the Reason Foundation. IFSA’s website, alpha.fdu.edu/~koppl/ifsa.html, contains more information on competitive self-regulation.

Comments

Peder|10.24.07 @ 1:05PM|#

Are there enough forensic experts for the reforms that Mr Koppl is recommending?

dbust1|10.24.07 @ 1:10PM|#

Probably not. So let's create another bloated gov. agency to take care of the problem.

|10.24.07 @ 1:18PM|#

How many wrongful convictions are caused by breath machines?We see police using a magic box for a money maker and yet it wasn't mentioned.

|10.24.07 @ 1:24PM|#

There certainly should be a ton of male forensic scientist students in the pipeline...at least if the television depiction of female forensic scientists
(all stacked, with low cut blouses) is true to life.

|10.24.07 @ 1:35PM|#

She blinded me with SCIENCE!

Russ 2000|10.24.07 @ 1:38PM|#

To establish their independence from police and prosecutors, crime labs should be organized by the courts, not the cops.

Yes, because the courts aren't pro-police.

/sarcasm

Russ 2000|10.24.07 @ 1:45PM|#

With 238,135 requests for latent fingerprint comparisons in 2002 alone, a false positive error rate of 2 percent implies up to 4,800 false convictions or guilty pleas made in hopes of a lighter sentence each year in the U.S., 1,700 of them in felony cases. (The number of improperly matched fingerprints is not completely clear. A 2005 study of fingerprint analysis suggests that the false positive rate may now be as low as 0.8 percent. But another recent study suggests it could exceed 4 percent.)

This has me curious as to the error rate on DNA comparisons. I know I supposed to believe that DNA comparisons are infallible, which is exactly why I don't beleive they are.

|10.24.07 @ 1:45PM|#

Private labs are subject to civil liability and administrative fines for poor performance.

To me, this is the killer. In the pharmaceutical industry, we have to produce mountains of data and paperwork to get FDA approval. Once we get that approval, it in no way protects us from lawsuit if there was something that we (and the FDA) missed. If an idiot in the police lab determines that the gun "must have" been fired by two people simultaneously, there's nothing the defendant can do once the evidence is exposed as garbage. Not to the lab, not to the forensic scientist who was clearly just making shit up.

|10.24.07 @ 1:50PM|#

I'm to the point now, where if I were on a jury, I would require non-enforcement confirmation of everything before I was ready to convict. There would need to be a third-party confirmation of forensic evidence and a third-party eyewitness confirmation of any account given by the LEO. Otherwise, I'm not gonna be able to overcome "reasonable" doubt.

|10.24.07 @ 1:58PM|#

There would need to be a third-party confirmation of forensic evidence and a third-party eyewitness confirmation of any account given by the LEO. Otherwise, I'm not gonna be able to overcome "reasonable" doubt.

I can't say I'd blame you.

Dave Woycechowsky|10.24.07 @ 2:25PM|#

I'm to the point now, where if I were on a jury, I would require non-enforcement confirmation of everything before I was ready to convict. There would need to be a third-party confirmation of forensic evidence and a third-party eyewitness confirmation of any account given by the LEO. Otherwise, I'm not gonna be able to overcome "reasonable" doubt.

Which is why the OJ jury did the right thing, even though OJ really did do those murders.

Dave Woycechowsky|10.24.07 @ 2:35PM|#

Oh, yeah, and there should be parity of funding for criminal prosecution and criminal defense. Neither should the defense be allowed to spend more than the prosecutors nor vice versa.

That would allow a more efficient, market driven solution to the problems ably noted by Mr. Koppl than what he proposes. It would solve other bad problems, too.

|10.24.07 @ 2:45PM|#

Which is why the OJ jury did the right thing, even though OJ really did do those murders.

Dave Woycechowsky,
Your post really pisses me off. Why, because it is true. Shit.

Dave Woycechowsky|10.24.07 @ 3:00PM|#

Your post really pisses me off. Why, because it is true. Shit.

I mean we know he did it because we know that if there was a police conspiracy, then the media would have been motivated to uncover it, or raise suspicions or something.

OJ had friends in the media and other high places, and the media and/or people in other high places can crack a police conspiracy if they are so inclined. They didn't. Ergo, there was no police conspiracy in the OJ case. Deep down, that is the real reason we were all so sure that he did it.

But, if you are the juror, you don't get to consider that stuff. You can consider the general reputation of police, and you get to consider credible and demeanor evidence offered in the trial itself. But, you don't get to consider what the media said and (more importantly) did not say. That is true even though a consideration of the full panoply of media investigations / coverage / commentary would have lead to a more objectively accurate verdict in OJ's case.

It is comforting to know that OJ's case is the exception in this regard. In many cases, if you make your decision based on media accounts, then you will make a worse decision than you would by following the trial.

sv|10.24.07 @ 3:47PM|#

This all sounds like it would go a long way towards solving the problem put forth. It also sounds really, really expensive, if it's feasible (enough practitioners, etc.), and besides, the public probably doesn't care about those accused of crimes, and even if it did, they probably wouldn't want to shell out for it, and even if they might be persuaded to do THAT, the government would surely interpret this as being a soft-on-crime strike against them (us vs. them) and find a way to scuttle it, probably budgetary fudging.

Avoid government, keep your nose clean, avoid police, type rant-y run-on sentences on blogs to vent, carry on.

sv|10.24.07 @ 3:52PM|#

re: the idea of the costs being offput by the savings in not incarcerating the innocent: like the govt. is interested in that (drug war anyone?)

Jessica|10.24.07 @ 9:28PM|#

While I in no way think San Antonio represents the state of forensics country-wide, based on my experience as a juror in a murder case....Some of Koppl's suggestions are starting to be used. There is room for improvement though.

Specifically, the suggestions of Privatization and the Redundancy were followed for DNA testing but not fingerprinting. For my jury case, the DNA testing was done by an outside lab who then verified their results by sending out to another lab (who had no info on what the test was for). Both reports were submitted to the jury for evidence.

But I do see your point about independence in fingerprinting analysis. In SA, the people ID-ing the fingerprints are in the Evidence division of the police force. If I remember correctly the main fingerprint guy and the person who double-checked his analysis didn't visit the crime scenes. BUT they were both aware of what case the print ID was for so your point about masking is valid.

It seems to me the newer tests for DNA have better control procedures possibly because there's less of a history of poor practice to overcome. But the evidence development should be more consistent between methods (DNA vs. fingerprinting vs. hair&fiber). Just my 2cents.

Iain McKie|10.25.07 @ 6:10AM|#

Roger Koppl does us all a service by highlighting the problem of forensic science error rates.

Separating the scientists from the police is an essential first step. The former are not involved in a 'war against crime' nor should they become caught up in the culture that such a battle creates. The scientist's role is the objective analysis of data free from emotional and cultural prejudice.

In addition the 'adversarial' approach, where one expert is set against another and where clarity and understanding are the biggest losers, should be dropped and the court should have the power to call its own experts to ascertain the truth affording all defendants and prosecutors a level playing field.

For more on the problem worldwide see www.shirleymckie.com

Crime Lab Report|10.25.07 @ 8:58AM|#

We have been following professor Koppl's comments over the last several months, and will continue to monitor his characterizations of forensic science.

It would be inappropriate for our editorial board to describe all of Professor Koppl's recent recommendations as being unworthy of consideration. However, we are interested in learning what fiduciary advantage he creates for himself and his university by inaccurately portraying the forensic sciences as being inadquate and "rife with errors." We challenge Professor Koppl to present the statistical evidence for this claim.

We also wonder if his ultimate goal is to secure grant monies to fund the research he would need to test the validity of his recommendations. If this is the case, which we believe it is, Professor Koppl and many of his colleagues in academia should have the professionalism to acknowledge their conflict of interest and refrain from making the ad hominem attacks that they have come to be known for.

If you would like more information about forensic science and how it is portrayed in the media, please visit us as http://www.crimelabreport.com.

|10.25.07 @ 5:16PM|#

My brother used to laugh when anyone mentioned forensic "science" to him. He told me that as a research scientist he would be lucky get 2-5% of his results to be repeatable and verifiable to the standard required for publication. That's under the strictly controlled environment of a university chemistry lab. Yet forensic "scientists" are able to get 100% results from a pair of old underpants buried in a swamp for ten years or find foetal blood sprays under dashboards where none exist (google Australia, lindy chamberlain), and send someone to prison for the rest of their life based on their "evidence".

Most of the disasters which result from forensic evidence are inevitable because they are based not on verifiable scientific techniques but on subjective estimates and educated guesses, which is mostly what forensic labs produce. Even the magic of DNA evidence will wear off once the mistaken convictions resulting from mishandling and misinterpretation (no, it's not just matching the dots despite what's on TV) start filtering through.

Forensic technology has an important role to play in helping police identify or eliminate suspects and to try to reconstruct the timing and scenario of crimes - especially violent crimes. But it should never be used in a court as evidence. As many people are beginning to realise, it never is!

|10.25.07 @ 6:59PM|#

Where I work, we have been preparing for three years to be an accreditated lab so we can test materials for the states of Washington and Oregon. We have to have a complete quality control system in place, including procedures for calibration of equipment, rules for reporting information, and so on. We've been receiving samples almost every month for the last three years to test our proficiency, we've spent thousands and thousands of dollars on equipment and training, and we have two massive inspections by outside agencies before we can finally be approved. And this is all just so we are able to test soil and rock for highway jobs. Maybe if we were able to send people to prison the requirements wouldn't be as strict.