Junk science

A Famous Study Found That Blind Auditions Reduced Sexism in the Orchestra. Or Did It?

More implicit bias research comes under scrutiny

|

One of the best-known scientific studies to posit that implicit bias—the idea that all people are unconsciously racist, sexist, etc.—can be counteracted via strategic effort is taking a well-deserved beating. It now appears that the findings were significantly overstated.

The study, "Orchestrating Impartiality: The Impact of 'Blind' Auditions on Female Musicians," by Harvard University's Claudia Goldin and Princeton University's Cecilia Rouse, was released in 2000. Its bombshell finding was that blind orchestra auditions—which prevented the choosers from seeing whether each auditioner was male or female—increased female auditioners' odds by 50 percent. The American Enterprise Institute's Christina Hoff Sommers notes that the study was "lionized by Malcolm Gladwell, extolled by Harvard thought leaders, and even cited in a dissent by Justice Ruth Bader Ginsburg." Jesse Singal, a contributing writer at New York magazine who has often criticized bad social science (and is writing a book about misleading statistics that have gone viral), has actually cited the orchestra study as one of the more important entries in the field of implicit bias. In a review of Jennifer Eberhardt's book, Bias, Singal wrote:

Eberhardt also presents some difficult-to-refute findings about the role of implicit bias in the real world, including famous studies in which identical résumés are sent out with white- and black-sounding names (with the white ones getting far more callbacks) and another famous experiment in which a screen shielding performers' identities during auditions led to more gender parity in previously male-dominated orchestra hiring. The results of these tight, elegant experiments suggest that implicit bias is at least part of the equation. And in some cases they offer clear partial solutions, such as shielding certain identifying information about job applicants and focusing more on tests of ability to perform the tasks required by a position.

My point is that the in an area of research fraught with replicability problems, the orchestra study was supposed to be one of the good ones.

Well, so much for that. In May, Columbia University statistician Andrew Gelman took a deep dive into the study. He described them as "not very impressive at all," and had great difficulty trying to locate the 50 percent statistic within the modest findings.

"You shouldn't be running around making a big deal about point estimates when the standard errors are so large," he wrote. "I don't hold it against the authors—this was 2000, after all, the stone age in our understanding of statistical errors. But from a modern perspective we can see the problem."

Sommers wrote about this discovery in a recent piece for The Wall Street Journal, which will probably attract more attention. She noted the existence of another study that had contradicted Goldin and Rouse:

In 2017 a team of behavioral economists in the Australian government published the results of a large, randomized controlled study entitled "Going Blind to See More Clearly." It was directly inspired by the blind-audition study. Iris Bohnet, a Harvard Kennedy School dean and Goldin-Rouse enthusiast, served as an adviser.

For the study, more than 2,000 managers in the Australian Public Service were asked to select recruits from randomly assigned résumés—some disguising the applicant's sex, others not. The research team fully expected to find far more female candidates shortlisted when sex was disguised. But, as the stunned team leader told the local media: "We found the opposite, that de-identifying candidates reduced the likelihood of women being selected for the shortlist." It turned out that many senior managers, aware that sexist assumptions had once kept women out of upper-level positions, already practiced a mild form of affirmative action. Anonymized hiring was not only time-consuming and costly, it proved to be an obstacle to women's equality. The team plans to look elsewhere for solutions.

Blind interviews and auditions may be preferable for other reasons. They may even reduce implicit bias in some situations. But as is so often the case, the sweeping claims of social scientists do not seem to survive scrutiny.

It will be interesting to see if diversity coordinators—many of whom incorporated this shoddy scholarship as part of their training seminars—adjust course, though I wouldn't count on it. The entire concept of microaggressions lacks scientific legitimacy, after all, but this hasn't stopped college diversity czars from policing them.

Sommers has more on the orchestra study—in video format—here.

NEXT: Seattle Public Schools Will Start Teaching That Math Is Oppressive

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

Please to post comments

62 responses to “A Famous Study Found That Blind Auditions Reduced Sexism in the Orchestra. Or Did It?

  1. Orchestral bias not a real thing? Luckily people today have greater options in the area of victimhood than we did back then. I’m sure everyone will find something else that can seem to fit.

  2. Dammit, there has to be bias here somewhere! *shuffles papers*

    1. Your bias is showing. Woke people do not need “evidence” or “reasoning”. (And certainly not math–see the article on Seattle.)

      1. Correction: I feel the bias.

  3. I think this article is biased against those who prefer rock and roll.

  4. Anonymized hiring was not only time-consuming and costly, it proved to be an obstacle to women’s equality. The team plans to look elsewhere for solutions.

    *** scratches head ***

    “Solutions” to *what*?

    1. “Solutions” to *what*?

      Zero public service workers means absolutely zero bias in every category of bias, implicit or explicit. Fire everyone equally. Problem solved.

    2. Solutions to showing the bias they like.

    3. A solution to the problem that treating women just the same as men results in the unfortunate treatment of women as if they were just the same as men. Women can’t be expected to be treated the same as men if you’re just going to treat them the same as men. Equal treatment requires unequal treatment.

      1. Now I understand.

  5. I admit that I wanted this finding to be correct because it reinforced my preconceived notion that elite liberal institutions are all breeding grounds for hatred and prejudice

  6. this was 2000, after all, the stone age in our understanding of statistical errors.

    Recent unearthed footage of statistical analysis circa 2001.

    Seriously, how generously misinformed and arrogantly warped; “Sure, the Romans had population statistics a couple of millenia ago but it wasn’t until *after* these halfwits did their biased research in 2000 that we really started to understand statistics and statistical error.”

    1. Do you know who gelman is? He was around in 2000. He knows quite well how things have changed since then, and has had a hand in it.

      1. So, prior to Gelman was the stone age and after Gelman is the modern or enlightened age of ‘good’ ‘statistics-based’ social science? Kinda makes it sound like the Romans should’ve nailed Gelman to the cross circa 2005.

        Call me skeptical. Gelman can and probably has brought more statistical rigor to the field but statistical rigor isn’t exactly what the field was lacking.

      2. Personally, I read his reply as him politely saying the work was so terrible that they shouldn’t have jobs without directly recommending they be fired.

      3. Math hasn’t changed. If social “scientists” were misusing statistics in 2000 – or 1960 – it wasn’t that the proper statistical approaches weren’t known, but that those individuals were incompetent.

    2. Gelman has done outstanding work in bringing some real, defensible statistical work to the social sciences. You’d think they would’ve already had it before the 21st century, but you’d be so goddamned wrong, unfortunately. He is not kidding around here; those people had no idea what they were doing when it came to making statistical conclusions.

  7. The original study revealed the bias. Free people learned about themselves, and they changed. I think the new studies that do not find bias show that the original study helped civilization become more free. If hiring managers are reverse- biased to women, that is not good for productivity. Companies should always hire the best worker, not an anti- bias poster child. It seems that blinding can reduce reverse bias as well as original bias.

    1. The new study did find bias. But it’s the good bias, so it has to be affirmed and blinding rejected.

    2. I think the new studies that do not find bias show that the original study helped civilization become more free.

      This is incorrect bordering on nonsense. A biased society is not an unfree society and a society dedicated to eliminating bias can, quite conceivably, be oppressive as hell. There’s loads of bias against President Trump would you say he’s been brutally oppressed, largely unfree to live his own life?

      It seems that blinding can reduce reverse bias as well as original bias.

      Assuming the bias meaningfully exists and isn’t more efficiently corrected through other means. What’s the correct gender breakdown of any given orchestra? Was the bias pushing the gender breakdown out of balance or back towards it? Pretty fundamental operating questions that not only does the study not answer, but in all likelihood can’t. If the answer to, ‘Does gender even matter at all (orchestrally)?’ is ‘No’, then all the gender studies and biasing one way or the other is pointless.

      An obvious and erroneous assumption in implicit bias is that humans, once subdivided *even without their awareness* are incapable of cooperation or natural self-correction. The idea that an unconscious bias is being systematically enforced is oxymoronic. It’s like the hokum behind subliminal text messaging.

  8. Anonymized hiring was not only time-consuming and costly, it proved to be an obstacle to women’s equality.

    Au contraire! It proved to be an obstacle about to lying about women’s equality.

    I don’t know about the original orchestral audition study, whether it used shoddy statistics or not, but these two are the same in general effect. The Australian study, if anything, did as reported: it shows spontaneous affirmative action bias without blinding, which presumably means unbiased results when blinded. The orchestral study also purports to have shown implicit bias when unblinded, and unbiased results when blinded.

    In other words, both show less implicit bias when blinded. It just sucks for proggies that they show different signs fro the implicit bias, and the ultimate conclusion is that proggies love implicit bias in support of their goals and hate it when opposed to their goals.

    1. Implicit bias is a fad, garbage, cargo-cult science. It’s a phenomenon that exists but, like lots of other parts of psychology, it has its roots in non-psychological areas and, as such, cannot be meaningfully manipulated or reduced to anything resembling policy or practice.

      1. But how else are you going to explain unequal outcomes? There must be some sort of bias since it simply cannot be the case that some group of people with certain physical attributes are inherently different than some other group of people with different physical attributes. If you can’t easily identify the bias, then it must be the sort of bias that you can’t easily identify. That’s the true test of faith, accepting proof that you cannot actually prove is a proof. You obviously just don’t believe hard enough.

        1. Not every one clapped their hands?

      2. Implicit bias is the keystone of institutional racism. Without it, the whole edifice collapses.

      3. Whether it’s called bias, implicit bias, bigotry, racism, sexism, ageism, or whatever, people do have biases. Both of these studies seem to have demonstrated its existence. How valid is either study? I don’t know, I don’t have the statistical expertise to judge.

        What I find fascinating is that the first study is taken as proof by the SJW crowd that they need to control everybody to remove implicit bias, while the second study, showing the same result, is taken as proof that they must control everybody to add more explicit bias to cover up the implicit bias.

        1. people do have biases.

          Who has how much of which kind and how can you tell?

  9. As a white boy I feel that all my achievements were truly earned what with everyone’s concern about booting our necks. I only got to second chair, but I went to prom with the concertmaster.

  10. “You shouldn’t be running around making a big deal about point estimates when the standard errors are so large,” he wrote. “I don’t hold it against the authors—this was 2000, after all, the stone age in our understanding of statistical errors. But from a modern perspective we can see the problem.”

    I hope this is sarcasm.

    1. No, it was him giving the authors the means to save face. His career would have been over if he had said: “You see what happens when women do math.”

      1. Hmmm. Maybe it was both 🙂

    2. Gelman has done a lot of work on bringing defensible statistical analysis to the social sciences. There’s a variety of particular problems with the statistics generated in social science disciplines that require careful handling, and until recently they were mostly ignorant of this fact. I do think he’s being a little more charitable than necessary here, but it doesn’t take much at all to start a bitter vendetta in academia so it’s probably in his best interests to allow them to remain in as good a light as possible.

  11. “I don’t hold it against the authors—this was 2000, after all, the stone age in our understanding of statistical errors.

    I’m so tired of people treating the recent past as ancient history. Now we’re pretending we didn’t understand statistics two decades ago? What crap.

    Anonymized hiring was not only time-consuming and costly, it proved to be an obstacle to women’s equality. The team plans to look elsewhere for solutions.

    Not true. What this shows is that women’s advocates aren’t interested in equality. Further even when that is proven they will continue their demands for special treatment.

    1. I am glad someone else noticed this. Statistics was not different in the year 2000.

      Perhaps in the fields of gender studies the application of statistics was less rigorous back then. But statistics has not changed, particularly not on something as simple as this study.

  12. i don’t get the orchestra bias who dafuq cares if the viola chick is a chick?

        1. yeah she’s a funny person too.

    1. I say hot viola chick gets priority.

      1. fine by me but if i’m running an orchestra the viola ability has to be in range to be covered by the hotness she can’t just be terrible but hot

        1. The violist doesn’t need to be hot, the whole POINT of a viola is that it burns longer (thereby producing more heat) than a violin.

  13. So you’re claiming that there’s no evidence that orchestras discriminate against women? Wow. Just wow. Yet another example of how cishet white men use math to marginalize women. I’m literally shaking.

  14. Nobody noticed that the results, had they been accurate, could have also shown that women play better when they can’t see the audience, or when they know people can’t see them?

    1. Quite astute. An equally plausible explanation, as presumably the study was ‘blinded’ in more than one direction. Or that the men play better when they see an audience directly viewing them.

  15. I sincerely doubt that there is an anti-female bias in the music industry.

    Having been through my fair share of auditions, blind and non blind, I can tell you that the industry is Rife with favoritism. Anytime you are judging the quality of something as subjective as musical ability, ancillary biases are going to become an important. Prior associations, common mentors, even simple familiarity are going to weigh heavily when you are trying to judge which of two violinists of similar ability sounds the best.

    I would be willing to bet that in an unblinded audition the hot young female is going to get the job over the old fat guy, all other things being equal. That is just the way the world works. And the fit 35 year old well-groomed young man is going to do better then the frumpy, fat, sour faced 57 year old woman.

    If you don’t believe me, just look at the musicians on the Grammy Awards. You don’t see a lot of fat old guys singing pop music. You don’t see very many out of shape and not very attractive young people either. This, even though you listen to music with your ears and what the person looks like shouldn’t make a bit of difference. That’s the theory. Reality is completely different. Somehow Beyonce being a generational beauty makes her mediocre pop music sound much better.

    1. >>You don’t see a lot of fat old guys singing pop music.

      MTV ruined Christopher Cross.

    2. This, even though you listen to music with your ears and what the person looks like shouldn’t make a bit of difference.

      But music industry executives use other body parts to probe new talent. That’s why it makes a difference.

    3. I sincerely doubt that there is an anti-female bias in the music industry.

      This is the SJW wheelhouse, the issue isn’t something as broad as the music industry and it’s not anti-female. It’s orchestra and only in very Simpson’s Paradox-esque ways. The flute is dominated by women and harp is near exclusively female. But, musically-speaking, you don’t need a big flute or harp section the way you need a string or brass section and, even when you break those sections down, it’s similar. There’s a small gap orchestra-wide, something like 55:45 M:F, and if you start looking at conductors, composers, and pieces performed, you can inflate things to an anti-female bias but it gets to be a bit like saying that there’s an anti-male bias in pre-school children’s educcation. There sure is a bias and it’s that men wouldn’t do it if you paid them.

      1. Well, most men aren’t willing to teach preschool for as little money as many women are, hence the divergence. I know that I certainly wouldn’t do the job for anything like what it usually pays.

    4. Whole underlying theme of Dreamgirls.

  16. When I’m making stew or soup, I cut celery on the bias.

    1. I bet you use rapeseed oil, too.

      Pervert.

  17. “”and had great difficulty trying to locate the 50 percent statistic within the modest findings.””

    Perhaps it would be easier to locate if he would just consider the beauty of math

    1. They probably can’t look for it in the same place the researchers found it – they don’t know each other that well.

      If you know what I mean and I think you do.

    2. Are we mocking mtrueman from that other article? Because I’m down for that.

  18. “”You shouldn’t be running around making a big deal about point estimates when the standard errors are so large,” he wrote. “I don’t hold it against the authors—this was 2000, after all, the stone age in our understanding of statistical errors. But from a modern perspective we can see the problem.””

    If only people applied this logic to climate science as well. The error terms aren’t large, but guess what? They fall within the margins, ie the observed “effects” are indistinguishable from mathematical noise.

  19. “I don’t hold it against the authors—this was 2000, after all, the stone age in our understanding of statistical errors.

    Then what was it in 1968 when I studied statistics? Pre-history?

    Of course, my statistics class was in the engineering department, not made-up-bigotry-studies department, so maybe there is a difference.

  20. Sounds like the Australia study proved the original study. The blind applications revealed that the managers held implicit bias towards women which vanished when they didn’t know the gender of the applicant.

    All I’m seeing is sour grapes that the implicit bias wasn’t in the direction they suspected.

  21. What rot this whole thing is.
    Have these statisticians never heard of the great equalizer; disparate impact?
    That’s where the SJWs get to look at the sex, race, ethnicity, nationality source of surnames, and just about anything else they decide could be the basis for a bias, and decide from the results, after the fact, that there has been discrimination, even if there was never a chance for the selection process to be able to use any kind of subjective criteria. “Blind” and “anonymous” are meaningless terms to these people.
    It is actually written into US government regulations and has been accepted by courts as proof of discrimination.
    Equality of opportunity has been replaced by equality of outcome and is entrenched in the only form of institutional racism/sexism in the US – affirmative action.