A "reality check" for the Regnerus study on gay parenting [UPDATED]

Three years ago, against the strong consensus of social scientists and professional child-welfare groups, University of Texas sociologist Mark Regnerus concluded that children of gay parents fare worse than children raised by married opposite-sex parents. In the face of intense criticism and a scorching assessment from a federal judge ("not worthy of serious consideration"), Regnerus doubled down on his conclusions and filed an amicus brief against gay marriage in federal court.

But a new critique of Regnerus' work by Professors Simon Cheng (University of Connecticut) and Brian Powell (Indiana University), published in the same journal as his original study, Social Science Research (available free to most academics and for a $35.95 fee to the general public), suggests that Regnerus misclassified a significant number of children as being raised in same-sex households. Based on a re-evaluation of the data, it concludes there are minimal differences in outcome for children raised by same-sex parents and married opposite-sex parents. From the introduction to the paper:

Research communities in the social sciences have long been aware that methodological decisions can potentially affect the inferences of survey research (Firebaugh 2008). This threat to the validity of research inferences is particularly challenging for studies that focus on a very small group of interest, such as some racial minory groups, atypical families, and same-sex couples (Cheng and Powell, 2005 and Cheng and Powell, 2011). In such research, even a tiny percentage of measurement errors for the small subsamples could powerfully distort patterns from the surveys, and other methodological choices can similarly affect empirical results. When research findings from these analyses are used as policy guidelines, the threat goes even beyond scientific communities. It therefore is incumbent for scholars to critically assess the implications of these decisions in their own work as well as that of others.

. . . In revisiting the Regnerus article and reanalyzing the NFSS [New Family Structures Study], we ask one fundamental question: To what extent are the patterns reported by Regnerus attributable to the conceptualization and operationalization of family types-in particular, gay/lesbian/bisexual families-and other analytical decisions?

Our empirical reexamination of Regnerus's analysis is designed to answer this question. More broadly, it underscores the importance of, in the words of Firebaugh (2008), "build[ing] reality checks into your research" (p. 64)-in particular, "internal reality checks" (p. 65), checks on "dubious values and incomplete data" (p. 65), and checks on "consistency in conceptualization and measurement" (p. 69)-and the serious implications of not attending to these concerns (Bearman and Parigi, 2004, Cheng and Powell, 2011 and Fischer, 2009). In addition, it highlights the general challenges that social scientists continue to face in our examination of same-sex parent households and other emerging family forms using nationally representative datasets.

Below, we first discuss the NFSS and Regnerus's measures of family types using the data, and then highlight the difficulties in using the NFSS to accurately distinguish between family types, using adoptive households and intact biological families as illustrations. We then discuss the challenges in accurately identifying same-sex families. We follow this discussion with a closer look at the NFSS survey and demonstrate the potential for misclassifying a non-negligible number of respondents as having been raised by parents who had a same-sex romantic relationship. Finally, we assess the cumulative implications of these possible classification errors and other methodological considerations from various stages of the research process by reanalyzing the NFSS in seven steps.

These reanalyses provide a "reality check" regarding the conclusions from the original Regnerus study. The patterns from these reanalyses offer evidence of the fragility of these conclusions-so fragile, in fact, that they are due primarily to the methodological choices made by Regnerus. Or to put it another way, when equally plausible and, in our view, preferred methodological decisions are used, a different conclusion emerges: adult children who lived with same-sex parents show comparable outcome profiles to those from other family types, including intact biological families. That this revised conclusion is consistent with those reported in most previous studies and inconsistent with Regnerus's findings illustrates how the accumulation of research decisions throughout the research endeavor-and, in particular, measurement decisions that overlook inconsistent information within the data-may lead to questionable conclusions, even with a population-based large sample.

The basis for these conclusions is more fully explained in the paper, of course, which interested readers can obtain here. Fundamental to the critique is the basic idea that Regnerus misclassified children as having been raised in same-sex households when, in fact, under a fairer assessment they had not been.

Regnerus does not check for, or apparently even consider the possibility of, inconsistent, uncertain, and unreliable cases in his data-even though some other items in the NFSS offer some limited means to assess this possibility. For example, Regnerus (2012c) acknowledges that, according to the aforementioned calendar data, over half of the respondents never lived with a parent's same-sex partner, but fails to mention that many respondents-approximately one-third-also never lived with their same-sex parents or lived with them very briefly. [emphasis original]

This likely means that a sizable number of the 236 respondents counted as "raised by" lesbian mothers and gay fathers had not, in fact, been raised by same-sex parents. For example, nine of the responses likely came from jokesters:

The most blatant example of highly suspicious responses is the case of a 25 year-old man who reports that his father had a romantic relationship with another man, but also reports that he (the respondent) was 7-feet 8-inches tall, weighed 88 pounds, was married 8 times and had 8 children. Other examples include a respondent who claims to have been arrested at age 1 and another who spent an implausibly short amount of time (less than 10 minutes) to complete the survey.

Another 53 of the 236 responses came from children who lived with gay mothers or fathers for less than a year. Another 20 came from respondents whose entries were otherwise inconsistent or improbable. Others had lived with gay parents for 2-4 years.

By the time you back these contested responses out of the analysis, here's what you're left with: "Of the 236 respondents identified by Regnerus (2012a) as living in a LM [lesbian mother] or GF [gay father] household, we identify only 51 that can plausibly be coded as being raised for at least a year in a same-sex couple household." [emphasis original]

How did these remaining 51 respondents compare to the children in the NFSS raised by their biological mothers and fathers?

Here we find only four significant differences, although the differences either are not indicative of any LM/GF disadvantage (i.e., sexual self-identification and having a same-sex romantic relationship) or do not gauge adult experiences (i.e., receiving public assistance in childhood and sense of safety and security while growing up). These patterns also are highly fragile and based in part on a couple of influential cases or outliers. Admittedly, even with a large overall sample, a subsample of 51 cases still limits the statistical power of the analysis. Still, the results are either inconclusive or suggestive that adult children raised by same-sex two-parent families show a comparable adult profile to their peers raised by two-biological-parent families.

The analysis is complex, nuanced, and qualified-as any such study should be-and it seriously undermines Regnerus' conclusions. Again, you can read the whole paper here.