Cracked Bell

James J. Heckman from the March 1995 issue

(Page 3 of 4)

This sort of empirical exercise prospers--or founders--on the details. The credibility of any empirical study depends on the care taken by the analyst in defining and measuring concepts, and in interpreting conclusions drawn from the data. It is at this point that the book becomes a policy polemic rather than a scholarly study of human differences.

First, consider the definitions of the two key variables used in the authors' empirical study: IQ and family background. In their empirical analysis, the operational definition of IQ is the Armed Forces Qualifying Test (AFQT) score, which is based on an aggregation of a subset of 10 separate exams given to more than 12,000 youth in 1980. The youth were 15 to 23 years old when the tests were administered. These tests were designed to predict success in military training schools, and there is much evidence that they do so, although by no means are they 100-percent reliable. Most of these tests appear to be achievement tests rather than ability tests (i.e., they partly measure factual knowledge and not pure ability). A subset of four tests was assembled to define the Armed Forces Qualifying Test (AFQT) score used by Murray and Herrnstein.

Ironically, the authors delete from their composite AFQT score a timed test of numerical operations because it is not highly correlated with the other tests. Yet it is well known that in the data they use, this subtest is the single best predictor of earnings of all the AFQT test components. The fact that many of the subtests are only weakly correlated with each other, and that the best predictor of earnings is only weakly correlated with their "g-loaded" score, only heightens doubts that a single-ability model is a satisfactory description of human intelligence. It also drives home the point that the "g-loading" so strongly emphasized by Murray and Herrnstein measures only agreement among tests--not predictive power for socioeconomic outcomes. By the same token, one could also argue that the authors have biased their empirical analysis against the conclusions they obtain by disregarding the test with the greatest predictive power.

More disturbing is the authors' treatment of family background. The index is based on parental education and occupational status, and on family income measured at one point in the entire life cycle of the child. For many young adults, the family-income measure is entirely missing and is omitted from the construction of the index.

The IQ measure used by Murray and Herrnstein is taken rather late in the life cycle of the child. (Recall that the IQ test is administered to youth 15 to 23 years old.) Many analysts suspect that IQ as measured by tests administered after early childhood reflects the outcome of social and cultural influences. The authors attempt to eliminate these influences by a standard statistical method called regression analysis.

But the standard statistical methods used by Murray and Herrnstein are vulnerable to measurement error. It would be incredible if 15 to 23 years of environmental influences, including the nurturing of parents, the resources they spent on a child, their cultural environment, their interaction with their children, and the influence of the larger community could be summarized by a single measure of education, occupation, and family income in one year. If environment is poorly measured but affects the test score--and there is solid evidence of environmental impacts on test scores--then Murray and Herrnstein's finding that IQ has a stronger impact on socioeconomic outcomes than the measured environment may simply arise from the poor quality of the measure of the environment.

The authors present evidence that IQ rises with age and with years of schooling completed. IQ may actually be a better measure of the environment facing children than the measure of environment used by Murray and Herrnstein. They use IQ to predict schooling, but schooling produces IQ. Hence, they are especially likely to find a strong measured effect of "IQ" on schooling.

The same remarks apply to their study of racial and ethnic differentials in socioeconomic outcomes. If racial differentials in environments affect ability and influence measured test scores, evidence that racial differentials weaken when ability is controlled for using regression methods does not rule out an important role for the environment in explaining performance in society. In the presence of measurement error in the environment, the authors' analysis will overstate the "true" effect of ability on those outcomes.

There are methods for addressing these problems, but Murray and Herrnstein do not use them. They should have tried a variety of measures of family background to explore the sensitivity of their reported results to the particular measures of family background they do use. A strict environmentalist could justifiably argue that the evidence reported by Murray and Herrnstein simply reveals the crudity of their measure of the environment and the strength of the correlation between the test score and their measure of environment.

One important technical point worth making here concerns the method used by the authors to measure standardized changes in IQ and family background. By its very construction as a measure that follows a bell curve, the "two-standard deviation" range in measured IQ used by the authors to gauge the sensitivity of outcomes to IQ represents a change in IQ ranging over 95 percent of the population. A "two-standard deviation" range of their family background index does not include 95 percent of the population, because that measure does not come from a bell curve. It may include as little as 75 percent of the population. By restricting the range of the environmental variable they understate the role of the environment in affecting outcomes relative to the role allocated to IQ.

Finally, the book fails due to a lack of coherence. The argument does not cumulate in a convincing way. Too many seams are visible. Its case against affirmative action and egalitarianism in education, and in favor of the use of testing in the workplace, does not require acceptance of a single ("g-loaded") scale of ability, or acceptance of the importance of heredity in producing socioeconomic inequality. The authors' evidence of growing stratification by cognitive ability in schools and workplaces does not require that they take a position on how the ability is produced. Yet the authors argue strenuously for their narrow view of ability and its heritability and thereby distract the reader's attention.

Nor do the two competing visions of the future of American society offered up in Part IV naturally flow from the arguments and evidence presented in the earlier parts of the book. The first dystopic vision relies on stronger sorting and heritability mechanisms than the authors have demonstrated actually operate in American society. Even if IQ is largely inherited, there is considerable scope for intergenerational economic mobility. The extreme pessimism of this scenario ignores the warnings issued by the authors that even among persons in the lowest ability grouping, there is still a lot of socially productive behavior. Their pessimistic vision relies on unsupported assumptions about the skill bias of future technological change and the inability of entrepreneurs--and social institutions--to efficiently utilize unskilled labor. This vision might be realized, but it reads more like a story borrowed from science fiction novels than a plausible extrapolation of existing social trends.

The second vision of communitarian harmony presented in the final chapter of the book is based on several implicit assumptions. The authors take it as self-evident that cognitively heterogeneous groups benefit both able and less-able members. That is an empirical statement about which the authors offer no evidence. If such heterogeneous associations are not mutually beneficial, strong forces of self-interest would arise that would promote segregation and separation along cognitive lines. In that case, their communitarian society of neighborhoods would only be viable if coercion were applied from central governments.

Cognitive stratification may be both economically efficient and socially desirable. Our cognitively stratified society may be a consequence of private choice, not governmental coercion. Murray and Herrnstein never discuss mechanisms or incentives that would lead persons to voluntarily embrace cognitively integrated local neighborhoods. They take it as self-evident that such neighborhoods would spontaneously emerge if central governments quit interfering in local communities.

Many of the other policy goals advocated by the authors could be embraced without any consideration of the problems--or benefits--of cognitive stratification. For this reason, their advocacy of these goals is not germane to this book. Simplifying the law, eliminating regulation, providing clearer moral and social rules, and eliminating restrictions on entry into business may differentially benefit the cognitively weak, but they are likely to benefit everyone else, too.