So Who's Counting?
Lies, damn lies, and statistics.
Labyrinths of Prosperity: Economic Follies, Democratic Remedies, by Reuven Brenner, Ann Arbor: The University of Michigan Press, 290 pages, $45.00/$17.95 paper
Tainted Truth: The Manipulation of Fact in America, by Cynthia Crossen, New York: Simon & Schuster, 272 pages, $23.00
The Tyranny of Numbers: Mismeasurement and Misrule, by Nicholas Eberstadt, Washington, D.C.: The AEI Press, 305 pages, $24.95
A Mathematician Reads the Newspaper, by John Allen Paulos, New York: Basic Books, 212 pages, $18.00
In the beginning was the Word, but the end will undoubtedly be announced with a number. A solemn poll: Sixty-eight percent disapprove of annihilation; 24 percent support it, with 8 percent undecided. Margin of error, plus or minus 3 percent.
Any consumer of news is surfeited with numbers; no statement can be made, no conclusion reached safely in public discourse nowadays without the cushion of often incomprehensible or irrelevant numbers. The presidential campaign is handicapped with almost daily poll results, even though the readers of the poll are often ignorant about how the relevant questions were asked. (The order and specific wording of the questions can affect the results.) The fate of our economy is forecast not in entrails but in numerals, huge macroaggregates that are actually measuring–what?
The sort of numbers we encounter in newspapers every day can be troublesome on many different levels. Among those problems:
? Practical problems of aggregation. Is it a count or a survey? Do you have reason to trust the sampling techniques if it is a survey? Do you have reason to trust the raw data of especially large counts? An anecdote from novelist Anthony Burgess's memoirs, Little Wilson and Big God, casts light on a problem with data collection that is often ignored, possibly because its ramifications are so unpleasant.
Burgess writes about having to send numerical records back to the home office in London about classes he supposedly taught fellow soldiers while stationed in Gibraltar during World War II. He fabricated them from thin air. Those "statistics were sent to the War Office. These, presumably, got into official records which nobody read." Surely we all have enough anecdotal awareness of similar lapses to guess that these sort of shenanigans are common. The basic counts that underlie the huge numbers we deal with ought not necessarily be trusted. We don't necessarily know everything that a cold number on a page can make us think we know.
? Theoretical problems of the meaning of aggregation. Economic macroaggregates dominate newspaper reporting–trade deficits, GDP, and the like. But what are these numbers good for? The bigger the aggregation, the more places error can creep in; and economic aggregates are often not even direct counts of anything meaningful. Do they tell us anything worth knowing? Overemphasis on such things as trade deficits (or racial population breakdowns) clouds discussion of whether such concepts are worth anyone's concern. Cobbling together macroaggregates and behaving as if they in some way interact to create real-world economic effects is bound to confuse. Knowing those sorts of numbers is not necessarily knowing anything useful about understanding the workings of a national economy. When dealing with areas where causality is as inherently complicated as economics, the wisdom of Austrian economist Ludwig von Mises is apposite: The kind of numerical empiricism implicit in macroaggregates doesn't always lead to useful knowledge of anything but history.
? The intentions of those spreading the numbers. Numbers don't come to the attention of the populace in some ideal Pythagorean way–beamed into our brains, untouched by grubby material considerations. They arrive via an information economy, the retail end of which is usually daily newspapers and TV news shows–media with particular qualities and predilections that make them easy conduits for shoddy or confusing numerical displays.
These four books, all concerned to one degree or another with quality control in the information economy, touch on permutations of all the above difficulties. They all to some degree scrape beneath the varnish to expose the sometimes unfinished and shoddy nature of the scaffolding on which so much public argument stands.
The information economy has a proclivity to spin the audience's mind with so many conflicting reports that neither trained nor untrained minds can be certain of what to believe. In her book Tainted Truth, Wall Street Journal reporter Cynthia Crossen walks her readers step by tortuous step through some current such conflicts related to health: the continuing sagas of caffeine and oat bran, the public scares over Alar and silicone implants. The lesson seems to be that little of the expensive studies and number manipulations of modern medical science have improved much on what the ancient Greeks could have told us: Try to consume a moderate diet containing small amounts of a wide variety of things. Some indulgences of science in our culture seem like God's way of telling us we have too much money: enormous amounts of cash and time to reach results of dubious value.
Crossen concentrates on the intentions of those spreading numbers. She is rightfully skeptical of what corporations say to sell their products or protect themselves from liability. Sometimes, though, that skepticism seems more reflexive than thought. She gives more credence to arguments about the hazards of silicone breast implants than the best overview of the literature indicates is proper. (See "A Confederacy of Boobs," October.)
Refreshingly, she is also skeptical of the professional public crusader, and puts the interest groups behind the Alar scare in their place. This is fitting, because many of the big, scary, and worthless numbers circulating in the public information economy come from such do-gooders, and often go unquestioned because of their seemingly blameless sources.
Examples include the oft-repeated number from cancer interest groups that American women face a one-in-eight chance of contracting breast cancer, a figure that applies only to women who have already managed to live to be 95. That failure of intention also rubs up against a failure of aggregation: Those figures are based on the assumption that we even know for certain how many people have breast cancer, which we don't; any data collection on cancer rates could easily be missing something. As two journalists who write about cancer admitted in The Nation, "Not only is there no central national agency to report cancer cases to …but there is no uniform way that cases are reported, no one specialist responsible for reporting the case." Another example is inflated figures on the number of homeless in America, spread by activist Mitch Snyder and repeated breathlessly throughout the '80s. Those figures, because of their sensitive political connotation, have been thoroughly debunked, usually by journalists with an ideological interest in doing so.
Crossen's tone is sometimes too somber and self-important. As Mencken put it, a good horselaugh is often worth a hundred syllogisms. Much of the number nonsense Crossen writes about deserves derision, not moping–for example, the numerical fakeries of cola taste tests were, believe it or not, once the subject of government fraud investigators in Massachusetts.
She is on target with some of her explanations of possible polling biases that make the many, many polls that fill newspapers of questionable value. A good example: If you ask someone a question that is calculated to raise negative feelings about the current national situation, you can apparently influence how they'll say they feel about the incumbent in the next question.
This could imply two things: Either people make their voting decisions based on the last thing that entered their head before they entered the polling booth, in which case the voting public is so stupid that who cares about polling bias–or that polls are completely artificial ways to get information about people's actual voting behavior, in which case who cares about polls?
Nicholas Eberstadt, a demographer and expert on the failures of communism, is even more portentous than Crossen in his The Tyranny of Numbers . Eberstadt concentrates on failures of aggregation and failures of analysis; often the problem with the way people use numbers to reach conclusions is that they aren't using the right numbers. In his introduction, Eberstadt thunders: "[M]odern man lives under a tyranny of numbers…ordinary people around the world routinely suffer injury through the agency of…dull statistics. On more than a few occasions, these injuries have been grave and irreversible and have afflicted large numbers of persons." An example: misleading data about world hunger that prompt further state control of food distribution, with ill effects.
But Eberstadt is something of a number tyrant himself. His profession as demographer requires him to pay obeisance to macroaggregates. Sometimes he is sensitive to the problems of data gathering, and warns the readers to be wary of certain numbers (even as he uses them to make points) because of doubts about how they were gathered. He points out that communist and Third World nations have often lacked the capability or will to accumulate honest data about the states of their nations.
But he rarely snorts at numbers, and the non-technical reader is given no help in judging how much salt to chew with the data that Eberstadt does present as gospel. Am I to take it on faith that the U.S. Census Bureau can calculate with trustworthy accuracy the life expectancy of the Chinese people from the 1950s to the present? Or that the U.S. Department of Agriculture is quite certain about trends in cigarette consumption in Eastern Europe? It's not impossible, I suppose, but doesn't anyone wonder about how such allegedly reliable figures are gathered?
But you know what? I wondered, but I didn't check. This is a deeper hazard of the cult of numbers. Whether numbers are the declarations of interest groups, the results of science papers, or the macroaggregates of governments, hardly anyone checks them. Burgess's anecdote chips away at the sturdy reliance the Eberstadts of the world place on huge data collections, as do more recent media reports. Celia Farber has reported in Spin about how numbers on African AIDS are often manufactured from whole cloth. Michael Maren has done the same in the pages of Forbes MediaCritic about deaths from the war and famine in Somalia. Crossen quotes a figure bruited about by Bill Clinton on the number of lobbyists in Washington that its ostensible source, a professor, admits to deriving "off the top of my head."
But beyond data collection problems lie deeper, conceptual problems: Even if the numbers could be trusted, so what? One of the central points of Eberstadt's early chapters is that counts of poverty in the United States count the wrong thing. They overemphasize income when they ought to look at consumption, as well as such secondary measures as health, nutrition, and infant mortality.
Reuven Brenner's curious book gives an iconoclastic answer to the "so what?" question: It makes no difference how "good" the numbers are when discussing economics and national prosperity; they're still best ignored. As Brenner wends his way through his often difficult book (dedicated to explaining his vision of the key to the wealth of nations), he keeps coming back to the point that economic macroaggregates–bothering to collect them or using them to make decisions–are always in error. He emphasizes the data collection errors inherent in key macroeconomic variables such as the consumer price index and gross national product. "Price indices give a distorted view of changes in price levels when many new products come on the market, or when there are yearly, drastic changes in the performance of computers, VCRs, and other communication and home entertainment equipment, as well as in people's expenditures on them." Besides, for most of us it is relative individual prices that are important in our decision-making, not some phantom national price level.
Brenner quotes a Bureau of Labor Statistics official admitting that estimating productivity "is just about impossible." He points out that figures on savings and investment run into inherent objectivity problems, since certain purchases some might see as consumption–say, a home computer or a house–could be made as investments, and thus perhaps ought to count as savings. There is an inherent subjectivity at the heart of all economic decisions that makes objective macroaggregations in economics difficult if not impossible. Thus, Brenner thinks that even if the technical problems he discusses in GNP, productivity figures, and price indices could be solved with better and more elaborate surveying techniques, who cares?
"There cannot be," Brenner writes, "any such thing as a 'general theory' about any of the following questions: How does the general level of a government's expenditures, in particular its debts and deficits, affect either the total production of goods and services, or the national income earned from production? How do the debt and deficit affect employment? How do the debt and deficits affect the allocation of resources between current consumption and investment? The reason that no general answers can be given to any of these questions is simple….It all depends on what the government does with the money and on the timing of its expenditures." Obsessed with the big number, Brenner argues, we ignore the fact that any given element of these huge aggregations is individual, and can't be assumed to have the same effect as any other element in the aggregation. So talk of the aggregation's effects is meaningless.
All of the error and confusion Crossen, Eberstadt, and Brenner discuss come down to us at the retail end of the information economy in daily newspaper or TV news report, so the buck, in a sense, stops there. Various researchers do occasionally make hay out of tracing faulty numbers back to their sources, as Christina Sommers did with the claims of feminist interest groups in her book Who Stole Feminism? But that kind of story is hard to write, and it's naive to make suggestions about changing the practice of journalism, as Crossen does in the recommendations at the end of her book, that depend on effort and scrupulousness that can't always be expected.
Sure, reporters pride themselves on a withering skepticism. "If your mother tells you she loves you, check it out," snarls the stereotypical grizzled city editor. But sociologist Richard Gelles, quoted in Newsweek, is more in line with the everyday practice of journalism: "Reporters don't ask, 'How do you know it?' They're on deadline. They just want the figures so they can go back to their word processors." Whenever I read a newspaper article about something I have firsthand knowledge of, I inevitably find at least one mistake. Conversations with others tell me this is almost universal.
John Allen Paulos's book is about the retail end of the information economy, and his stance is a gentle antidote to the abrasive despair that contemplation of Crossen and Eberstadt might lead to. Though he touches on many of the same topics as Brenner and Crossen, he is far more bemused, less appalled. His tone is more urbane, more charming than their sometimes strident alarmism. Paulos, a mathematician at Temple University, takes his reader on a friendly walk through the various elements of a typical daily newspaper, and applies the reasoning skills and outlook of the mathematician to them.
This often leads him to inculcate skepticism about the numbers you read in the paper, but with the quiet recognition that we don't know where all these numbers are coming from, many of them seem to conflict, and anyway "ambiguity, randomness, and lack of information in response to obsessive questions and concerns can…breed delusions and mirages." That's the root of the cult of numbers: We crave certainty we can't have, and numbers, however gathered, give us the illusion we crave. "Whether the issue is trade, the environment, or health care, politicians who abjure unwarranted expressions of certitude deserve plaudits, not pillory."
Because of Paulos's quiet good sense and wide-ranging mind, his book is far more of a pleasure to read than the others under discussion. It is simply nifty, larded with clever and informative tidbits as he strolls his broad, discursive way through typical newspaper reporting.
Some examples: Paulos explains the paradox of how even a diagnostic test that is 99 percent accurate for a disease, if that disease is rare, will give far more false positives than accurate ones. And while calming fears about minute contaminants that sound huge on the molecular level, he writes: "[A]ny mathematically expressed scientific fact can be transformed into a consumer caveat…that will terrify….Warning: This product attracts every other item in the universe with a force equal to the product of their masses divided by the square of the distance between them."
There is no one solution to the complications that numbers can add to our understanding of the world. One unfashionable bit of advice is to say a good word for reason (and moderation) over reflexive empiricism. We need more empiricism of the "let's see what's happening" vein and less of the "we've got to count it" vein–especially when attempting to sift out accurately the causes of inherently complicated, multicausal events like disease, poverty, and the like. Obviously, in some cases, especially epidemiology, some attempt at counting is an inherent part of finding out what's happening. But we shouldn't assume, especially when the counts are huge, that the data we get from them is unfailingly trustworthy.
Even when numbers do have valuable things to tell us, there are many slips between even rigorous work of science and what gets out to the public. Ignorance and carelessness on the parts of both the messenger (the disseminators of the data and the journalists who report them) and those receiving the message (we out here in the reading public) make numbers tricky things indeed.
The proper response to Big Numbers is sometimes just a horselaugh, or at the very least suspended judgment. The one thing we all must economize on is time; few of us can explore, probingly, "How do they know that?" and pursue the question ruthlessly. The information economy is curious: The demand for knowledge outstrips the reliable supply. The economy of knowledge sometimes produces counterfeits as a result.
A chapter in Eberstadt's book is an excellent summary example of how the cult of numbers can lead to trouble. Eberstadt is writing about the CIA's studies of the Soviet Union, and calls them "the largest single project in social science research ever undertaken." He then goes on to expose the largest social science project ever undertaken as fatally flawed in almost every aspect.
The CIA took Soviet data at face value, because "that assumption facilitates the use of the complex econometric models they have devised," leading analysts to "estimates of spurious precision and questionable accuracy." (They trusted flawed data because their method required numbers, any numbers.) They never publicly explained their methods and procedures. (As newspaper numbers are rarely presented with enough information for the skeptical reader to judge how trustworthy they are.)
The CIA was in a position where it had to come up with knowledge that its analysts really had no way of knowing; unable to just admit ignorance, they attempted to fake it. The cult of numbers bound them to a bad method and blinded them to what simple reasoning from fast-and-loose empiricism might have taught them. Their exaggerated reports of the Soviet economy's productivity flew in the face of the impressions of poverty and squalor of those Westerners who were presumably seeing the best face the Soviets had to offer; but that sort of non-numerical information had no place.
Too much energy and resources are expended today in pursuit of a precision that is just beyond us. Not all of it is as potentially damaging as the CIA's exaggeration of Russia's productive prowess, but all of it clutters the world with fake knowledge that is often not necessary.
I once had occasion to read, in rough draft, a piece of political writing by a friend. He knew the trends he was discussing very well, but hadn't yet had time to look up precise numbers. So the draft merely featured bolded xs in place of all the numbers: blank signposts of irrelevant larding to come.
The lack of precise numbers affected my comprehension of his argument not a whit; the real numbers, if they had been there, would have stuck in my mind no more firmly than those bolded xs. Whose eyes do not glaze over at the array of survey results, polls, and macroaggregations that are necessarily marshalled to buttress every argument made nowadays, like the banner of the Lord marshalled behind warriors in olden days?
The misuse of numbers in our culture is not a crisis. But it is an annoyance to those of us who read newspapers every day along with the charming Mr. Paulos. We can combat that annoyance with his remedies, plus a dash of Ludwig von Mises: Good spirits, a solid grounding in some basic mathematical and statistical principle, and a realization that there are some things that can't be counted anyway, and so what if they could?