Measuring Up
Testing the pretensions of market research and polling.
Finding Out: Personal Adventures in Social Research—Discovering What People Think, Say, and Do, by Leo Bogart, Chicago: Ivan R. Dee, 306 pages, $27.50
Polling, Policy, and Public Opinion: The Case Against Heeding the "Voice of the People," by Robert Weissberg, New York: Palgrave Macmillan, 232 pages, $39.95
On a warm, bright winter's day in Southern California recently, I was caught up in an "explosion of research."
That's how a skeptical Leo Bogart—recipient of prestigious awards from the American Marketing Association and the American Association for Public Opinion Research and one of the first two men (with George Gallup) inducted into the Market Research Council's Hall of Fame—describes the current market research trend of "focus groups." These are, in Bogart's apt description, "collective interviews with small groups of people unsystematically selected without reference to any randomized sampling plan."
I helped comprise a six-person focus group dedicated, it seemed, to some health insurance company's plan to reconfigure its image or selling points. (Exactly who was paying for the research and its precise aims were deliberately withheld.) Four women and two men—five white and one black—were shown three short statements of purpose for a health insurance provider. We were asked to restate what we thought the statements said and describe how they made us feel.
With no explanation or context, we all had to guess—sometimes wildly—as to what the statements were getting at and their purpose. Despite the artificiality and silliness of the situation, all of us rose to the occasion and tried to take our task seriously. (We were getting paid for our trouble, after all.) But what came out seemed banal at best.
What did we want from health insurance? We wanted security from our provider; we wanted flexibility; one of us (me) wanted to feel taken care of by our provider, as opposed to being a partner to them in our traipse through the health care system. We mostly liked the suggestion that we'd be provided with information and options. The slogan "health security" could mean anything to us, or nothing. (We were asked to react to it de novo, but we all realized that the phrase was a cipher out of context.) Our comments were not necessarily stupid but in all cases obvious, not quirky or surprising—though my fervent insistence that I didn't want information and partnership from my health insurance provider so much as I wanted them to pay my medical bills for me didn't win instant assent from my group mates.
We were asked, unsurprisingly, to quantify our approval of the statements on a 1-to-10 scale. Can't call it knowledge without numbers, right?
In addition to our modest stipend, the facilitator managing our group and the company that gathered us were being paid as well, and probably not quite so modestly. But I have a hard time imagining what good whoever paid for the research could have gotten out of it.
Leo Bogart does too. He has been on the front lines of social science research in many different battlefields—from the University of Chicago to Standard Oil's P.R. department to Fulbright scholarships studying communist propaganda in France to serving as the first market research director for Revlon to working for the TV-battered Bureau of Advertising for the American Newspaper Publishers Association. He has just issued his memoirs, Finding Out: Personal Adventures in Social Research—Discovering What People Think, Say, and Do. It's a delightful book, filled with colorful characters quickly but deftly limned. It makes that most stereotypically soul-deadening middle-class postwar American role, the Marketing Man, seem intellectually vital and fun. Bogart is mostly indulging himself in this book with reminiscences of people he's known and tasks he's performed rather than rigorously arguing a point of view or defending the values and methods of his profession.
His details are frequently funny and charming, and add humanity to a potentially dull topic. He describes Earl Newsom, his P.R. chief at Standard Oil of New Jersey, as "the opposite of the common notion of what a public relations wizard should be like. He was gaunt, humorless, and withdrawn….In meetings he spoke sparingly and in an almost inaudible croaking whisper, so that all those present had to lean forward to hear him. This gave his most banal utterances an aura of great consequence."
Bogart relates anecdotes of desperately trying to reassure his paymasters at Esso Brazil of the value of the polls he was conducting for them on the very day that America's major pollsters miscalled the Truman-Dewey presidential contest in 1948; of Croatian fascists-turned-pollsters enthusiastically joining in on old partisan songs at a Yugoslavian conference; and of the chairman of Revlon interrupting an annual sales meeting to rebuke his 12-year-old son for not doing his homework.
Bogart's profession could be attacked in different ways. There is a popular anti-corporate, progressive argument that sees market research and advertising as a sinister attempt to manipulate people. This perspective is exemplified by such hip magazines as Adbusters and rooted in the theories of such popular writers as Vance Packard (The Hidden Persuaders) and Wilson Bryan Key (Subliminal Seduction), as well as the highfalutin' sociologists of the Frankfurt School.
Alternatively, the rise of market research could be seen not as threatening or malign but as a silly sign of our society's swelling wealth. That wealth allows us to indulge in what are often goofily scientistic attempts to stack up quantitative "knowledge" about men's minds. In the buzzing, blooming confusion of human choices and actions, such knowledge can be harder to pin down than many social research mavens care to admit.
The art and science of finding out and quantifying what people think and want, the basis of Bogart's profession, has an obvious political application as well. That function is explored—and attacked—in Robert Weissberg's new book, Polling, Policy, and Public Opinion: The Case Against Heeding the "Voice of the People."
Weissberg is a professor of political science at the University of Illinois-Urbana. He believes the basic techniques and goals of market research—trying to discover what people want so that you can give it to them—are pernicious when applied to politics. In proving this, he makes some more-daring claims about the unimportance of political democracy and the dangers of reducing civic virtue to political participation.
Does polling even work? Despite some delightful anecdotes of spectacular failure and amusing tales of prankster pollsters getting people to say they'd fight Canada over America's silverfish supply, for the most part the modern version does passing well. More often than not, the candidate whom pre-election polls indicate will win does in fact win. The Gallup Organization, for example, has correctly called every presidential election within three percentage points since 1952, after 1948's embarrassing, poll-driven "Dewey Beats Truman" gaffe. Polls in that year often used "quota sampling" techniques, trying to cover a certain number of people in specific gender, race, or income level categories. That method has since been replaced by the more accurate random sample.
But the 2002 elections saw some fresh poll missteps. Most major polling organizations, including Gallup, called at least one election result wrong, and Zogby International got five results wrong out of 17 polls performed. This led The Wall Street Journal to run a front-page feature casting doubts on polling's future in a world of cell phones and ever-easier ways for us to avoid talking to pollsters.
But Weissberg's critique is not aimed at the scientific art of polling per se. He has a narrower thesis to defend: that polls designed to ask voters if they want more government spending on any given item don't generate politically useful information. But he is also clearly fighting against what he sees as a growing belief within the political science profession: that poll-driven democracy, which permits citizens' voices to be heard (and heeded) at all times, is preferable to our traditional republican system. Weissberg defends periodic elections as all the democracy we require, thank you very much. He thinks most people don't have the slightest idea what they are talking about when it comes to public policy. That's why, he argues, we should be thankful for having elected representatives to make decisions for us.
He's tough about it too. Not for him the pat-on-the-head doctrine of "rational ignorance." That's the widely accepted notion from the public choice school of economics that says it makes perfect sense, given the high cost-benefit ratio of public policy savvy, for citizens to have only dim notions of what the hell is going on in government.
Economists have gone even further in explaining/excusing public sloth in regard to political beliefs and actions. George Mason University economist Bryan Caplan recently has posited the appropriateness of rational irrationality, whereby we choose an optimal amount of absurd and counterfactual things to believe based on what it costs us to hold these unrealistic beliefs.
Caplan's concept would have helped clarify Weissberg's findings, which show that people seem to credulously accept the endless possibilities of government goodies, believing they will all deliver exactly the benefits they promise. Weissberg argues that most polls are systematically biased toward manufacturing a vox populi that clamors for an ever-growing welfare state.
To test this thesis, he designed and executed a pair of surveys that he thinks provide a more sophisticated and accurate way of gauging an intelligent, informed decision—not just an ignorant wish. He used these polls to retest public support for a couple of Clinton-era government expansions: shrinking public school class size by hiring tens of thousands of new teachers, and increasing government-supported day care.
Weissberg found exactly what he was looking for (and one wonders how often that happens in social science research—there's a poll whose results I'd like to see). If you give longer, more detailed polls that demand citizens balance costs within a necessarily limited total budget, and inform them of both the possibilities of failure and the real dimensions of the problem allegedly being solved, previous apparent support for government action and spending quickly fades. For example, if respondents were told that the new teacher program could lead to cutbacks in other school programs, 71 percent of the support evaporated; when informed that an expenditure of $1.2 billion would lower average class size only from 17.8 to 17, 43 percent of supporters changed their minds.
Underlying Weissberg's argument is the dire hint that, to the extent that politicians speak and act in reaction to polls, we are in effect living in a plebiscitory democracy. His larger point is that the people are far too stupid to be heeded by politicians.
Weissberg is openly contemptuous of democracy, falling back on Bell Curve-type arguments about the invincible thickheadedness of many people. He forces the average would-be active citizen to choke down that sour persimmon, then feeds him the ripe plum that "surrendering one's voice is perhaps the surest path to advancement among those outgunned when politics is reduced to hyperindividualism….Powerful interest groups—energetic pluralistic democracy, if you will—offer one sensible alternative….The pretence of grassroots vitality and input would now be sacrificed to tangible accomplishment. Gone are the endless survey consultations….Ordinary citizens pay dues and obey orders." This is the way he thinks it should be.
However much anger that attitude might inspire, one can still agree with Weissberg's observation that most Americans aren't well-educated on political issues or well-trained in sophisticated cost-benefit analyses without embracing the conclusion that they are hopeless dunderheads begging for leaders. When it comes to their private lives, as Weissberg admits and as Bogart's book indicates, even the supposedly civically ignorant are perfectly able to make decisions for themselves. This is because of the vital and too often overlooked (especially by those who conflate marketing with oppression) difference between politics and markets.
In marketing research of Bogart's variety, people are discussing personal wishes and desires for themselves over which they have ultimate power, even if the people gathering the knowledge hope to use it to try to sell them something. A company may take the results of market research polls and craft its ad campaign to appeal to what it has deduced you want. But you decide whether you are going to buy or not, with your money and based on an intimate understanding of your own desires and circumstances that only you can possess.
In Weissberg's world of political polling, people express judgments about how much of other people's property should be used by still other people to pursue goals through methods the effectiveness of which the people being polled have no special ability to judge. While the adbusters of the world try to muddy the distinction, that's a clear and important ethical difference.
Seeking and quantifying the public's opinion is big business. Bogart reports that up to $10 billion annually is spent on commercial survey research. Government studies cost another $3 billion, and academic and nonprofit studies—many also funded by government, of course—add yet another $3 billion. The field of "social research" arose out of an intellectual ferment in the postwar years that is somewhat scary in retrospect: a governmental/academic/corporate movement encompassing Bogart's intellectual roots in the work of Paul Lazarsfeld and his Bureau of Applied Social Research at Columbia, along with CIA-sponsored psychology research and the early days of cybernetics.
Bogart notes that the immediate post-World War II era was "the heyday of the effort to apply social science knowledge to the ideological conversion of Germany" and mentions the success of some prominent ex-Nazis in postwar social research. It was a movement of big institutions, both private and public, trying to figure out ways to understand, quantify, and in many cases control an unruly and frightening mass populace. In reaction to this wide-ranging attempt at social measurement and control, we got the Beats and everything that followed—a cultural ferment reacting against bloodless quantification and control from above.
There have been major successes from the social research field. Gallup lately has been quite reliable at calling presidential elections, and surely many companies have done successful product or service launches based on market research. But a careful reader will notice that the samples Bogart presents of his most interesting work read more like journalism than science: accounts of an observant man talking to people or surveying the media and integrating and explaining what he thinks they are saying.
While it seems to me that the mysterious health insurance company was wasting the money it paid for my focus group, it's their money to waste. Believing in free markets doesn't mean believing in an error-free, best-of-all-possible-worlds equilibrium. Free markets are rife with errors, and the market process and profit and loss system are an efficient method of sorting them all out. As the old saying goes, at least half the money you spend on marketing is always wasted.
My experience with focus groups and many of Bogart's sideways comments (though he is by no means out to bury market research) cast shadows on the bright, clean, rational world of scientific market research. Bogart notes that "many studies are planned and questionnaires are written without the theoretical underpinning that provides insight and understanding rather than mere factual detail of transient interest." He says otherwise savvy business executives often take meaningless numbers seriously, and he mocks those who posit that "the inchoate and monosyllabic utterances of respondents commenting on test advertisements [are] 'opening a window into the human soul.'" He thinks the current marketing obsession with youth is more "herd behavior…than marketing wisdom." Mistakes, it seems, are being made. How many and at what cost? Ask the marketers of New Coke; they might have some clue.
It might be that polling and market research are often a pure consumption expense: entertainment for political junkies, or a way for corporate executives to feel better about risky decisions. Bogart cites a 2000 survey of executives who said that what they overwhelmingly wanted from their market research was "to be told what to do." He observes that "the validity of the advice they received seemed subordinate to the air of assurance with which it was uttered." A lot of the information Bogart presents suggests that, when it comes to marketing, we often don't know what people are going to do and we can't do anything about it. I can see why people would pay a lot to avoid that grim message.
Is this thing called "public opinion" even real and measurable? Weissberg takes care to say he isn't questioning that. But Bogart dismisses the concept as "an amorphous and unstructured combination of sentiments and loyalties." What's more, as marketing guru Oren Harari has noted, our desires and opinions about products and services are eternally changeable and only really discoverable through action. Opinions we express in any artificially created now will be constrained by extant options. The most telling critique of most polling and market research is that it leaves no room for, or doesn't pay enough attention, when people say, "I don't know."
Harari, a professor of management at the University of San Francisco's Graduate School of Business and a Tom Peters associate, noted in a 1994 speech that market research indicated public disdain for answering machines and hair mousse before their introduction. "When people are unfamiliar with a product, or cannot fathom its possibilities," he said, "market research will reflect that and nothing more….Market research can suggest, in a narrow way, what people might prefer or dislike today, but not what will excite them tomorrow."
If grilled about it in a focus group, I'd admit that the pretensions and some of the practices of social research make me uneasy. But when I think of it as an example of how fabulously wealthy our society is, such that we can afford to have whole disciplines full of intelligent people dedicating themselves to it, I'd have to give it a 3 on a 1-to-10 scale of important worries about our culture.
