Measuring Up

Testing the pretensions of market research and polling.

Brian Doherty from the May 2003 issue

(Page 3 of 3)

Bogart notes that the immediate post-World War II era was "the heyday of the effort to apply social science knowledge to the ideological conversion of Germany" and mentions the success of some prominent ex-Nazis in postwar social research. It was a movement of big institutions, both private and public, trying to figure out ways to understand, quantify, and in many cases control an unruly and frightening mass populace. In reaction to this wide-ranging attempt at social measurement and control, we got the Beats and everything that followed -- a cultural ferment reacting against bloodless quantification and control from above.

There have been major successes from the social research field. Gallup lately has been quite reliable at calling presidential elections, and surely many companies have done successful product or service launches based on market research. But a careful reader will notice that the samples Bogart presents of his most interesting work read more like journalism than science: accounts of an observant man talking to people or surveying the media and integrating and explaining what he thinks they are saying.

While it seems to me that the mysterious health insurance company was wasting the money it paid for my focus group, it's their money to waste. Believing in free markets doesn't mean believing in an error-free, best-of-all-possible-worlds equilibrium. Free markets are rife with errors, and the market process and profit and loss system are an efficient method of sorting them all out. As the old saying goes, at least half the money you spend on marketing is always wasted.

My experience with focus groups and many of Bogart's sideways comments (though he is by no means out to bury market research) cast shadows on the bright, clean, rational world of scientific market research. Bogart notes that "many studies are planned and questionnaires are written without the theoretical underpinning that provides insight and understanding rather than mere factual detail of transient interest." He says otherwise savvy business executives often take meaningless numbers seriously, and he mocks those who posit that "the inchoate and monosyllabic utterances of respondents commenting on test advertisements [are] 'opening a window into the human soul.'" He thinks the current marketing obsession with youth is more "herd behavior...than marketing wisdom." Mistakes, it seems, are being made. How many and at what cost? Ask the marketers of New Coke; they might have some clue.

It might be that polling and market research are often a pure consumption expense: entertainment for political junkies, or a way for corporate executives to feel better about risky decisions. Bogart cites a 2000 survey of executives who said that what they overwhelmingly wanted from their market research was "to be told what to do." He observes that "the validity of the advice they received seemed subordinate to the air of assurance with which it was uttered." A lot of the information Bogart presents suggests that, when it comes to marketing, we often don't know what people are going to do and we can't do anything about it. I can see why people would pay a lot to avoid that grim message.

Is this thing called "public opinion" even real and measurable? Weissberg takes care to say he isn't questioning that. But Bogart dismisses the concept as "an amorphous and unstructured combination of sentiments and loyalties." What's more, as marketing guru Oren Harari has noted, our desires and opinions about products and services are eternally changeable and only really discoverable through action. Opinions we express in any artificially created now will be constrained by extant options. The most telling critique of most polling and market research is that it leaves no room for, or doesn't pay enough attention, when people say, "I don't know."

Harari, a professor of management at the University of San Francisco's Graduate School of Business and a Tom Peters associate, noted in a 1994 speech that market research indicated public disdain for answering machines and hair mousse before their introduction. "When people are unfamiliar with a product, or cannot fathom its possibilities," he said, "market research will reflect that and nothing more....Market research can suggest, in a narrow way, what people might prefer or dislike today, but not what will excite them tomorrow."

If grilled about it in a focus group, I'd admit that the pretensions and some of the practices of social research make me uneasy. But when I think of it as an example of how fabulously wealthy our society is, such that we can afford to have whole disciplines full of intelligent people dedicating themselves to it, I'd have to give it a 3 on a 1-to-10 scale of important worries about our culture.