What Can We Learn From National Primary Polling? Virtually Nothing.
Who's up? Who's down? Who cares?
In the spring and early summer of 2007, the frontrunners for the Democratic and Republican nominations for president were—by healthy double-digit margins—New York Sen. Hillary Clinton and former New York City mayor Rudy Giuliani. Oops.
It's become a cliche to joke about these comically inaccurate results from exactly eight years ago, but they are far from the exception. National survey research, even right before primary elections and caucuses, is almost useless as a barometer of who is most likely to win a party's nomination—except in those circumstances when one candidate is so far ahead that no polling is necessary to know it.
This fact will not deter the press from spilling barrels of real and virtual ink writing about each new data point for the next 12 months. "A new poll of possible 2016 presidential contenders shows a crowded Republican field," begins a recent, and utterly typical, article from the International Business Times, "while former Secretary of State Hillary Clinton has a wide lead on the Democratic side."
But primary polling does not measure who is most likely to win a party's nomination. Betting markets, such as the now-defunct Intrade, which shuttered its operations after federal regulators brought suit against it in late 2012, come closer to doing that. They ask participants to look into the future and guess what public opinion will be like on the actual day of the election. Primary polling, on the other hand, measures whom people would support were the voting to happen right now.
The further out from Election Day a survey is conducted, the less likely the result is to hold. National polls taken months before the first primaries are often astonishingly bad at forecasting how the chips will have fallen by the end of a several-months-long series of nominating contests.
Consider the 2008 race. Summer 2007 saw Hillary Clinton leading all comers by an intimidating margin. National surveys conducted for ABC News and The Washington Post in June and July of that year put her 15 percentage points ahead of her main opponent, Illinois Sen. Barack Obama. In December, mere weeks before the Iowa caucuses, an NBC News/Wall Street Journal poll found her winning by 22 percentage points. Even heading into Super Tuesday (February 5), most national polls gave the former First Lady an edge. Nonetheless, Obama became the nominee.
Things were even more volatile on the Republican side of the aisle. Through the summer of 2007, Giuliani ran far ahead of the pack, while the eventual winner of the nomination, Arizona Sen. John McCain, was straggling somewhere between second and fourth place. When Giuliani started slipping in September, it wasn't because McCain was finally making a run for it; it was because Fred Thompson of Law & Order fame had jumped into the race and grabbed some of the mayor's supporters.
Thompson's moment proved short-lived. By November, Giuliani was back on top and, according to an NBC News/Wall Street Journal poll, capturing a third of the primary vote—a remarkable feat in a packed field of strong contenders. Onlookers might have reasonably concluded he was likeliest to win the nomination. After all, he'd held the top position for more than a year.
Then in December came a breathless memo from CBS News and The New York Times: "The race for the Republican nomination nationwide has undergone sweeping changes since October: former Arkansas Governor Mike Huckabee now challenges Rudy Giuliani for the lead, with Mitt Romney on their heels. Just six points separate the three candidates." Not mentioned there: McCain, whom the poll found tied for fourth and taking just 7 percent of the vote.
Anyone trying to judge the relative chances of the various candidates based on national primary polls conducted before voting began would have been hopelessly lost. There was little indication the Arizona senator was capable of coming back to win the January 8 New Hampshire primary, let alone the whole shebang. Yet somehow, McCain would go from capturing 12 percent of likely voters in a December Washington Post/ABC News poll to 48 percent in February en route to winning the nomination.
The 2012 race was even crazier. GOP voters flirted seriously with nearly every other candidate before finally settling on Romney. The chart below from HuffPost Pollster shows four different people—Texas Gov. Rick Perry, pizza mogul Herman Cain, former House Speaker Newt Gingrich, and former Pennsylvania Sen. Rick Santorum—beating the erstwhile Massachusetts governor at various moments. The shape of the graph mirrors the roller coaster ride taken by anyone trying to predict the GOP nominee solely from polling.
Pollsters contend with an array of methodological obstacles. Response rates today are a fraction of what they were when everyone had landline telephones and no one had caller ID. And as we get closer to elections, pollsters try to limit their surveys to "likely" voters, but their processes for doing this are inexact.
The American system of holding many discrete statewide primaries also makes national polls questionably relevant. Because states don't all vote on the same day, the outcome of one election invariably alters expectations and thus choices in subsequent surveys. "You could measure voter preferences right now in Texas all you want," says Mark Blumenthal, a former pollster who now runs the HuffPost Pollster site, "but I guarantee you they're going to be different after Iowa, New Hampshire, and South Carolina than they are right now. And they may be different in the midst of those three primaries than they were before or after."
Giuliani famously discovered this in 2008, when his strategy of ignoring the early primary states and focusing on Florida and the Super Tuesday elections ended in spectacular failure. McCain and Romney, his top opponents, were able to pick up momentum with wins in Nevada, South Carolina, and Michigan. Meanwhile, Giuliani could not recover from the perception that his campaign had collapsed even before voters in the Sunshine State, where he'd concentrated his resources, went to the polls.
The biggest weakness of primary polls is evident from the very question they pose. Most surveys ask some variation of: "If the election were held today, for whom would you vote?" Respondents can only take into account the information they have at hand at the moment. The situation on Election Day, after months of news coverage and millions of dollars in advertising, is sure to be vastly different.
And while general election voters get to choose between the nominees of political parties they have long known and loved (or hated), primary voters have no equivalent default option to vote for or against a party. This means the answers they give pollsters are subject to violent swings.
"When you have a lot of candidates who are poorly known, there are events that can happen that might temporarily alter the snap preferences people give when interviewed," Blumenthal explains. "A lot of the people who will vote in the primaries haven't thought about this yet [and] don't know the names of all the candidates. Yet when intercepted in the midst of doing whatever they're doing and read a list of names, they will try to pick one…If there was a lot of news this week about a candidate jumping into the race, some people are going to pick that name just because they heard it recently."
Journalists are already showing they haven't internalized these quadrennial lessons. Since Barack Obama was re-elected, newspapers and political commentators have declared no fewer than five different Republicans as the man to beat in 2016.
Last September, The Washington Post was calling Kentucky Sen. Rand Paul the frontrunner for the Republican nomination. By the end of January, the same paper was describing former Florida Gov. Jeb Bush that way. In December, The Huffington Post ran this headline: "Congrats, America: Ben Carson Is Now Your 2016 GOP Frontrunner." But in February, as Wisconsin Gov. Scott Walker inched his way to the top of the polls, the BBC called him "a little-known 2016 frontrunner." A few weeks later, National Review titled a post: "No, Really, Scott Walker is the Frontrunner." Even Mike Huckabee, fresh off Fox News and boasting a huge popular following in Iowa, has been tagged as a frontrunner in some conservative outlets, including the Washington Examiner.
One of these predictions may end up looking prescient. But surveys conducted now, more than half a year before the primaries even start, do virtually nothing to tell us which one it will be.
This article originally appeared in print under the headline "What Can We Learn From National Primary Polling? Virtually Nothing.."
