Eyeing the Calorie-Watcher Researchers

A new study on the impact of calorie information at fast food and chain restaurants raises more questions than it answers.

Research in the area of food law and policy is both varied and fascinating—even if more than a little of it isn't worth the paper it's printed on.

I've previously looked at studies on whether food logos make kids fat, whether organic food is a waste of money, whether Americans are cool or warm to the idea of food freedom, and whether alcohol is the new soda.

A new study highlighting how youths eat in fast food restaurants serves up some interesting lessons on caloric intake—and on research on that topic.

The new article in the Journal of Public Health, by Centers for Disease Control and Prevention (CDC) researcher Holly Wethington and her colleagues, explores whether adolescents and teens utilize calorie information to inform their food choices in fast food/chain restaurants.

The article looks at "the proportion of youth who reported using calorie information when available at fast food/chain restaurants."

Previous research has often questioned the validity and effectiveness of mandatory menu labeling.

As Chicago Tribune columnist Monica Eng wrote last year, "People may notice calorie counts on menu boards but, so far, few use the data to make significant changes to their orders."

Eng noted a New York University study that found just 9 percent of teens used menu labeling to inform their choices.

The NYU report concluded that while "a few considered the information when ordering…. [w]e found no statistically significant differences in calories purchased before and after labeling" regulations in New York City took effect.

In the case of the CDC researchers' study, the authors correctly point out several limitations of their work. They note that there's no way they can determine whether the youths surveyed made better or worse choices with the calorie information they had in hand; that the study cannot determine causality; and that the reliability and validity of the questions posed were not tested.

I noticed a few other important limitations.

At first glance, the utilization of calorie information seems quite high in this study compared to that of the NYU study cited by Eng, also referenced by the CDC study authors.

The CDC authors claim "40% of youth who noticed calorie labeling information within a fast food/chain restaurant reported using calorie information[.]" Recall that the NYU study reported just 9 percent utilized calorie information.

But a closer look at the data and collection methods tightens some of the discrepancies.

The CDC authors did not include teens and adolescents who reported eating at fast food/chain restaurants but who never noticed menu labeling. When included in the data analysis, the percentage of respondents who eat at fast food/chain restaurants and don't notice or use calorie information rises to 65.9 percent, while the percentage of those who say they noticed the information and used it at least once falls to 34.1 percent.

The number who use calorie information, though still small compared to those who do not, is still much higher in the CDC study—nearly four times as high—than the number who use calorie information in the NYU study.

Why the discrepancy? I can't say for sure. However, I suspect the difference arises largely from the CDC researchers' total reliance on self-reporting of data by teens and adolescents.

In the NYU study, researchers collected purchase receipts from 349 teens, adolescents, and their families who visited fast food/chain restaurants both before and after the implementation of New York City's menu-labeling law.

That's hard data.

The CDC study, on the other hand, collected data from the self-reported YouthStyles survey given to teens and adolescents nationwide. They analyzed data from respondents who said they "ate at fast food restaurants and noticed the [calorie] information while ordering."

Furthermore, while the CDC authors report that "some fast food/chain restaurants have begun to post [calorie] information voluntarily," the data Wethington and her colleagues analyzed comes exclusively from 2010.

That date largely precedes both voluntary and mandatory menu labeling (in which calorie information is posted alongside menu items). McDonald's, the largest national fast food/chain restaurant to post calorie information, did not begin doing so nationwide until 2012. Other competitors may follow (before a mandatory national law takes effect sometime in the future), but few have done so to date.

Meanwhile, the largest jurisdictions to participate in the YouthStyles survey the CDC analyzed and which also had menu-labeling laws in 2010, according to this April 2011 map created by the Center for Science in the Public Interest (CSPI), were New York City and the state of California.

But data from those two key places is slim. The CDC study authors note that they did not collect sufficient data for New York City, stating respondents included "too few youth from NYC." Meanwhile, it appears only a maximum of 13 percent of survey respondents hailed from California (which is presumably part of the study's results for its "Pacific" region).

What about menu labeling outside New York City and California? An overwhelming majority, 39 out of 50 states, had no menu-labeling laws within their borders whatsoever in 2010, according to the CSPI map. Nine of the 11 states had only one or more municipal laws (e.g., Philadelphia had a law in place but Pennsylvania and every other city in the state did not) and only two states (Vermont being the second) had implemented menu-labeling laws.

How can so many young Americans have claimed to "notice[ calorie] information while ordering" in cities and states where restaurants most often don't provide that information at the point of sale (though many do online, or elsewhere in stores)?

In short, I believe that many of the 9-18 year-old survey respondents were for whatever reason mistaken in their response to the question about whether they use calorie information provided in a fast food/chain restaurants to inform their decisions.

If I'm right, then that would make the validity of the CDC researchers' study, based wholly on these responses, open to debate—something that holds true for an increasing number of studies in the area of food law and policy.