MENU

Reason.com

Free Minds & Free Markets

The Junk Science Behind 'Smarter Lunchrooms'

A federal program to help public-school students eat healthier is based on highly problematic—and perhaps fraudulent—research.

USDA/FacebookUSDA/FacebookFor the better half of a decade, American public schools have been part of a grand experiment in "choice architecture" dressed up as simple, practical steps to spur healthy eating. But new research reveals the "Smarter Lunchrooms" program is based largely on junk science.

Smarter Lunchrooms, launched in 2010 with funding from the U.S. Department of Agriculture (USDA), is the brainchild of two scientists at Cornell University: Brian Wansink, director of the school's Food and Brand Lab, and David Just, director of the Cornell Center for Behavioral Economics.

The program is full of "common sense," TED-talk-ready, Malcolm Gladwell-esque insights into how school cafeterias can encourage students to select and eat more nutritious foods. Tactics include things like fruit before chips in cafeteria lines; pre-sliced rather than whole fruit; non-fat white milk prominent in beverage displays; giving fruits and vegetables "creative names;" providing "fruit factoids" on cafeteria white-boards; hiding cafeteria cleaning supplies; and having lunchrooms "branded and decorated in a way that reflects the student body."

This "light touch" is the foundation upon which Wansink, a former executive director of the USDA's Center for Nutrition Policy and Promotion and a drafter of U.S. Dietary Guidelines, has earned ample speaking and consulting gigs and media coverage.

Recently, however, scientists are challenging Wansink's work, pointing out errors and inconsistencies going back decades. For instance...

  • The abstract to one of Wansink's published papers claims that schools implementing Smarter Lunchrooms tactics saw a 71 percent increase in apple sales, when the data given in an article table places the actual increase around 4-5 percent.
  • One Wansink-led paper purports to address the sustainability of Smarter Lunchrooms interventions—yet uses data from an experiment that lasted just one day.
  • One published paper lists three different figures throughout (147, 115, and 113) for the number of participants, which should have all been the same, with no explanation.

A History of Inconsistency

In a new paper—which has not yet been peer-reviewed or published—Eric Robinson, a professor with the University of Liverpool's Institute of Psychology, Health, & Society, points to problems ranging from simple sloppiness to errors that seriously call into question the integrity of all of Wansink's work.

Earlier this year, Nicholas Brown, a PhD student at the University of Groningen, discovered that much of Wansink's work was lifted directly from his previous work without citations or acknowledgement—a practice that's at least frowned upon in academia. And in at least one instance, two Wansink papers that purportedly rely on vastly different data sets yielded almost identical end results, down to decimal points, but with enough slight differences to discount simple clerical error.

"If the two studies in question are the same, then it would appear that there has been duplicate publication of the same results, which is not normally considered to be a good ethical practice," writes Brown. "On the other hand, if the two studies are not the same, then the exact match between the vast majority of their results represents quite a surprising coincidence."

In a January paper titled "Statistical heartburn: An attempt to digest four pizza publications from the Cornell Food and Brand Lab," researchers Tim van der Zee, Jordan Anaya, and Nicholas Brown analyze four articles from Wansink and his colleagues, finding "a remarkably high number of apparent errors and inconsistencies." These include:

  • many instances in which the mean or standard deviation given was impossible given the sample size stated in the same table. ("For example, with a sample size of 10 any mean reported to two decimal places must always have a zero in the second decimal place; yet, this table contains means of 2.25 and 3.92 for a sample size of 10.")
  • inconsistent sample sizes and other numbers within and across articles that purportedly use the same data

In total, they found approximately 150 inconsistencies in reported statistics from the four papers.

Those papers are all based on one data set, from a single field-research experiment involving diners at an Italian lunch buffet. "However, none of the articles mentions that they are based on the same data set as their predecessors, even though they were published over a period of many months," the van der Zee study notes. "We consider that this may constitute a breach of good publication ethics practice."

In a now-deleted blog post, Wansink characterized his Italian-buffet experiment as a "failed study which had null results." But when Wansink's first thesis (which he never reveals) didn't pan out, he went looking for ways to reverse-engineer the data he did have into a thesis.

When [a visiting fellow] arrived, I gave her a data set of a self-funded, failed study which had null results (it was a one month study in an all-you-can-eat Italian restaurant buffet where we had charged some people ½ as much as others). I said, "This cost us a lot of time and our own money to collect. There's got to be something here we can salvage because it's a cool (rich & unique) data set."

I had three ideas for potential Plan B, C, & D directions (since Plan A had failed). . . . Every day she came back with puzzling new results, and every day we would scratch our heads, ask 'Why,' and come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually we started discovering solutions that held up regardless of how we pressure-tested them. . . .

Wansink's colleagues—including Andrew Gelman, director of the Applied Statistics Center at Columbia University—quickly called him out on this cavalier attitude toward theory and apparent willingness to hack statistics into something publishable.

"Statistical tests are a messy business, our criteria are not stringent, the samples are small, something is bound to come up as significant if we look hard enough," noted neuroscientist Ana Todorovic. "Now, it's significant whether we looked at it or not–testing the data in many different ways is not the problem. The problem is not reporting all the other variables that were collected and all the other tests that were carried out. Because if we knew this was one result out of, say, 200 tests, then we would be less likely to give it much credence."

Gelman called Wansink's work "run-of-the-mill, everyday, bread-and-butter junk science."

No Evidence? No Problem

As Jesse Singal at New York asked in March: "What possible reason is there to trust [the Cornell Food and Brand Lab's] output at all, let alone for journalists to continue to publicize its findings?"

Or, we might add, for the federal government to continue funding its implementation?

Far from being a well-tested approach, the Smarter Lunchrooms program was implemented in schools across America before the first randomized control trials on its effectiveness were even begun. "The speed at which this intervention approach has been implemented in schools surprised me because of the limited and low quality evidence base supporting it," wrote Robinson in the most recent comprehensive critique of Wansink's work.

The first serious study testing the program's effectiveness was published just this year. At the end of nine weeks, students in Smarter Lunchroom cafeterias consumed an average of 0.10 more fruit units per day—the equivalent of about one or two bites of an apple. Wansink and company called it a "significant" increase in fruit consumption.

But "whether this increase is meaningful and has real world benefit is questionable," Robinson* writes.

Nonetheless, the USDA claims that the "strategies that the Smarter Lunchrooms Movement endorses have been studied and proven effective in a variety of schools across the nation." More than 29,000 U.S. public schools now employ Smarter Lunchrooms strategies, and the number of school food service directors trained on these tactics increased threefold in 2015 over the year before.

One study touted by the USDA even notes that since food service directors who belong to professional membership associations were more likely to know about the Smarter Lunchrooms program, policy makers and school districts "consider allocating funds to encourage [directors] to engage more fully in professional association meetings and activities."

Earlier this year, Wansink responded to criticism by promising that he and his colleagues would review the data involved in questionable studies and make it publicly available. But this has not happened yet, and the blog post where Wansink made this offer has since been deleted.

A spokesman for the USDA told The Washington Post that while they had some concerns about the research coming out of Cornell, "it's important to remember that Smarter Lunchrooms strategies are based upon widely researched principles of behavioral economics, as well as a strong body of practice that supports their ongoing use."

Some might wonder what it hurts to put fruit in front of chips in school-cafeteria lines or ask lunchroom staff to consider aesthetics. For libertarians, one answer is obvious: we are shelling out all sorts of money and government resources to incentivize activity with no measurable benefit (all while creating new administrative positions and layers of bureaucracy to oversee these efforts).

Not only is there scant evidence to support the pricey program, the "Smarter Lunchrooms" boondoggle also distracts from figuring out how to really reform school cafeterias—something that folks on all sides should find concerning.

We might disagree on whether federal authorities should micromanage lunchroom menus or if local school districts should have more control, and what dietary principles they should follow; whether the emphasis of school cafeterias should be fundraising or nutrition; or whether school meals need more funding. But confronting these challenges head-on is a hell of a lot better than a tepid consensus for feel-good fairytales about banana placement.

*Correction: Eric Robinson, not Anderson, is the author quoted here.

Photo Credit: USDA/Facebook

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  • Juice||

    Whole milk is better for you than non-fat milk.

  • Hugh Akston||

    That caught my eye too. Are we still pretending that removing one of the principal reasons for milk's existence somehow makes it better?

  • Rhywun||

    Mine too. It's especially annoying because the skim milk bandwagon is one of the nannies' favorites - I see commercials pushing that crap all the time.

  • Elizabeth Nolan Brown||

    I know http://reason.com/blog/2014/04.....-for-kids, but USDA obviously doesn't

  • Chipper Morning, Now #1||

    Let's replace whole milk with skim milk, but let's serve the kids pizza and burgers.

  • Fist of Etiquette||

    Nice link... NOT.

  • Griffin3||

    I would rather light a candle than curse the darkness --> http://reason.com/blog/2014/04.....k-for-kids

  • Chipper Morning, Now #1||

    So the vampires could find you easier? No, thanks.

  • ||

    Got milked.

  • Headache||

    Cows don't drink that stuff after 6 months.

  • ||

    It makes it more profitable.

  • Microaggressor||

    Deliberately stunting the development of public school children - bug or a feature?

  • Unlabelable MJGreen||

    But the program is to stop kids from getting fat. So why would you give them fat?

    Think about it!!

  • Agammamon||

    Tactics include things like fruit before chips in cafeteria lines; pre-sliced rather than whole fruit; non-fat white milk prominent in beverage displays; giving fruits and vegetables "creative names;" providing "fruit factoids" on cafeteria white-boards; and having lunchrooms "branded and decorated in a way that reflects the student body."

    I don't understand this. If the government has a duty to 'promote health' then doesn't that mean we've given it the privilege to force healthy choices? That's the way it works in all other areas - if the government has been given a duty to perform a task then its also been given the privilege to compel cooperation.

    So why the soft-touch? You can 'encourage' kids to eat healthier simply by *removing* alternative choices. Don't want them to eat chips? Don't serve chips.

    Or maybe the *real* goal here has nothing to do with health but in inculcating the idea that if the government 'suggests' something its not a suggestion and you had better comply. I guess that saves money on jackboots or something.

  • CE||

    Out of an abundance of caution, we should shut down the public schools until we can be sure that not one more child will get obese or sick or less healthy from eating at public school cafeterias.

  • Marcus Aurelius||

    Peanut butter jelly time!
    Peanut butter jelly time!
    Peanut butter jelly with a baseball bat!

  • Fist of Etiquette||

    I continue to advocate for lunch feeding school children Soylent brand food paste. I don't care if they did go out of business.

  • Chipper Morning, Now #1||

    The reason kids are fat today is the lack of physical movement: social media, video games, etc. Our entertainment was running around in the woods and throwing rocks at each other.

  • Eric Bana||

    Plus, when people are sedentary they tend to snack on things. You can't really snack on chips and pretzels while you're in a game of tag. So spending an hour looking at facebook on your phone while chowin' down on corn chips means burning fewer calories and consuming even more.

  • CE||

    Plus riding bikes down steep hills without helmets, shooting arrows into the sky and running like mad so they don't land on you, plus running around with sharpened sticks having sword fights.

  • ||

    which is not normally considered to be a good ethical practice

    Wow. This makes 'Not OK' seem forceful and decisive.

  • Scarecrow Repair & Chippering||

    Are they British exiles because of Brexit?

  • Rhywun||

    No because then he would have written "practise".

  • Rhywun||

    fruit before chips in cafeteria lines

    My HS cafeteria didn't have chips. Only the rich kids who brought their lunch from home had junk food.

  • Rhywun||

    I might add this was 30 years ago.

  • ||

    Stop lying. You didn't attend HS with five.

  • Rhywun||

    Wut?

  • ||

    I said you can't be older than 35. Either that is or I am too old for this.

  • Juice||

    We had chips and coke in the vending machine. They were 50c 25+ years ago. The concession stand sold candy and picked pig lips (it was a mostly black school in the South).

  • ||

    Coke in glass bottles. I'm annoyed by the rise of plastic bottles, which make stuff taste worse.

  • BYODB||

    It's not the glass or plastic that makes it taste worse or better, it's the sweetener's they use. AFAIK 'real cane sugar' versions generally only come from Mexico and in glass bottles, but I live in Texas so it's possible other places export that delicious crap to us now.

  • ||

    I'm comparing classic sugar in plastic with classic sugar in glass. That's an effect on its own. I also agree with you regarding artifical sweeteners.

  • Rhywun||

    Vending machines? Oo, la la.

  • CE||

    When I went to school, vending machines hadn't been invented yet. Or we couldn't afford them. And nobody had those tiny individual potato chip bags yet either, or juice in a box.

  • BestUsedCarSales||

    More Reason articles need to shit on Ted talks. Someone tell Gillepsie to prepare more snide sarcastic references to destroy Ted's credibility and emotional state.

  • NoVaNick||

    I get a good laugh out of TED talks. Its hysterical to listen to these "experts" pontificate for an hour about things that are mostly common sense, or could be explained in 5 minutes or less, or essentially are meaningless. What's funnier though are all the dolts who spend good money to go see them.

  • BYODB||

    TED Talks serve only one purpose, to line the pockets of public speakers who have written a junk science book and need to go and promote it.

    My wife-to-be got suckered by one of these organizations that charges something to the tune of $20,000 and purports to 'teach you how to write a book, market it, and sell it' but what they don't tell you is that they're basically con artists grifting people and occasionally those marks learn how to grift by example. If someone proves they can grift others, they'll hire them on to their con artist company.

    I told her as much at the time, but after their 'free seminar' she had that 'need for action' that they oh-so-desperately want to instill into people at point of sale. Madness. Thankfully one of her family members fronted the cost, but I suppose it doesn't bode well long term that she's that gullible.

  • ||

    Write a book or do a TED talk to teach her.

  • Suckin' Seed||

    "it's important to remember that Smarter Lunchrooms strategies are based upon widely researched principles of behavioral economics,"

    Maybe I am ignorant or naive, but I thought "behavioral economics" was about studying behavior to explain a result not manipulating behavior to elicit the expected result. Also when I say "N-Word" what is flashing in your skull right now? AHA! Racist!.

  • geo1113||

    Naggers!!!

  • Zeb||

    I guess the latter would be "applied behavioral economics".

  • Marcus Aurelius||

    Ntifa?

  • Cynical Asshole||

    ...feel-good fairytales about banana placement.

    Hey now, proper banana placement is a very serious issue.

  • ||

    Banana first. Make apples great again. An apple a day keeps Sunstein and Thaler ... alright, I had too much sugar.

  • Juice||

    We've already done passion fruit!

    We haven't done bananas yet, have we?

  • Crusty Juggler||

    It takes a village and junk science to keep our fat kids from becoming fatter kids.

  • Eric Bana||

    When you look at kids today, you do see a lot that are overweight. It's kind of scary.

  • ||

    One in five apples will be eaten.

  • C. S. P. Schofield||

    Somehow, no 'initiative' on school lunches ever involves asking the kids what they want to eat, or the lunchroom ladies what the equipment they use would do reasonably well.

    Said; I remember, going back some decades, the flap over (supposedly) some Reagan administration functionary calling catsup a vegetable, for school lunch purposes. I recall thinking at the time "Have any of the people carrying on about this SEEN how kids use catsup? In those quantities it damn well IS a vegetable serving"

  • Ron||

    the school system will make the best of meals the worst tasting crap ever. Improve the cooking methods and kids will eat it, no fancy names or tricks needed

  • BYODB||


    Wansink's colleagues—including Andrew Gelman, director of the Applied Statistics Center at Columbia University—quickly called him out on this cavalier attitude toward theory and apparent willingness to hack statistics into something publishable.


    Uhh...yeah? Do you even science? If you don't publish, what do you do? You perish. Did this guy make a shit ton of cash, or no? If he did, success.


    What, you expect a scientist to pay out a lot of cash for a poorly thought out study that turned out to be useless? Hell no! You work backwards and simply redo your thesis. That's how guys like this got their Masters and probably their Ph.D. so it's no surprise that's how they conduct themselves once they're a 'Doctor'.

  • ||

    Possibly he got the behavioral economics of monetizing "science" right.

  • AndrewJJ||

    They are missing big picture.
    All options are mac and cheese/pizza/nuggets/grilled cheese/hot dog and burgers-just like most kids menus

    But the problem is most parents are lazy and feed their kids that do not sure who it to blame.

    Maybe a variant would be chicken product-it's actually called that.

    There has got to be a crony deal there providing all the basic crap so who cares if they throw a few bananas at it.

    The salads they do have are crap as well-iceberg and little else.

    Yeah I'm a food snob and I'm raising my kid that way as well so there are no good options for us.

    I'd gladly pay if they had a good option-but they don't so it's bring your lunch for us.

    Some free investment advice-invest in "fast casual/hip" mac and cheese places in a few years. It's all many of these kids will eat

  • buybuydandavis||

    many instances in which the mean or standard deviation given was impossible given the sample size stated in the same table. ("For example, with a sample size of 10 any mean reported to two decimal places must always have a zero in the second decimal place; yet, this table contains means of 2.25 and 3.92 for a sample size of 10.")

    This is nonsense.

    The mean is a mathematical calculation. It is what it is. You're confusing standards on the use of *significant digits* in limited sample sizes, which are arguable, with mathematical impossibility.

  • Telcontar the Wanderer||

    "how to really reform school cafeterias"

    1. Rule the school cafeterias

    2. Find the school cafeteria management staff

    3. Bring them all to a dark room or closet

    4. In the darkness, bind them.

  • Gimlet Eye||

    The deleted blog post plus its lively comment section. (Second link includes an addendum. Both go to Wayback Machine. Links shortened due to posting requirements).

    http://bit.ly/2xOmpWu
    http://bit.ly/2xzGL6C

GET REASON MAGAZINE

Get Reason's print or digital edition before it’s posted online