The Challenge of Measuring Obamacare's Results

Several recent decisions have made tracking the health law's outcomes more difficult.



In September 2009, when President Obama made a primetime speech pitching his not-yet-passed health care overhaul, he made the following promise: "I will not sign a plan that adds one dime to our deficits—either now or in the future. Period." To prove his seriousness, he further promised that "there will be a provision in this plan that requires us to come forward with more spending cuts if the savings we promised don't materialize."

The promise of deficit reduction was repeated over and over in the months before the bill became law, and it was central to Obamacare's passage. Congressional Democrats would likely not have voted to pass it had the Congressional Budget Office (CBO) not scored the bill as a net reduction in the nation's deficits. Yet earlier this month, the CBO, which provided the original estimate and evidence that the law would be deficit neutral, said that it can no longer score the net fiscal effect of the law in its entirety.

The problem is that in addition to expanding health coverage, Obamacare also modified a variety of existing federal programs, and it was through those modifications that it achieved much of its deficit reduction. But those changes have not all been implemented as originally planned, and now the agency says it can no longer judge the complete fiscal effects. "Isolating the incremental effects of those provisions on previously existing programs and revenues four years after enactment of the Affordable Care Act is not possible," a budget office report said in April.

What that means is that if the deficit savings Obama promised don't materialize, we may never know.

Indeed, there's a lot that will be tough to know about Obamacare, both now and in the future—and it's not just because of the CBO. Over the last few months, a series of reporting and measurement changes from a variety of government agencies have made it vastly more difficult to usefully measure the law's outcomes. A law as sprawling and complex as Obamacare was always going to be a challenge to measure; these decisions have made it harder still.

The pronouncement from the CBO, which came buried in a footnote that was not noticed for more than a month, came on the heels of an earlier announcement from the administration that it would cease its recurring reports on Obamacare sign-ups following the close of the law's first open enrollment period. Those reports, which provided official numbers on sign-ups and basic demographic information, were often frustratingly incomplete. But as limited as those reports were, they were one of the few official sources of information about the results of the law.

The administration's decision means that, at minimum, it won't be possible to see how enrollment fluctuates outside of the open enrollment period, an issue that has frustrated some supporters of the law. And it may mean that we will lack even the limited clarity of this year's reports in coming years: The administration won't commit to releasing the reports again once next year's open enrollment period begins this fall, according to a report in Politico.

Tracking the effects of the law on health insurance coverage has been made even more difficult by a series of changes to the way the Census Bureau measures the uninsured. The Census has long provided the canonical estimate of the number of uninsured around the country, and when advocates made the case for Obamacare based on the number of uninsured, they usually used figures pulled from Census data.

But in April, The New York Times reported that the Census Bureau would be changing the way it asked about insurance status in what the Bureau describes as a "total revision to health insurance questions."

Researchers say that the old methodology significantly overstated the number of uninsured; the new questions will produce a lower number. That's convenient for Obamacare. But what it means is that it won't be possible to determine how much of the reduction is caused by the change in questions rather than by policy changes or other factors.

As a result, the new data on the uninsured won't be comparable with the Census' long history of data on the uninsured. It's a break in the time series, one that leaves only a single year of pre-Obamacare coverage data for comparison—a year, 2013, in which the insurance market was already undergoing significant upheaval.

Census Bureau officials say the change was driven by a desire to get more accurate information about the uninsured. But several of the new questions were requested by the Obama administration's Department of Health and Human Services, according to The New York Times, and the White House budget office approved the update. 

In one case, federal health officials even tried to intervene to stop state authorities from reviewing health insurance data. Earlier this year, the Centers for Medicare and Medicaid issued a directive saying that states should not audit eligibility for public health insurance programs like Medicaid because they are developing their own system.

But in Minnesota, the state's legislative auditor, James Nobles, has to proceed with a review of how the state's Obamacare exchange determines eligibility as part of a comprehensive examination of the insurance portal, despite the order not to proceed. He described the federal order as "unprecedented," according to a report by Minnesota public radio. "The reason we have external auditors, independent auditors," he said, "is because you need somebody from the outside to come in and hold people accountable [for] what they've pledged to do."

That's not just true in Minnesota. President Obama promised accountability for the health care law. But these changes have made tracking the law's results, and holding anyone accountable, much more difficult.

NEXT: Beltway Journos Grasp at Credential Authority to Try to Keep Out Blogger Experts Like SCOTUSBlog

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  1. Several recent decisions have made tracking the health law’s outcomes more difficult.

    Feature, not bug.

  2. Everything becomes vaguer and harder to pin down–if not outright unknown altogether–as we continue our brisk stroll to outright tyranny.

    1. That’s sort of the plan, ProL.

      1. It’s almost like they know exactly what they’re doing.


      2. Do you have access to this plan? Is it in Power Point? I have a feeling it’s in Power Point.

        1. Of course it is… PowerPoint is the Devil.

      3. You mean The Plan. Because like the Cylons they really don’t have one as they make shit up on the fly towards some ill-defined nefarious purpose.

        1. Yeah, whatever. They had a Plan that none of the writers had bothered even thinking up, deferring that until some later date, which came and went without any Plan being concocted. Except some sort of random walk Plan, which I believe involved monkeys and keyboards.

          With all due respect to Ron Moore and his team, who kicked ass for a couple of seasons.

          1. Is it really that hard to have a basic overall story arc in place from the beginning?

            Hell, Babylon 5 had one.

          2. Their plan was the extermination of humanity, then it morphed into finding Earth and giving their race a fresh start. The story arc of the series was always about finding Earth, and I don’t think they ever really diverged from it…the other major plots were about the relevance of religion and faith and man containing the seeds of his own destruction in failing to learn from past errors.

            The line about the “plan” at the beginning of each episode was just a hook to get you to watch.

        2. Trust me, they do have a plan…!

          1. “If you like our plan you can keep it”

  3. “Yeah, we lied about deficit reduction and everything else. What are you gonna do about it?”

    1. Definitely not on my Christmas list.

  4. “Every great magic trick consists of three parts or acts. The first part is called The Pledge…”

    1. “Lookest thou over there….madest thou look!”

      1. +1 oldest trick in the book

  5. OT: How did the government know to put out a bid for services for 65,000 undocumented children back in January, long before tens of thousands of them began showing up? I try to resist conspiracy theories, but this seems pretty damning.

    1. Look what good planners they are!

    2. It’s not. 65,000 is the anticipated number based on 2013’s number of UACs. It just makes sense to plan ahead based on projections.

      1. Seewhutamean, Vern?

      2. That’s what I was going to say.

        As rare as they are, *every* agency does have one or two guys who know how to gauge utilization and know how to plan ahead.

        Sometimes they are even listened to.

        1. Regression analysis, FTW!

      3. Projection is the only skill they have.

        Oh, you mean numeric projection.

  6. …”escort services for unaccompanied alien children”…

    I’m as libertarian as the next guy, but I don’t think children should be using escort services. Age of consent, and all.

    1. Exploitation Olympics.

      Kid hires a hooker – is he being exploited by the adult or is the ‘trafficked person’ being exploited?

      1. It’s exploitation all the way down

        1. What you did there…

  7. I think if you consider the operating mantra of the Obama administration to be “obfuscate as much as possible until we’re out of here”, you will find yourself able to predict the things they’re going to do very well.

    1. I consider the Operating Mantra? of the Obama Administration to be


    1. Everyone dies OR a character introduced late in the game “wins”.

      Did I get it right?

      1. Probably.

      2. Rickon’s dire wolf wins. The whole thing turns out the be a Shaggy Dog story.

    2. Wow. So jealous of the Queen and Prince Philip.

      They got to touch Rose Leslie!

    3. Do the zombies finally kill the annoying characters (by which I mean everyone)?

      I guess that question works for ‘The Walking Dead’ also.

    4. Lemme guess… Universal Healthcare?

    5. God damn, what vows I would break (or “bend” depending on your interpretation) for Ygritte.

  8. When the new Census data is released regarding the uninsured, does anybody really believe the media won’t tout it as a success for Obamacare? Sure, they’ll mention that the way it was counted was changed, but it will be buried in the story and probably limited to one sentence.

    1. It will be as valid as the prison head count in North Korea.

  9. I haven’t been paying attention, are they still claiming to be the most transparent admin ever?

    1. You can see right though them, can’t you?

  10. President Obama promised accountability for the health care law.

    “Let me be clear. We’re running in the black!”

  11. this scam of “promising certain results, then adjusting the systems used to measure said results” is one of oldest political con games in the world.

    It is a testament to how badly run most federal programs are that even the new, highly biased systems re-invented in order to confer some sign of ‘progress’ fail to produce much in the way of exciting news.

    if anything, the rejigging of the old measures simply provides for a shallow grave in which to dump the body of ‘egregious failure’.

  12. OT: I think I’ll call him “bitey”. — Homer J Simpson

    1. You’re not you when you are hungry.

      1. So Suarez should have had a pre-match meal?

        1. I think that will be the next Snickers commercial.

  13. Yet earlier this month, the CBO, which provided the original estimate and evidence that the law would be deficit neutral, said that it can no longer score the net fiscal effect of the law in its entirety.

    That should be “which provided the original estimate and evidence, on the basis of assumptions provided by the then Democrat controlled congress“. The CBO is required by law to base its reports on whatever assumptions congress tells them to use, no matter how ridiculous. When those ridiculous assumptions lead to ridiculous conclusion, everyone tends to throw the CBO under the bus.

    1. Very true.

  14. Another myth bites the dust:

    Something unprecedented may be unfolding in Florida’s individual health-insurance market: None of the nine companies that have filed their 2015 rate requests so far wants an increase.

    In fact, two of the companies — Molina Healthcare of Florida and Sunshine Health — actually requested a price cut.

    None of the companies will talk about their filings until they are approved; and a spoiler could still pop up in the four days that remain before the deadline for plans to compete in the 2015 enrollment.

    But consumer advocates are cautiously optimistic that this is evidence that the Affordable Care Act is restraining prices for individuals who don’t get coverage through a group.

    “The fact is, an overall pattern of insurers not seeking rate increases — and even seeking rate decreases — is unheard of,” said Greg Mellowe, policy director for the consumer advocacy group Florida CHAIN.


    1. The ACA has a provision that makes it impossible for them to lose money next year regardless of what premiums they request.

      Given this incontrovertible fact, the only risk the insurers face is political risk: the risk that they will piss Obama off somehow, and he will use his magical law-reading skills to discover that he doesn’t actually have to make the risk corridor payments the law calls for.

      Given that second incontrovertible fact, the insurers have no incentive to raise premiums and every incentive to obey Obama when he suggests that they keep them the same.

      Given that third incontrovertible fact, we are now governed by the fuhrerprinzip, and you are applauding it. Because you are a cunt.

      1. ^^ this

      2. The provision that rebates insurers for unexpected losses? Don’t be an idiot. Such a provision does not destroy their profit motive.

        I like the fact that you hacks are continuously wrong on so many issues and then I get to rub your noses in it. That makes me an asshole with balls and not a cunt. Go look in the mirror.


        2. So what you are saying is that you believe my objection to the ACA was that it raised premiums?

          1. I don’t know about you but many of the Peanut Gallery here have been predicting mass rate hikes and an ensuing death spiral.

            They will be wrong, of course. The only good thing about the ACA is that it is in fact a market with the competition markets bring.

            1. A ‘market’ where everyone has to make the same basic product, and where the customers are penalized for not buying it. And where ‘competitors’ are made whole if anything ‘unexpected’ happens.

              yay, markets.

              1. Viva markets! Viva Venezuala!

            2. We weren’t predicting *mass rate hikes*.

              We were predicting increased costs for insurance and actual healthcare.

              Keeping the rates the same (or lowering them) and making up the difference with hidden subsidies is not a *good* thing.

              1. “When politicians promise to ‘bring down costs’ what they actually is mean is that they will prevent the price system from reflecting those costs”

                Paraphrased from Thomas Sowell

                This is the fundamental problem-the costs can’t go away, they have to be absorbed. Prog morons always confuse price with cost, the price is just info…that is all. You cannot get rid of costs, you can only move it around; shortages, rationing, longer wait times etc-all of these things have a foregone monetary value. “Bringing down the cost” is complete bullshit. Only competition and innovation can do that.

            3. Of course you fail to mention that many Obamacare plans have very narrow options that limit what the insurance companies have to pay, or that deductibles are a lot higher than they were before Obamacare. So of course there haven’t been massive rate hikes, they just cut benefits.

        3. Whoever you are, you’re persistent. Since Obamacare is so hard to measure how can I truly know how awesome it is? I can’t. Sometimes it feels like counting the stars in the sky…

        4. Such a provision does not destroy their profit motive.

          The law already destroyed their profit motive.

          Their revenue motive, not so much.

        5. Burn that straw man shit for brains.

        6. The provision that rebates insurers for unexpected losses? Don’t be an idiot. Such a provision does not destroy their profit motive.

          Handouts never destroy profit motive(s), they prey on them. As Fluffy pointed out they (falsely) distort risk.

          Whether the it’s a pyramid scheme or socialism, that’s how it works. You provide short-term capital/profits in order to obfuscate long-term risk.

          Nobody goes into communism thinking, “I want to put 20 million people in the gulags, starve another 7.5 million, and rocket the price of moldy bread for everyone else to $25+/loaf.” they see the 5-yr.-plan and think, “Yeah, that *could* work.”

          Well… almost nobody.

        7. I like the fact that you are such a ham fisted and obvious Obama fellator?..and that I’m seeing a marked decline in the amount of CHRISTFAG that you have been bringing to the table recently.


    2. Why raise prices again when you already have?

      You need to start using more than 85 of your brain.

      1. It’s 8%?.the percentage the shreeky can generally muster due to the lack of O2 in moms basement is 8%.

        Know it, Learn it, Lov eit.

    3. Ladies and Gentlemen !!

      Step Right Up and See The Butt Plug Unveiled !!

      That Right! The Butt Plug Behind The Mask!

      See The Butt Plug Thrust and Parry in Real Life!! See The Butt Plug !!

      Butt Plug’s True Identity for All The World to See !!

      Only 5c. One Nickel is All It Takes to View the Butt Plug at Work ! No Childrenz Allowed !!


    4. There was an update.

      “Update: Late Tuesday, the state Office of Insurance Regulation said information posted about 2015 individual market health insurance rates was incorrect and has been taken off its website. See more.”

      And here it an updated article. Note the title.


      I’m…… sorry.

      1. This is in response to PBP’s linked article.

        I clicked on the link EXPECTING a correction. I’m not lying.

  15. Several recent decisions have made tracking the health law’s outcomes more difficult.

    Wasn’t that the whole point? Provide quality healthcare to those that could hardly afford it or not at all, and then make it much less obvious that you did?

    1. It is their overwhelming modesty, OM…

      1. The standard talking point I hear is “We’ve added 8mil to the pool of insured and all Repubs are trying to do is take it away from you”

        Which conveniently ignores the fact that at least 5mil lost existing plans.

    2. Several recent decisions have unexpectedly made tracking the health law’s outcomes more difficult.


  16. Measuring this alt-text is super easy.

  17. Start working at home with Google. It’s a great work at home opportunity. Just work for few hours. I earn up to $100 a day. I can’t believe how easy it was once I tried it out http://www.Fox81.com

  18. This is incredibly lazy journalism. The CBO will of course score the budget effects of any proposed legislation, including a full repeal of the ACA, with or without something resembling a “replacement”. That is their job. What they will not do anymore is retrospective analysis – i.e. comparing the budget effects of all or part of the ACA to so-called “prior law”. That is, compared to a hypothetical world in which the ACA was never passed at all.

    In the CBO’s words:

    “A retrospective analysis of the effects of a current law is very different from a cost estimate for proposed legislation, particularly because it requires formulation of a counterfactual benchmark representing what would have happened if the law had not been enacted?a challenging undertaking that is beyond the scope of CBO’s usual analyses.”


    “Consistent with their statutory responsibilities, CBO and JCT can continue to estimate the effects of prospective legislative actions, such as proposals to modify provisions of the ACA or to repeal the law entirely. Because of the complexities involved in implementing a repeal of the ACA, the budgetary effects of repealing the act at this time would not simply be the opposite of the budgetary effects of the ACA itself.”

Please to post comments

Comments are closed.