The Congressional Budget Office's regular reports on the effects of the American Recovery and Reinvestment Act—otherwise known as the economic stimulus package—have been a point of pride for the administration and its defenders. Last December, for example, President Obama bragged that ARRA had created or saved "up to 1.6 million jobs," citing CBO reports to back up his assertion. And it's true that, in these legally required analyses, that's what the CBO has estimated.
But if the stimulus had failed to create jobs, would the CBO's reports have picked up on this? Probably not.
See, the CBO doesn't actually count jobs created. Instead, it uses models that assume that putting taxpayer money into the system results in additional demand, additional spending, and, consequently, additional jobs. Before the stimulus passed, it used these models to predict that the stimulus would create jobs. And now, in analyzing its effects, it's using those same models to estimate that it has created jobs. But because the CBO relies on slightly updated versions of the same, original models throughout the process, it wouldn't necessarily detect the fact that the stimulus didn't work if that were the case.
Stimulus-boosters have basically ignored this. But the CBO, to its credit, has been fairly forthcoming about its methods and their limitations. In response to a question at a speech earlier this month, CBO director Doug Elmendorf laid out the CBO's methodology pretty clearly, describing the his office's frequent, legally-required stimulus reports as "repeating the same exercises we [aleady] did rather than an independent check on it." CBO tweaks its models on the input side, he says—adjusting, for example, how much money the government has spent. But the results the CBO reports—like the job creation figures—are simply a function of the inputs it records, not real-world counts.
Following up, the questioner asks for clarification: "If the stimulus bill did not do what it was originally forecast to do, then that would not have been detected by the subsequent analysis, right?" Elmendorf's response? "That's right. That's right."