Mixed Choice

Another study evaluating Milwaukee's school choice program, which provides scholarships for approximately 1,300 low-income students to attend secular private schools, gives the program mixed grades. In the study, Cecilia Elena Rouse, assistant professor of economics and public policy at Princeton University, finds that the program has produced "quite large" improvements in students' math test scores. However, she finds no statistically significant improvement in reading scores.

Previously, a University of Wisconsin study had concluded that Milwaukee's program had no effect on math or reading test scores. That finding was contradicted last summer by researchers from Harvard University and the University of Houston who found that the choice program had improved both math and reading test scores. (See "Choice Cuts," Dec.)

All three studies relied on the same data, so how did they come to different conclusions? The data are "messy," says Rouse. They cover only a small number of students over a brief period of time, and much of the information is missing. So seemingly trivial differences in interpretation and methodology have a big impact on the final results.

One example: Researchers must infer students' grade levels from test scores. If a student misses a test one year, researchers must "impute" the student's grade level by looking forward to future test scores, or back to previous tests. But the direction from which researchers impute can determine whether the findings show students improved by a statistically significant amount, or not at all.

Nevertheless, Rouse concludes that the program appears to be a qualified success: "I find that the Milwaukee Parental Choice Program and the participating private schools likely increased math scores by 1.5 to two percentage points a year….When I total the math and reading scores, I estimate that private students gained approximately 1.3 percentage points a year."