Catholic Schools: Private and Social Effects

Kluwer Academic Publishers, 2000, $100; 160 pages

By William Sander

The Education Gap: Vouchers and Urban Schools

Brookings Institution, 2002, $28.95; 275 pages

By William G. Howell and Paul Peterson, with Patrick J. Wolf and David E. Campbell

The advantage of reading The Education Gap and Catholic Schools together is in being able to appreciate their use of diverging research strategies. At the heart of The Education Gap is a large-scale study of privately funded school-voucher systems in three cities, New York, Washington, D.C., and Dayton, Ohio. The authors, Paul Peterson [editor-in-chief] of Education Next] and William Howell, mounted a randomized field trial like those used in medicine in order to rest the effects on achievement of being given a voucher to attend private school, Catholic Schools, by contrast, uses a nonexperimental approach in studying the influence of Catholic schools. The strengths of the experimental approach compensate for the weaknesses of the nonexperimenral approach and vice versa. When both types of studies yield similar conclusions, the results inspire greater confidence.

In The Education Gap, Howell and Peterson call randomized field trials the “gold standard” of social-science research. Randomized trials, built on the model of medical experiments, allow researchers to estimate the effects of a policy change by randomly sorting individuals into two comparable groups–in the case of school vouchers, a test group that receives vouchers and a control group that doesn’t. The random allocation ensures that there are no systematic differences–such as income, achievement levels, or parental involvement–between the two groups that might influence the results. Despite their advantages, randomized trials have weaknesses, the most basic being that they do not tell the researcher whether the results may be generalized to other situations, to nonvolunteers, or how the treatment variable will interact with other policy-relevant variables.

Randomized field trials are so expensive that they often must be limited to only a subset of the population of interest. For example, the study might include only a few locations and only a few grades. Even if the outcomes of the treatment and control groups differ significantly, we cannot be sure that the results will generalize to other locations and to other grades. If the experiment was limited to a single income group, we cannot know if the treatment would work equally well for other income groups. Despite these limitations, however, randomized field trials are an extremely useful type of evaluation, and they have an exceptional level of internal validity. We generally can be more confident in conclusions reached by randomized field trials than those produced by nonexperimental research as long as we do not generalize beyond the subgroups studied.

Most social-science research cannot use randomized field trials because of their expense and because it is rarely feasible to assign a sample population to treatment and control groups. Social-science researchers therefore typically observe relationships in nonexperimental settings and attempt to adjust statistically for all the relevant variables, For example, if we are interested in whether private school students and public school students have different outcomes, we try to obtain good measures for all other variables that influence the outcomes and then use regression analysis to discover whether private school students did better than the public school students. In Catholic Schools: Private and Social Effects, William Sander uses this method to study the effects of attending a Catholic school on various academic and nonacademic outcomes.

Nonexperimental research in education has two important methodological problems to overcome; omitted variables and selection bias. Selection bias is a special case of the omitted variable problem. Assume for a moment that two families live next door to each other, The parents have the same occupations, educations, and incomes, and the children have similar abilities. One family pays private school tuition, while the other enrolls its children in public schools. If we observe that the children in the private school learn more, can we conclude that private schools are better? No, because the families obviously have different values and goals. It is likely that the parents paying tuition have other characteristics that encourage their children to value education highly and to work hard in school. Statistical comparisons of public and private school outcomes must find a way to account for these unobserved, but very important, characteristics.

Randomized field trials essentially eliminate the problem of selection bias. At the same time, nonexperimental research can use large sample sizes to test the effects of Catholic schools in a wide variety of situations and to test for interactions between attending a private school and such variables as race, class, religion, and ability level. That these diverging research strategies have common findings lends them serious credibility. The most important common finding was that attending a private school significantly improves the education outcomes of African-American children in the inner cities but not of other students.