The Future of Children, Princeton - Brookings: Providing research and analysis to promote effective policies and programs for children.

In the studies in this journal issue, the evaluators typically randomly assigned families to groups, and then compared the results of the groups to see if the families receiving home visiting services outperformed the families in the comparison groups. In many studies, no or only a few overall differences were seen.

For example, in the Elmira study of nurse home visitation, when comparing all the participants in the home visitation group with all the participants in the comparison group, Olds and colleagues found no differences in measures such as child development or birth outcomes. However, striking differences were found when the analyses were restricted to unmarried, low-income women. For some analyses, Olds and colleagues compared the unmarried, low-income women in the home visitation group with the unmarried, low-income women in the control group and found that home visitation for those families had indeed produced benefits in several domains, including deferral of subsequent pregnancies.

In their report on PAT, Wagner and Clayton employ similar subgroup analyses to assess the differential impact of the program on Caucasian, English-speaking Latino, and Spanish-speaking Latino families. Other articles also report differences among subgroups of the participants.

The risk of this approach is that such analyses can essentially eliminate the benefits of randomization—unless the evaluators plan ahead for these sorts of analyses by randomly assigning families to groups using special techniques to stratify the groups initially. This means that the evaluators would randomize the groups such that the same number of families with each particular characteristic being analyzed would wind up in each group (for example, the same number of unmarried women, English-speaking Latinos, and so on).

Without this initial step, it is possible that the families in the intervention and control subgroups will somehow differ from one another in unmeasured ways, and that any differences that are found will really be a reflection of those underlying differences, and not of their exposure to home visitation. Researchers often use statistical techniques to try to equalize the groups, but there is no guarantee that the techniques will be successful.

The most conservative approach, therefore, is to treat subgroup analyses that were not part of the initial analytic plan for an evaluation as preliminary findings. It is quite plausible that a given home visiting program will be more effective with some families than with others, but researchers should seek to build upon the subgroup findings from one study to see if they appear in subsequent, more carefully controlled studies. Olds and colleagues report on a line of research that follows this approach.