The Future of Children, Princeton - Brookings: Providing research and analysis to promote effective policies and programs for children.

The first major focus of research regarding the HIPPY program was to assess the effects of participation in HIPPY on children's school functioning: Did participation in HIPPY improve children's cognitive skills, performance on standardized school achievement tests, or adaptation to the classroom? Two cohorts of HIPPY children and control/comparison group children were followed from the beginning of the program until one year after program completion at two HIPPY sites.⁴

Results of the two-cohort randomized experimental study conducted at the New York site are described first.⁴ At the New York site, families were randomly assigned to the HIPPY program or the control group. Cohort I began HIPPY in the winter of 1990, and Cohort II began HIPPY in the fall of 1991. All families—HIPPY and control group—participated in a high-quality, full-day preschool program during the first year of HIPPY and kindergarten during the second year. Thus, this study assessed the impact of HIPPY over and above the impact of the preschool.

Results of the Arkansas quasi-experimental study, in which HIPPY families were compared with families drawn from the community and matched on several key characteristics,¹⁰ are reported last. The results of Cohort I at this site partially replicate the results in New York.

The participating HIPPY program was situated in a large city in New York State. The program became operational in 1989 as a parent involvement component of the city's public early childhood center. The center provided an enriched, high-quality early childhood education setting for prekindergarten and kindergarten students.

All families enrolled in the prekindergarten of the agency sponsoring the HIPPY program were invited to participate in a lottery. For Cohort I, a total of 90 volunteer families (70.5% of all the families in the prekindergarten) were then randomly assigned into HIPPY (n=52) or a control group (n=38). Some families moved away before the program began or dropped out of the program within the first month, leaving 69 families in Cohort I: 37 families in the HIPPY group and 32 families in the control group. For Cohort II, 70 children were assigned to the HIPPY group and 87 to the control group. At the time of program start-up, 113 families remained in Cohort II: 47 in the HIPPY group and 66 in the control group. Thus, a total of 182 families in both cohorts were included in this study. Table 1 presents a description of the sample by group and cohort at pretest (not including families that moved away before the program began or that received less than one month of the program).

As can be seen from Table 1, about two-thirds of the New York families were from ethnic minorities, with primarily African-American and Latino backgrounds. One-third of the families reported that public assistance was their primary source of income. The sample was evenly divided between single-parent and two-parent families. While a substantial proportion of the families had postsecondary education, about 35% of parents did not speak English as their primary language.

Baseline data were collected during home visits conducted by trained research assistants not associated with the HIPPY program. For Cohort I (n=69), baseline data were collected in February 1991, as a result of delays in the beginning of the HIPPY program. Baseline data collection for Cohort II (n=113) took place at the beginning of the next school year (September 1991), which coincided with the beginning of the first year of the HIPPY program. Thus, while both baseline data collection sessions occurred at the beginning of the HIPPY program, Cohort I children were older at their baseline than Cohort II children were at their baseline.¹¹ This lack of comparability in the timing of baseline data collection necessitated that outcome analyses be conducted separately for each cohort.

To determine whether control group and HIPPY families were equivalent at baseline and to control for any preexisting differences between the two groups, information about children's cognitive skills and family backgrounds was collected. Children's cognitive skills were assessed by the Cooperative Preschool Inventory (CPI),¹² a 64-item, individually administered assessment of preschoolers' cognitive achievement, which has been used extensively with low-income populations in preschool intervention evaluations.¹³

The National Evaluation Information System,¹⁴ a comprehensive family questionnaire, was used to gather information about family characteristics through parent self-reports. Information was collected regarding the children's ages and genders, adults' ethnicity and level of education, family structure, and household sources of income.

Children's cognitive skills were assessed at two points in time and with several measures. The CPI was administered at program completion, when children were approximately six years of age and completing kindergarten (n=153). Kindergarten and first-grade standardized achievement data (n=144) were collected from school records using scores on the Metropolitan Readiness Test in kindergarten (1976 edition) and the Metropolitan Achievement Test in first grade (fifth edition). Both tests are group-administered assessments of children's mastery of school curricula. The tests are divided into instructional subtests that measure facts, skills, and concepts and their applications in language, reading, and mathematics.

Children's classroom adaptation—their interest in learning and their behavioral and motivational readiness to learn—was assessed by teacher ratings on the Child Classroom Adaptation Index (CCAI)¹⁵ one month after the beginning of the first (n=146) and second grades (n=152). Teacher ratings have been shown to be a reliable and valid measure of children's functioning in the classroom¹⁶ and to be related to children's school performance.¹⁷ The CCAI is an 11-item teacher-report rating scale. Teachers were asked to rate each child on a scale from one (representing poor adaptation) to five (representing very successful adaptation) for factors such as the child's enjoyment of books, listening and paying attention, task orientation, self-direction in learning, seeking and using assistance appropriately, curiosity, and initiative. The items were combined into a single index score.¹⁸

Analyses were conducted separately for each cohort, using a statistical technique called analysis of covariance. With this technique, results for the HIPPY and control group families were compared, holding constant children's ages, parents' level of education and ethnicity, family structure, source of family income, and children's baseline scores on the CPI. Results are summarized in Tables 2 and 3.

In Cohort I, HIPPY children outperformed control group children on measures of cognitive skills at the end of kindergarten, on measures of classroom adaptation at the beginning of the first and second grades, and on a standardized reading test at the end of first grade. These differences were both statistically significant and large enough (ranging from about 0.56 to 0.76 in effect size) to be deemed important from an educational point of view.

None of these effects was replicated in Cohort II (see Table 3). The next series of analyses was conducted to try to understand why the results were not replicated in Cohort II.

Analyses were conducted to determine whether Cohorts I and II differed from each other in some way that might explain their different results. For example, had there been differential rates of attrition from the study? Did the program vary? Were families with and without posttest data different from one another in terms of a number of child and family background characteristics?

Unfortunately, none of these possible explanations was supported by the analyses.

Concurrent with the evaluation in New York, an evaluation of HIPPY was conducted in Arkansas, employing similar procedures and measures, with two notable exceptions. First, this was not a randomized trial. Second, none of the 121 HIPPY or 105 comparison group children participated in any other preschool programs during the first year of the HIPPY program, although most children (92%) were enrolled in kindergarten during the second year of HIPPY. Thus, the design of the Arkansas study was less methodologically rigorous than that of the New York study, and it examined HIPPY's effects separately from preschool enrollment. Once again, children's cognitive skills and classroom adaptation were assessed using the CPI and the CCAI, but a different standardized achievement test, the Stanford Early School Achievement Test (second edition), was administered and a new variable—timely movement through the grades—was assessed.¹⁹ An overview of the samples by group and cohort is presented in Table 4.

Despite these community context and design differences, results revealed a pattern of effects somewhat similar to the findings in New York (see Tables 5 and 6), at least for Cohort I in the Arkansas program.

As can be seen from Table 5, there was a trend (though not statistically significant) for HIPPY children to be rated as better adapted to the classroom in first grade than their peers who had not participated in HIPPY. By the beginning of second grade, this effect was statistically significant and large enough to be educationally meaningful in Cohort I (that is, it had an effect size of 0.59). In addition, at the time of the one-year follow-up posttest data collection session, 87% of the HIPPY children were in first grade, compared with only 69% of the comparison group–children in the comparison group were more likely to have been retained in kindergarten or enrolled in school a year behind their peers. However, there were no effects in Cohort I for the CPI or for standardized achievement.

In contrast, in Cohort II, the control group outperformed the HIPPY group on school readiness and standardized achievement at the end of kindergarten. There were no other significant group differences in Cohort II on any measure or at any posttest.

As was the case for New York, subsequent analyses could not account for the differences in results between the two cohorts in Arkansas.

Findings from Cohort I in New York indicated that children who had participated in the HIPPY program scored higher on important measures of school success than children in the control group, over and above the effects of a high-quality preschool program. There was some confirmation of the positive effects of HIPPY from the findings from Cohort I in Arkansas. Considering only Cohort I, results in both sites suggested that the HIPPY children had a more successful entry into elementary school, with better skills and better performance, and with higher assessments from their teachers. These findings are promising because children who perform well as they begin their school careers tend to continue to do so, while children who have poor starts tend to continue to do poorly in school.^20,21

Nevertheless, conclusions regarding HIPPY's effectiveness must be tempered, as these findings were not replicated in Cohort II in either site, and indeed, in Arkansas, control group children outperformed HIPPY children on two measures. Analyses revealed no differences between cohorts or in the program delivery that would explain the failure to replicate the results. These mixed results demonstrate the importance of replication studies and why caution is warranted before generalizing positive or negative results from single-sample, single-site evaluations.

Clearly, one important area in which the two cohorts might have differed is in their level and type of involvement in the HIPPY program. The next section summarizes the results of extensive qualitative research conducted on HIPPY regarding variations in parental involvement in the program. This research supports the idea that differing levels of involvement in the program may well influence outcomes.

Although attempts were made throughout the Arkansas and New York studies to gather systematic, objective, and valid indices of the level of each family's involvement in the HIPPY program, it was not possible to gather such data for a variety of programmatic and logistical reasons. How many home visits each parent received, how many group meetings they attended, and how many parent-child daily lessons they worked on could not be determined from available records. Without such data, it was not possible to determine whether the cohorts with positive outcomes were those that were more involved in the HIPPY program. Nevertheless, the qualitative research concerning parental involvement presented below indicates that such an idea might be plausible.