Psychometric Analyses of Child Outcome Measures with American Indian and Alaska Native Preschoolers: Initial Evidence from AI/AN FACES 2015

The purpose of this technical report is to present findings from analyses of how preschool cognitive and social-emotional measures performed in AI/AN FACES 2015. We examined the internal consistency of measures when administered to AI/AN children, reviewed descriptive statistics as context of difference in mean ability across groups in the AI/AN FACES 2015 and FACES 2014 samples, conducted analyses of differential item functioning (DIF) within cognitive measures to compare the performance of AI/AN children and White children (including data from FACES 2014), and examined the strength of bivariate correlations between measures of similar constructs and different constructs to assess evidence of concurrent and discriminant validity. The findings, therefore, provide initial evidence on the reliability and validity of the measures for AI/AN preschoolers.

For most of the measures, findings from these analyses suggest that it is appropriate to report the AI/AN FACES 2015 preschool child outcomes scores, the exception being one of the two measures of executive function (Heads-Toes-Knees-Shoulders or HTKS, which was added to AI/AN FACES 2015 to expand measurement of this construct beyond what is used in FACES 2014).

• All measures demonstrated acceptable reliability with alphas of 0.70 or above.
• The strength of correlations between measures is in an expected pattern. Correlations are stronger between measures of similar constructs (for example, receptive and expressive language) than between different constructs (for example, social behavior and language).
• Among six cognitive measures flagged across reviews, none warrant additional follow-up based on the DIF analyses. Most cognitive measures did not show evidence of performing differently across groups based on DIF analysis. Two cognitive measures (Peabody Picture Vocabulary Test-Fourth Edition and Expressive One-Word Picture Vocabulary Test-Fourth Edition) had items demonstrating DIF; however, the number of items with DIF was close to or less than the number we would expect by chance and were balanced overall with some easier for AI/AN children and others easier for White children.
• None of the teacher- and assessor-reported social-emotional measures exhibited performance concerns based on the current review.
• Examination of the executive function measures indicated that the pencil tapping task is an appropriate measure for this sample. However, a floor problem was found with the HTKS, indicating the measure provided limited information to distinguish the children in this sample.

These analyses are based on a specific sample of children—AI/AN children in Head Start programs operated by federally recognized tribes. While this information provides initial evidence of the reliability and validity for these measures of cognitive and social-emotional development, researchers should keep in mind the diversity of tribal communities and the AI/AN population nationwide and in Head Start more generally as compared to Region XI AI/AN Head Start when considering the use of these measures with other AI/AN children.

The AI/AN FACES 2015 sample provides information about Region XI Head Start children, their families, classrooms, centers, and programs. We selected a sample of Region XI Head Start programs from the 2012-2013 Head Start Program Information Report, selecting one to two centers per program and two to four classrooms per center. Within each classroom, all children (both AI/AN and non-AI/AN) were selected for the study. Twenty-one programs, 37 centers, 73 classrooms, and 1,049 children participated in the study.

The FACES 2014 sample provides information at the national level about Head Start programs, centers, classrooms, and the children and families they serve. We selected a sample of Head Start programs from the 2012-2013 Head Start Program Information Report, with two centers per program and two classrooms per center selected for participation. Within each classroom, we randomly selected 12 children for the study. One-hundred seventy-six programs, 346 centers, 667 classrooms, and 2,206 children (in 60 programs) were still study participants in spring 2015.

L. Malone, S. Bernstein, S. Atkins-Burnett, and Y. Xue (2018). Psychometric Analyses of Child Outcome Measures with American Indian and Alaska Native Preschoolers: Initial Evidence from AI/AN FACES 2015. OPRE Report 2018-21, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.