Introduction
Research Questions
- Which criteria should evaluators use when selecting and testing measures of self-regulation skills in the context of impact evaluations of employment programs that serve low-income populations?
- To what extent do the measures selected for the Evaluation of Employment Coaching for TANF and Related Populations consistently capture what they were designed to measure?
People’s ability to find, keep, and advance in a job depends on self-regulation skills in addition to education, work experience, and technical skills (Almlund et al. 2011). Self-regulation skills include the ability to finish tasks, stay organized, and intentionally control emotions and behaviors. Research has shown that these skills are essential to attaining goals and determining life outcomes, including those related to employment (Almlund et al. 2011). Research has also shown that interventions can both strengthen self-regulation skills and encourage their use (Kautz et al. 2014).
In response to this research, some employment programs, including some that are offered as part of the Temporary Assistance for Needy Families (TANF) program, use coaching and other strategies designed to strengthen and boost participants’ use of self-regulation skills (Cavadel et al. 2016; Kautz et al. 2014). To assess the effectiveness of these strategies, evaluators need a way to measure self-regulation skills accurately. Accurate measurement is important not only for understanding the results of individual studies, but for the ability to synthesize findings across studies and across developmental age groups (Murray and Rosanbalm, 2017).
The selection and testing of measures of self-regulation skills for evaluating the impact of such programs present unique challenges and considerations. To provide guidance on how to address these challenges and considerations, this report describes a practical approach for selecting and testing measures of self-regulation skills for use in impact evaluations. It complements three earlier works: (1) Cavadel et al. (2018) (PDF), which encourages practitioners to consider measuring outcomes related to self-regulation skills; (2) Kautz and Moore (2018), which provides guidance to researchers on how to measure self-regulation skills in evaluation settings; and (3) the Self-Regulation and Toxic Stress Series (Hamoudi et al. 2014; Murray et al. 2014, 2015, 2016), which presents information on the relationship between stress and self-regulation, as well as evidence and implications for interventions designed to improve self-regulation among children and youth. This report describes the experience of implementing this approach for the project under which this report was developed, provides empirical findings from tests of the measures selected for this study, describes how the results guided decisions about the use of the measures in this study, and discusses how these decisions could differ for studies that use measures with different empirical findings.
Purpose
This report discusses issues related to selecting and testing measures of self-regulation skills in evaluations of employment programs for low-income populations. First, it presents an overview of criteria for selecting measures of self-regulation skills. Second, through a presentation of empirical evidence, this report demonstrates a process for developing and testing self-regulation measures in the context of an impact evaluation of employment coaching programs for low-income populations. Third, it discusses how the process could be adapted to other studies.
Key Findings and Highlights
To meet the challenges that are involved in developing and testing measures of self-regulation skills in evaluations of employment programs for low-income populations, this report recommends:
- Basing self-regulation measures on existing measures as much as possible;
- Selecting a subset of items from existing measures to reduce burden;
- Using a set of both general measures of self-regulation as well as measures specific to the employment context;
- Pre-testing measures to ensure that they are appropriate given the background of respondents; and
- Assessing the reliability and validity of the measures but considering carefully whether standard criteria for acceptable levels apply.
This evaluation met these criteria by selecting and developing appropriate questions for a baseline survey that was administered when participants enrolled in the study and follow-up surveys that will collect data on participants’ outcomes.
This report provides empirical findings from tests of the measures selected for the baseline survey and describes how the results guided decisions about how to use the measures in the impact evaluation. These results demonstrate the feasibility of developing reliable and valid measures of self-regulation skills for use in impact evaluations of employment programs for low-income populations. Findings include:
- Both the baseline survey as a whole and each individual item met the criteria for adequate levels of response rates
- The self-regulation measures captured distinct skills (exhibited discriminant validity)
- The items designed to measure a given skill consistently measured that skill (were reliable)
- The grouping of items into different self-regulation measures fit the data well overall, suggesting that the items were grouped with the right measures (exhibited model validity)
- The self-regulation measures were correlated with contemporaneously measured employment outcomes as well as with perceptions of potential employment challenges (exhibited concurrent validity)
Methods
The report includes:
- A brief review of self-regulation skills and criteria for selecting corresponding measures in evaluations of employment programs for low-income populations
- Tests of the selected measures, including:
- A nonresponse analysis that revealed the extent to which respondents completed items
- A confirmatory factor analysis that assessed the extent to which measures captured distinct skills and the grouping of items into skills were appropriate
- An assessment of reliability that shed light on the extent to which a group of items designed to measure a given skill consistently measured that skill
- A correlational analysis that examined the relationship between the measures of self-regulation skills and employment-related variables
- Discussion of methodological approaches that could apply to different types of studies