Health Profession Opportunity Grants Research and Evaluation Snapshot

ACF Research and Evaluation Agenda

Publication Date: December 30, 2020
Health Profession Opportunity Grants Research and Evaluation

Download Report

Download PDF (926.41 KB)
  • File Size: 926.41 KB
  • Pages: 6
  • Published: 2021

Overview

The ACF Research and Evaluation Agenda covers research and evaluation activities and plans in Health Profession Opportunity Grants, as well all eight other ACF program areas: Adolescent Pregnancy Prevention and Sexual Risk Avoidance, Child Care, Child Support Enforcement, Child Welfare, Head Start, Healthy Marriage and Responsible Fatherhood, Home Visiting, and Welfare and Family Self-Sufficiency. Explore other snapshots and the full agenda >

ACF’s Office of Family Assistance administers the Health Profession Opportunity Grants (HPOG), which provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other individuals with low income for occupations in the health care field that pay well and are expected to experience labor shortages or be in high demand.

In 2010, 32 organizations across 23 states received the first round of HPOG grant awards (HPOG 1.0) to carry out five-year programs in their areas. In 2015, ACF awarded a second round of HPOG grants (HPOG 2.0) to 32 organizations located across 21 states for a new five-year period. This second round of grant awards has since been extended an additional 12 months, ending September 2021. Each round of grants included five Tribal grantees. HPOG target skills and competencies demanded by the healthcare industry; support clearly defined career pathways; result in an employer- or industry-recognized certificate or degree; combine supportive services with education and training services to help participants overcome barriers to employment; and provide services at times and locations that are easily accessible to targeted populations.

HPOG, an $85 million discretionary grant program, is authorized by the Patient Protection and Affordable Care Act (ACA) of 2010 and extended by subsequent legislation. ACF supports a number of research and evaluation activities as well as learning from a broad array of other activities such as performance management, technical assistance, stakeholder engagement, site monitoring, and continuous quality improvement.

Past Research and Evaluation

There is a large body of research on education and training interventions for individuals with low income which seek to build individuals’ skills and/or human capital to help them qualify for higher paying or more stable jobs. Among the earliest studies was ACF’s National Evaluation of Welfare-to-Work Strategies (1989-2002), which directly compared two approaches—mandatory education-or-training-first and mandatory job-search-first. It found that both strategies increased individuals’ earnings, but the gains were not long-lasting. However, the training provided under the education-or-training-first approach was primarily basic education, not occupational skills training.[1] Since then, the field has shifted to focusing on the provision of occupational skills training aligned with local employer demand. In particular, the career pathways framework has attracted increasing attention as a promising approach to post-secondary education and training for adults with low income and educational attainment. Its central feature is instruction organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment.

Beginning in 2007, with the launch of the Pathways for Advancing Careers and Education (PACE) project and continuing with the evaluations of the Health Profession Opportunity Grants (HPOG) Program, OPRE has developed a robust portfolio of research on the career pathways approach. OPRE uses a multi-pronged research and evaluation strategy to assess the success of the HPOG Program. The strategy aims to provide information on program implementation, systems change, participant outcomes, impacts, and costs and benefits.

Promising findings are emerging regarding the effectiveness of HPOG. An experimental impact evaluation of HPOG 1.0 found that in both the short- and intermediate-term (15 months and three years after random assignment, respectively), HPOG produced significant, positive impacts on educational progress, employment in the healthcare sector, and access to employer-provided health insurance. In the intermediate term, HPOG also produced significant, positive impacts on career progress, defined as a combination of training completion and earnings growth and general employment. HPOG produced a small, positive impact on quarterly earnings in the short-term but did not produce an impact on quarterly earnings at the intermediate-term follow-up. Subgroup analyses show that some outcomes are related to participants’ baseline characteristics. For example, at the short-term follow-up, HPOG had larger impacts on educational progress for those who were receiving TANF or Supplemental Nutrition Assistance Program (SNAP)/the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) at baseline, relative to those who were not receiving any public assistance at baseline. In contrast, labor market gains have accrued only to those who had traits at baseline commonly associated with labor market success, such as higher levels of education or fewer barriers to school or work.[2]

An analysis of HPOG 1.0’s long-term impacts, assessed at six years after random assignment, is underway. Evaluations of the HPOG 2.0 grants are also in progress, including an impact evaluation, descriptive evaluation (which includes implementation, outcomes, and systems studies), and cost-benefit analysis of the non-tribal grantees; and an implementation and outcomes study of the tribal grantees.


[1] Fishman, M., et al. (2020). Employment and Training Programs Serving Low-Income Populations: Next Steps for Research (OPRE Report #2020-72). Washington, D.C.: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. /opre/resource/employment-and-training-programs-serving-low-income-populations-next-steps-for-research
[2] Peck, L. R., et al. (2019). Health Profession Opportunity Grants (HPOG 1.0) Impact Study: Three-Year Impacts Report (OPRE Report #2019-114). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. /opre/resource/health-profession-opportunity-grants-hpog-10-impact-study-three-year-impacts-report

Research and Evaluation Stakeholders

In setting HPOG research and evaluation priorities, ACF takes into account legislative requirements and Congressional interests; the interest and needs of ACF, HHS, and Administration leadership; program office staff and leadership; ACF partners; the populations served; researchers; and other stakeholders. ACF routinely interacts with these stakeholders through a variety of engagement activities. These activities inform our ongoing research and evaluation planning processes.

Who

  • State, territory, tribal, local, and non-profit TANF, other human services, and workforce development administrators and staff
  • HPOG technical assistance providers
  • Institutions that operate or partner with HPOG programs including state entities, local Workforce Investment Boards, Indian tribes and tribal organizations, institutions of higher education, tribal colleges and universities, community-based organizations, and employers
  • HPOG program participants and their families
  • Federal partners in HHS and other agencies, such as the Health Resources and Services Administration (HRSA), the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Chief Evaluation Office at the Department of Labor (DOL/CEO), the Office of Disability Employment Policy at the Department of Labor (DOL/ODEP), the Institute of Education Sciences at the Department of Education (ED/IES), the Office of Career, Technical, and Adult Education at the Department of Education (ED/OCTAE), the Food and Nutrition Service at the Department of Agriculture (USDA/FNS), and the Office of Policy Development and Research at the Department of Housing and Urban Development (HUD/PD&R)
  • Researchers and policy experts
  • National organizations, such as the American Public Human Services Association
  • Partners in research and evaluation on education and training interventions for individuals with low income and career pathways programs

How

  • Conferences and meetings, such as the Research and Evaluation Conference on Self-Sufficiency (RECS), the National Association of Welfare Research and Statistics (NAWRS) conference, Association for Public Policy Analysis and Management (APPAM) conference, Society for Research on Educational Effectiveness (SREE) conference, American Evaluation Association (AEA) conference, ACF TANF and Tribal TANF Summits, National Indian Health Board (NIHB) Tribal Public Health Summit, and National Rural Health Association (NRHA) conference
  • HPOG Grantee Annual Meeting and Grantee Roundtables
  • Engagement with TANF technical assistance networks
  • Surveys, focus groups, interviews, and other activities conducted as part of research and evaluation studies
  • Interagency collaborations, such as the Interagency Working Group on Career Pathways; the Federal Employment, Training, and Education Working Group (FETE); and the Adult and Career Education (ACE) Meeting
  • Structured mechanisms for broad stakeholder engagement, such as the Family Self-Sufficiency Research Technical Working Group, the ACF Tribal Advisory Committee, and the Secretary’s Tribal Advisory Council

Examples of Broad Questions

  1. What are the characteristics of and outputs and outcomes for HPOG participants?
  2. How are HPOG programs designed and implemented?
  3. What impacts do HPOG programs have on the outcomes of participants and their families? To what extent do impacts vary across selected subpopulations? What are the impact drivers of HPOG?
  4. To what extent do the education and employment experiences of HPOG participants over time suggest that they are following a career pathway?
  5. What changes to the service delivery system are associated with HPOG program implementation?
  6. Do the benefits resulting from HPOG/PACE programs exceed the costs of those programs?

Examples of Recent and Ongoing Research and Evaluation Activities

 

 

Question 1

Question 2

Question 3

Question 4

Question 5

Question 6

HPOG Implementation, Systems and Outcomes Project

X

X

 

 

 

 

HPOG National Implementation Evaluation

X

X

 

 

X

 

HPOG Impact Study

 

 

X

 

 

 

Evaluation of Tribal HPOG

X

X

 

 

 

 

Pathways for Advancing Careers and Education

 

X

X

 

 

 

Career Pathways Intermediate Outcomes Study

 

 

X

X

 

X

Career Pathways Long-Term Outcomes Study

 

 

X

X

 

X

Career Pathways Secondary Analysis Grants

X

X

 

X

 

 

HPOG 2.0 Evaluation and System Design

X

X

 

X

 

 

HPOG 2.0 National Evaluation

X

X

X

X

X

X

HPOG 2.0 Tribal Evaluation

X

X

 

 

 

 

HPOG University Partnership 2.0 Research Grants

X

X

 

X

 

 

HPOG 1.0 Research and Evaluation Activities

  • HPOG Implementation, Systems and Outcomes Project: provided recommendations for the design of an evaluation to assess implementation, systems change, and outcomes of the non-tribal HPOG 1.0 programs. The project also built and maintained the HPOG Performance Reporting System, a web-based management information system, to track grantee progress for program management and to record grantee and participant data for use in HPOG 1.0 evaluations. (#1) (#2)
  • HPOG National Implementation Evaluation (HPOG NIE): carried out an in-depth examination of the non-tribal HPOG 1.0 programs’ design and implementation, a systems analysis of networks created by the programs, and an analysis of program outputs and outcomes. (#1) (#2) (#5)
  • HPOG Impact Study: used an experimental design to examine the short-term impacts of HPOG 1.0 on participants’ educational and economic outcomes at about 15 months after random assignment. The study included 42 local HPOG programs implemented by 23 non-tribal HPOG 1.0 grantees. In 19 of the 42 programs, the evaluation also conducted a three-armed test to analyze the impacts of three specific program enhancements—emergency assistance, non-cash incentives, and facilitated peer support. Additionally, the evaluation used non-experimental methods to examine the relative contributions of various program characteristics. (#3)
  • Evaluation of Tribal HPOG: conducted a comprehensive implementation and outcomes study of the five Tribal HPOG 1.0 grantees. (#1) (#2)
  • Pathways for Advancing Careers and Education (PACE): conducted implementation and impact evaluations of nine innovative career pathways programs around the country, including three HPOG 1.0 grantees. The project assessed program implementation and short-term impacts at 15-18 months after random assignment. (#2) (#3)
  • Career Pathways Intermediate Outcomes Study: is rigorously evaluating the intermediate impacts (those assessed at about 36 months after random assignment) of the programs first studied in the HPOG Impact Study and PACE project on participants’ educational progress, labor market outcomes, and family well-being. The study will also conduct cost benefit analyses for selected PACE programs. (#3) (#4) (#6)
  • Career Pathways Long-Term Outcomes Study: is rigorously evaluating the long-term impacts (those assessed at about 72 months after random assignment) of the programs first studied in the HPOG Impact Study and PACE project on participants’ educational progress, labor market outcomes, and family well-being. (#3) (#4) (#6)
  • Career Pathways Secondary Analysis Grants: are supporting rigorous, policy-relevant secondary analysis of data collected through PACE, the HPOG NIE, and the HPOG Impact Study to add to the body of knowledge and gain a deeper understanding of the implementation and effectiveness of career pathways programs. (#1) (#2) (#4)

HPOG 2.0 Research and Evaluation Activities

  • HPOG 2.0 Evaluation and System Design: provided recommendations for the design of an evaluation to assess the implementation, outcomes, systems change, and impacts of HPOG 2.0 programs. The project also built and provides ongoing support for the HPOG Participant Accomplishment and Grant Evaluation System (PAGES), a web-based management information system, to track grantee progress for program management and to record grantee and participant data for use in HPOG 2.0 evaluations. The project uses PAGES data to develop annual reports of program and participant characteristics and outcomes. (#1) (#2) (#4)
  • HPOG 2.0 National Evaluation: is a rigorous evaluation of the non-tribal HPOG 2.0 programs that includes three key components: (1) an impact evaluation, which is using an experimental design to assess the impacts of the HPOG 2.0 programs; (2) a descriptive evaluation, which includes implementation, systems change, and outcomes studies; and (3) a cost-benefit analysis to assess the costs and benefits of a standard HPOG 2.0 program. (#1) (#2) (#3) (#4) (#5) (#6)
  • HPOG 2.0 Tribal Evaluation: is a comprehensive implementation and outcomes study of the Tribal HPOG 2.0 programs. The evaluation is grounded in culturally appropriate practices and using a mixed-methods approach to answer research questions focused on the Tribal HPOG 2.0 programs’ structure, processes, and outcomes. (#1) (#2)
  • HPOG University Partnership 2.0 Research Grants: are funding university research teams that partner with HPOG 2.0 program grantees to conduct research and evaluation studies focused on questions relevant to HPOG program goals and objectives and that benefit the broader employment and self-sufficiency research field. (#1) (#2) (#4)

Future Directions for Research and Evaluation

The broad questions listed above will continue to drive much of ACF’s research and evaluation activity in this area. Future activities will also be informed by emerging findings from ongoing research and evaluation activities, other learning activities, and continued engagement with HPOG stakeholders.

Examples of activities planned for the next few years include:

  • Analyzing the long-term impacts of PACE and HPOG 1.0 programs on participants’ educational progress, labor market outcomes, and family well-being
  • Analyzing the short-, intermediate-, and long-term impacts of HPOG 2.0 programs on participants’ educational progress, labor market outcomes, and family well-being
  • Analyzing HPOG 2.0 programs’ implementation, systems change, and participant outcomes
  • Analyzing participants’ perspectives on HPOG 2.0 programs, healthcare training, support services, and career pathways
  • Assessing the costs and benefits of a standard HPOG 2.0 program
  • Synthesizing findings across programs participating in PACE and/or HPOG evaluations
  • Continued secondary analysis of data collected through PACE and previous and currently ongoing HPOG evaluations
  • Collecting information on how the COVID-19 pandemic and recovery affect HPOG 2.0 program implementation and service delivery, and individuals’ participation in healthcare training and employment
  • Analyzing secondary and publicly available data to assess how COVID-19 has changed the demand for healthcare workers and their wages more broadly