Program Evaluations and Reports

ACL strives to evaluate programs in an integrated manner combining process, outcome, impact and cost-benefit analysis of evaluation activities. Following the learning agenda approach, adopted in 2018 and incorporating stakeholder feedback, ACL has developed an interim learning agenda which involves annual reviews with each ACL center to support the generation and use of evidence to inform agency strategies and decision making. ACL’s evaluation policy  confirms our commitment to conducting evaluations and to using evidence from evaluations to inform policy and practice. ACL seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. This policy addresses each of these principles. 

ACL Guidance and Toolkits

ACL Performance Strategy 

ACL’s performance strategy presents a high-level approach to the planning, conduct, and implementation of performance management. This strategy represents ACL’s commitment to providing rigorous, relevant, and transparent performance data highlighting all the programs and initiatives ACL supports. It also reflects ACL’s continuous effort to build and enhance its repository of data and evidence including high quality performance data in support of our mission and vision. The strategy requires close collaboration with various partners, with a vested interest in ACL’s performance including senior leadership, program offices, budget officials, and external stakeholders.

ACL Performance Measure Guidance

ACL’s Performance Measure Guidance introduces the practice of performance measurement, describes indicators, and presents a structured process for developing performance measures. It includes examples of performance measure and resources for performance measure developers. This guide should be used in conjunction with additional resources, including consultation with the Office of Performance & Evaluation (OPE). 

ACL Data Quality Guidance

ACL’s Data Quality Guidance is a comprehensive guide describing the characteristics of high-quality data, its uses and benefits, and recommendations for improving data quality. The guide is designed for all ACL staff who oversee data systems or monitor the collection or reporting of data. In addition to ACL’s Data Quality Guidance, in October 2020 OPE developed an ACL Data Council. The council will support ACL’s mission by improving the coordination of ACL’s data governance, including the development of improved processes and standards for defining, collecting, analyzing, and presenting data ACL collects. The council’s ongoing work will provide additional guidance on these topics.

ACL Logic Model Guidance

ACL’s Logic Model Guidance presents a brief primer on the development and use of logic models for clarifying program theory, demonstrating progress towards objectives, and answering evaluation questions. This primer should be used in conjunction with additional resources, including consultation with ACL’s Office of Performance & Evaluation (OPE).

ACL Strategic Planning Guidance

In 2018, the Office of Performance and Evaluation (OPE) within the Administration for Community Living (ACL) completed a series of planning sessions resulting in the development of a strategic plan that articulates the goals, objectives, and activities OPE will undertake in support of ACL. This Strategic Planning toolkit is designed to support other ACL centers and offices with developing a strategic plan or framework.

ACL Systems Change Measurement Guidance

This brief is intended as a resource for ACL staff who oversee programs focused on focus on improving the systems that serve older adults and individuals with disabilities. It provides an introduction to measuring systems change, including how systems and systems change are defined and how they can be operationalized, as well as how to approach their evaluation. The brief also includes selected examples from ACL evaluations that have focused on systems change and that can provide additional guidance to ACL staff.

Evaluation Planning

ACL’s annual evaluation plans include the evaluation activities the agency plans related to the learning agenda and any other "significant" evaluation, such as those required by statute. Each annual plan describes the systematic collection and analysis of information about the characteristics and outcomes of programs, projects, and processes as a basis for judgments, to improve effectiveness, and/or inform decision-makers about current and future activities. You can view ACL’s 2022 evaluation plan and its 2023 evaluation plan.

Completed Evaluations and Studies

Evaluation of the Value of Volunteerism for Older Americans Act Programs

Volunteers play a key role for the capacity of the Older Americans Act (OAA) Title III programs, the OAA Title VII Long-Term Care Ombudsman Program, and the State Health Insurance Assistance Program to provide support services to older adults (generally age 60 and older). The results of this study show that unpaid volunteers across the three program areas contribute 6–14 hours per month on average. Importantly, volunteers contributed about 56% of the total annual labor in fiscal year 2019 for OAA Title III area agencies on aging, for an estimated annual value of $1.7 billion. Volunteers for the Long-Term Care Ombudsman Program contributed about 15.9% of the total certified ombudsmen annual labor in that year, or roughly $14 million in total estimated annual value. For the State Health Insurance Assistance Program, volunteers contributed about 28.2% of the total annual labor, or roughly $28 million in total estimated annual value. Given fiscal year 2019 federal funding levels for these programs, volunteers represented an essential resource that all these programs depended on to serve clients. For more details, please see the final study report, a report documenting examples of effective volunteer practices, stories from volunteers that highlight their value  to ACL programming, and a short infographic highlighting key findings.

Evaluation of Home and Community-Based Support Services for Older American Indians, Alaskan Natives, and Native Hawaiians (Title VI Programs)

In 2016 ACL began a participatory, culturally responsive evaluation through an implementation and outcomes study of the Title VI Grant Program, which provides home and community-based supportive services for older American Indian, Alaskan Native, and Native Hawaiian populations. The evaluation sought to answer the following questions:

- How do tribes/organizations operate their Title VI Programs?
-  What is the impact of Title VI programs on elders in the community? Are there differences nationally or by tribe/organization?
- Do Title VI programs that are sole-sourced funded have a different impact than programs that are funded through multiple sources?


The Evaluation of the ACL Title VI Programs: Year 1 Interim Report was focused on the approach and design of the evaluation as well as broadly laid out information on evaluation participants, the project timeline, and some initial findings.

The Evaluation of the ACL Title VI Programs: Year 2 Interim Report, focused on understanding the Title VI Programs from the perspective of program staff and elders. The evaluation found that Title VI Evaluation grantees were representative of the overall population of Title VI grantees across several components including elders’ health characteristics, health and nutrition status, and level of services they receive. Further, the evaluation uncovered how important the nutritional, social, and cultural aspects of the Title VI program are for Native elders. 

In the Title VI Evaluation: Year 3 Interim Report, the evaluation found that the Caregiver program is implemented in uneven ways across grantees, an issue that ACL began to address immediately. Emerging themes from this year’s findings were that caregiving is a rewarding but challenging experience, that caregivers do not always see themselves as caregivers, but that the services received through Title VI Part C provide important support to improve caregiver’s overall quality of life.  

Additional attachments include:

- Title VI Evaluation Appendix A – Medicine Wheel
- Title VI Evaluation Appendix B – Methods
- Title VI Evaluation Appendix C – In Their Own Words: Caregiver Quotes
- Title VI Evaluation Appendix D – Focus Group Summary
 

The Title VI Evaluation: Final Report shows that Title VI programs are serving the elders most in need of support. Title VI elders are older, have lower income, and are more likely to have difficulty with ADLs and/IADLs. Elders receiving any Title VI services experienced significantly fewer hospitalizations and falls per year in comparison with elders who did not receive or participate in Title VI services. Further, elders participating in Title VI services participate in 35% more socializations a month and are almost 20% more likely to engage in cultural practices per month than non-Title VI elders.

Additional attachments include:

Title VI Evaluation Final Report Executive Summary

Title VI Final Report Appendices
 

 

Tribal Ombudsman Case Study

The Administration for Community Living (ACL) administers the Long-Term Care Ombudsman Program, which serves individuals living in long-term care facilities, such as nursing homes, board and care, assisted living, and other residential care facilities, and works to resolve problems related to the health, safety, welfare, and rights of residents in these facilities.

In 2019, ACL funded a case study of the Tribal Ombudsman program within the Inter Tribal Council of Arizona (ITCA) to identify successful practices and generate recommendations for improving how Ombudsman programs across the United States serve Native populations.

View the completed study.

Title VI and Title III Grantee Collaboration Study

The Older Americans Act (OAA) requires coordination between Title VI and Title III grantees, but does not specify how grantees should work together and collaborate. To better understand how this requirement is implemented at the Tribal, state, and local levels, ACL funded a study to identify examples and best practices of Title VI and Title III grantee collaboration. The study addressed the following questions:

- How do Title VI and Title III grantees collaborate?
- What are best practices related to Title VI and Title III grantee collaboration?
- What are facilitators and barriers related to Title VI and Title III grantee collaboration?

Results of this study include a full-length report, summary report, and collaboration action document. Title VI and Title III grantees are encouraged to use these tools to initiate or improve collaboration in their communities. 

The Caregiver Outcome Evaluation Study of the National Family Caregiver Support Program

The National Family Caregiver Support Program (NFCSP) represents a significant Federal investment in supporting caregivers who provide care and assistance to aging adults and to grandparents raising grandchildren. Through this program, the Aging Network helps meet the immediate needs of caregivers and care recipients while also being the catalyst for broadening the long-term care (LTC) service systems at State, Territory, local, and Tribal levels to better support families. Its ultimate goal is to support individuals who prefer to age in their own homes and communities—as opposed to institutional settings—through lower-cost, nonmedical services and supports. The National Family Caregiver Support Program, authorized by Title III-E of the Older Americans Act (OAA), provides information to caregivers about available services, assistance to caregivers in accessing supportive services, individual counseling, support groups and caregiver trainings, respite care, and supplemental services.

ACL recently conduced a two-part evaluation of the NFCSP program. The first part, conducted by the Lewin Group, was a process evaluation with the overall purpose of understanding and documenting the strategies used to meet NFCSP goals. This part was completed in March 2016 and produced a briefing of the results (PDF, 2.72MB), an executive summary of the results (PDF, 1.21MB), and a final process evaluation report (PDF, 2.31MB). In addition, State Units on Aging were asked to submit their assessment tools. The evaluation team compiled them and grouped them into the following categories:

Community Assessment Materials (PDF, 14.3MB)

General Customer Satisfaction Survey Materials (PDF, 2.23MB)

Grandparent Assessment Materials (PDF, 1.05MB)

High-Level Administrative Materials (PDF, 3.00MB)

Monitoring Materials (PDF, 3.93MB)

State Caregiver Assessments (PDF, 5.87MB)

State Care Recipient Assessments (PDF, 8.40MB)

Task Force Materials (PDF, 31.1MB)

Uniform Satisfaction Materials (PDF, 3.0MB)

Other Materials (PDF, 9.59MB)

The second part, conducted by Westat, was an outcome evaluation with the purpose of describing NFCSP participants’ demographic and caregiving characteristics, as well as examining the impact of the program on key caregiving outcomes such as burden and caregiving confidence. This part was completed in 2018 and produced a final outcome evaluation report, a webinar briefing, comments and responses from the briefing, and audio or transcript from the briefing (available on request). Appendices on the methodology, data collection instruments, and tabulations are available upon request.

Brief data profiles about NFCSP Outcome Evaluation participants are available. A data profile examining caregivers who have been providing care for two or fewer years, and caregivers providing care for longer than two years, on a set of demographic characteristics is available here.

A data profile examining caregivers of care recipients with Alzheimer’s disease or related dementias (ADRD), and caregivers of care recipients without ADRD, on a set of demographic characteristics is available here.

A data profile examining caregivers that are NFCSP service users and caregivers are not NFCSP service users on a set of demographic characteristics is available here.

A data profile examining caregivers of older veterans, and caregivers of older non-veterans, on a set of demographic characteristics is available here

ACL also conducted a small, retrospective study of caregivers whose care recipient passed away. The resulting supplementary report highlights caregivers’ experiences in providing care and accessing services, and it provides “advice from caregivers to caregivers.”  A factsheet also summarizes key themes and insights from this group of caregivers.

Evaluation of Title III-C Elderly Nutrition Services Program (ENSP)

The Administration for Community Living (ACL) conducted a three-part evaluation of its Title III-C Nutrition Services Program (NSP). Part one is a process evaluation that provides information to support program planning by analyzing program structure, administration, staffing, coordination, and service delivery as well as the interactions between the many levels and types of organizations that provide congregate meals, home-delivered meals, and collateral services under the Title III-C NSP. Part two is a cost study that estimates the costs of program operations, the most important being the cost of the congregate and home-delivered meals provided using Title III funds, and to examine cost variation within the program by cost component and program characteristics. Part three assesses program effectiveness (also called client outcome study), as measured by the program's effects on a variety of important outcomes (including nutrient adequacy, socialization opportunities, and health outcomes and, ultimately, helping older adults avoid institutionalization) through comparing program participants' outcomes with those from a matched comparison group of eligible nonparticipants. The results are available in the Process Evaluation Report and the Cost Study Report a briefing on these reports or if you download the briefing handouts (PDF). Information from the outcome study is available in the Client Outcome Study: Part I,  Client Outcome Study: Part II, briefing handouts as well as responses to questions posed during the briefing for Part 1 of the client outcomes study, and Briefing handouts from the webinar for the Part II report.

Additional issue briefs and reports from this evaluation include:

  • An issue brief that examines the practice of encouraging participants to make contributions, and, for those Local Service Providers (LSP) who do encourage it, the amount of the suggested donation. It also describes the types of LSPs that are more likely to encourage participant contributions, the characteristics of participants served by those LSPs, and the characteristics of participants who make monetary contributions.
  • An issue brief that examines NSP participants’ patterns of health care utilization, including hospital admissions and readmissions, emergency department care, doctor visits, home health episodes, and admissions to long-term care facilities (nursing homes) and skilled nursing facilities over a three year period. Prior reports that examined program impacts (effects) looked at outcomes over a one-year period.
  • An issue brief that assesses how program entrance and retention vary with participant characteristics and evaluates how retention and program entrance have changed over time. These findings are important for understanding how to best target and retain participants who can benefit from services.
  • An issue brief that examines the most common diagnoses underlying adverse health events experienced by NSP participants.
  • An issue brief that assesses the types and prevalence of adverse health events experienced by new home-delivered meal participants; the timing of these events relative to starting to receive home-delivered meals; and how these events varied by participants’ characteristics.
  • An issue brief that assesses prevalence of chronic conditions observed over approximately a 2.5-year period for a sample of nationally representative congregate meal participants.
  • An issue brief that describes the nutritional quality of congregate and home-delivered meals offered through the NSP and examines their compliance with the 2010 Dietary Guidelines for Americans, which were in effect at the time of the data collection.
  • A report that describes NSP participants’ needs, frequency of program use, and participation in other programs and use of services.
  • An issue brief that examines the social activities at congregate meal sites and the role of those activities in improving socialization outcomes for participants.
  • An issue brief that describes the types of foods that NSP participants consumed as part of the program and examines differences between participants and nonparticipants in foods consumed over 24 hours.
  • An issue brief that identifies key food sources of sodium, saturated fat, empty calories, and refined grains in the diets of NSP participants and examines differences between participants and nonparticipants.
Evaluability Assessment of the American Indian, Alaska Natives and Native Hawaiian Programs (Title VI)

In 2014, ICF International (ICF) was contracted by ACL/AoA to conduct the Evaluability Assessment (EA) of the Title VI Grant Program. The purpose of the contract was to provide ACL with:

  • A description of the program model (What do grantees do, and how do they operate?)
  • An assessment of how well defined the programs/program services are (Are the programs/program services stable, distinct, consistent and established?)
  • Information about what federal and tribal stakeholders want to get from an evaluation and how the evaluation data findings would be used.

The study examined the program characteristics of Title VI grantees’ nutritional, supportive, and caregiver support services to assess the feasibility of, and best approaches for, formal evaluation of the Title VI Program. The Evaluability Assessment of the Title VI Grant Program Final Report (PDF, 1.26MB) includes program background, recommendations for enhancing Title VI grantees’ readiness for evaluation, and evaluation design recommendations. A comprehensive program logic model and a medicine wheel model are included.

Evaluation of the Aging and Disability Resource Centers (ADRCs)

There are more than 550 ADRC sites across the nation. Since the inception of this initiative, ADRCs have had more than 4.8 million contacts with consumers, caregivers, providers, and professionals. The focus of the evaluation is to determine the extent to which Aging and Disability Resource Centers (ADRCs) are fulfilling the goal of improving awareness of and access to long term supports and services for older adults and individuals with disabilities and also how well ADRCs are contributing to the overall ACL mission. The process data collection, completed in the fall of 2013, had high response rates at 100% of State-level ADRCs and 84% of local ADRC grantees. The outcome study data collection was completed in the spring of 2014 with more than 600 respondents from 33 sites. Data from a presentation (PDF, 2.23MB) about the process evaluation is available as is the final study report (PDF, 5.85MB) with appendices (PDF, 9.84MB). The final report provides background on the ADRC program, study methodology and results for both the process and outcome studies. The results of this evaluation will influence future performance measures and indicators. This work was completed by IMPAQ International with Abt Associates under contract HHSP233201000692G.

Process Evaluation of the Chronic Disease Self-Management Education Program (CDSME)

In September 2011, AoA awarded Contract HHSP233201100492G to IMPAQ International and Altarum Institute. This process evaluation examined state CDSMP programs funded through Communities Putting Prevention to Work: Chronic Disease Self-Management Program, an initiative of the Administration for Community Living/Administration on Aging (AoA) in collaboration with the Centers for Disease Control and Prevention (CDC) and the Centers for Medicare & Medicaid Services (CMS). The research team employed a multi-method approach using multiple qualitative and quantitative data sources. The program data were also used to conduct regression analyses to examine the influence of various factors on participant completion rates. The report provides information about: characteristics of State grantees, CDSMP participants, CDSMP implementation, participant completion rates, site-level data collection, program sustainability, and recommendations for program improvements. The final process evaluation report (PDF) with Appendices (PDF) is now available. In addition to the process evaluation, ACL is coordinating with CMS’ efforts in evaluating evidence-based wellness programs. A copy of the CMS report which includes CDSMP outcomes is available at innovation.cms.gov (PDF). For additional information about this grant program please also see the reports created by ACL’s technical assistance grantee the National Council on Aging at www.ncoa.org/improve-health/center-for-healthy-aging/capping-reports.html.

Evaluation of the Ombudsman Programs of the Older Americans Act

Real People Real Problems: An Evaluation of the Ombudsman Programs of the Older Americans Act (The National Academies Press)

2004 Survey of Adult Protective Services

2004 Survey of Adult Protective Services: Abuse of Adults 60 Years of Age and Older (PDF) (National Center on Elder Abuse)

Title III-B Supportive Services Evaluation
Developmental Disabilities Act Program Outcomes

State Councils on Developmental Disabilities

Report Title

Year

Individuals Trained by State DD Councils

2012 | 2013 | 2014

Systems Change

2012 | 2013 | 2014

Protection and Advocacy for Individuals with Developmental Disabilities

The Developmental Disabilities Assistance and Bill of Rights Act of 2000 requires Administration on Intellectual and Developmental Disabilities (AIDD) grantees to report annually on progress achieved through advocacy, capacity building, and systemic change activities.

P&A Agencies, using their annual Program Performance Report (PPR), submit data to AIDD. The following tables provide select data for each state/territorial P&A as well as for the nation.

Report Title Year
 Clients Served 2012 | 2013 | 2014 | 2015
 Clients by Age 2012 | 2013 | 2014 | 2015
 Clients by Race and Ethnicity 2012 | 2013 | 2014 | 2015
 Clients’ Living Conditions 2012 | 2013 | 2014 | 2015
 Reasons for Closing Cases 2012 | 2013 | 2014 | 2015
 Intervention Strategies 2012 | 2013 | 2014 | 2015
 Problem Areas/Complaints of Clients Served 2012 | 2013 | 2014 | 2015

University Centers for Excellence in Developmental Disabilities

Report Title

Year

Training Count by Areas of Emphasis

2012 | 2013 | 2014

Technical Assistance Participants

2012 | 2013 | 2014

Performing Research or Evaluation

2012 | 2013 | 2014

Direct Clinical Services / Model Services

2012 | 2013 | 2014

Demonstration Services

2012 | 2013 | 2014

Community Education Participants

2012 | 2013 | 2014

Community Training Participants

2012 | 2013 | 2014

Products Created By UCEDDs

2012 | 2013 | 2014

Ongoing Studies

Process Evaluation of the Aging Network and its Return on Investment

ACL is conducting a process evaluation to study the role of the Aging Network in implementing Older Americans Act programs and ensuring the well-being of older Americans. The evaluation seeks to determine how the Aging Network operates at the local, state, and federal levels, as well as how the various levels of Aging Network work together and how they work with other partners.  Additionally, ACL is interested in learning about the role the Network plays in identifying and responding to emerging needs, how it measures and improves the quality of services, and how it measures success.

The evaluation will also investigate feasible ways to evaluate the Aging Network with regards measuring the value of the activities facilitated via Aging Network in support of individuals, families, and communities, specifically with regards to return on investment.

ACL Rapid Cycle Research and Evaluation

The ACL Rapid Cycle Research and Evaluation contract is designed to study, evaluate, improve, and otherwise examine ACL’s programs and services. This work will be done through a series of relatively quick turnaround research and evaluation projects to help address questions of interest to ACL linked to our Learning Agenda, as well as other areas of inquiry that arise from our internal and external stakeholders.  

Tasks under the contract currently involve work such as reviewing and reanalyzing administrative and application data, designing research and evaluation activities, and small scale data collections. The work being done under this contract is exempt from the Common Rule requirements for Institutional Review Board approval and oversight. See 45 CFR 46.104(d)(5).  ACL will share project abstracts and findings as they become available.

The Social Determinants of Health and ACL

The purpose of this contract is to begin looking at how the services provided by ACL grantees influence the social determinants of health (SDOH). The World Health Organization describes the social determinants of health as the “conditions in which people are born, grow, live, work, and age, including the health system. These circumstances are shaped by the distribution of money, power, and resources at global, national, and local levels, which are in and of themselves influenced by policy choices.” ACL’s programs address these conditions through grants designed to improve organizations and systems, and to mitigate their effects on individuals through the delivery of direct services such as providing nutrition, linking people to services, preventing/addressing violence, health education, mobilizing community partnerships, providing transportation, investing in economic support, social integration, and education, among many others. 

Older Americans Act Fidelity Evaluation

The Fidelity Evaluation is an evaluation of the fidelity with which ACL and its grantees under the Older Americans Act implement the required evidence-based programs. The evaluation will help ACL and our stakeholders understand the following:

  • The process ACL staff use when making awards (i.e., discretionary grants) or verifying grantees (e.g., formula grantees) are using funds for evidence-based programming;
  • How grantees select evidence-based programs they implement and how they determine what grants to which they will apply;
  • How grantees implement those programs and verify they are being implemented with fidelity to their original models, including the identification of intentional and unintentional adaptations;
  • The strengths and weakness of the current award and implementation process; and
  • How and why grantees work with developers of the evidence-based program(s) to identify program adaptations that maintain the integrity of the evidence-based nature of the program.

Upon the completion of the Fidelity Evaluation a report documenting what ACL and its grantees can do to improve the selection, implementation, and monitoring of evidence-based programming. 

The evaluation project began in October 2020 and will end September 2022.

Adult Protective Services Client Outcome Study

The Adult Protective Services (APS) Client Outcome Study is a new three-year data collection that examines if and how APS programs make a difference in the lives of APS clients.

APS programs are provided by state and local governments nationwide and serve older adults and adults with disabilities in need of assistance due to maltreatment, which can include: physical, emotional, and sexual abuse; financial exploitation; neglect; and self-neglect. APS is an important avenue through which maltreatment is reported to law enforcement or other agencies. 

Specifically, the data collection will help examine:

(1) What changes clients report as a result of receiving APS services;
(2) How satisfied clients are with the APS services they receive;
(3) To what extent clients report APS helps them achieve their goals;
(4) To what extent clients report APS supports their right to self-determination;
(5) To what extent APS programs affect client safety (risk of maltreatment);
(6) How APS program intervene to reduce client risk of maltreatment;
(7) What factors help or hinder APS efforts to reduce risk of maltreatment;
(8) To what extent APS programs affect client well-being (e.g., quality of life, financial, physical health, etc.);
(9) How APS programs intervene to improve client-well-being; and
(10) What factors help or hinder APS efforts to improve client well-being.

The data collection will be conducted with three target populations: (1) APS clients, (2) APS caseworkers, and (3) APS leaders. APS leaders will consist of APS state and APS county leaders.

The data collection project began in October 2018 and is expected to end in September 2021.

Living Well (a Project of National Significance)

During the last two decades, the number of individuals with intellectual and developmental disabilities (I/DD) living in community settings and receiving home- and community-based services (HCBS) increased dramatically. While this led to better outcomes, including people with I/DD making more of their own choices and being a part of the community, there are great variations in the accessibility of quality HCBS across the country.

In 2017 and 2018 ACL awarded Living Well grants to help develop and test model approaches for enhancing the quality, effectiveness, and monitoring of home- and community-based services (HCBS) for people with developmental disabilities. These Model Approaches for Living Well grants, awarded as Projects of National Significance by ACL’s Administration on Disabilities, are focused on building the capacity of HCBS systems and enhancing community monitoring to prevent abuse, neglect, and exploitation.

ACL funded a cross-site evaluation of all eight grantees from cohorts one and two in 2018. Each grantee has designed one or more models integrating community monitoring and capacity building with the goal of enhancing health, safety, integration, and independence of individuals with I/DD living in the community. The cross-site evaluation of the Living Well grants focused on eight key features, organized into three categories. The evaluation has produced the first annual report broken into several interrelated products:

Executive Summary
Volume I: Introduction and Grantees
Volume II: Findings
Volume III: Methodology
Volume IV: Conclusion and Next Steps
Living Well State Profiles
Living Well Emerging Practices
Living Well Final Report Tables

In the third year of the evaluation, ACL continued to follow the eight Living Well grantees to understand their work around community monitoring and capacity building in service to promote the health, safety, and independence of their I/DD communities. As with the prior year, the report is broken into several components and this year also includes briefs on areas selected by the Living Well grantees as areas of particular interest – strategies to address COVID-19, Self-Advocate Engagement, and Community Monitoring.

- Executive Summary

- Volume I: Introduction

- Volume II: Grantee Profiles

- Volume III: Cross-Site Analysis and Conclusions

- Volume IV: Methodology

- Living Well State Profiles 

- Brief: Strategies for Responding to the COVID-19 Pandemic 

- Brief: Strategies for Self-Advocate Engagement 

- Brief: Strategies for Improving Community Monitoring 

Supporting Families Community of Practice Evaluation

In 2012, the Administration for Community Living (ACL) funded a National Community of Practice for Supporting Families (CoP) to build capacity across and within states to create policies, practices, and systems to better assist and support families that include a member with intellectual and developmental disabilities (I/DD) across the lifespan. In 2018, ACL funded a retrospective evaluation to inform the Administration on Disabilities (AoD) by providing outcomes data and recommendations to improve supports to families of individuals with I/DD, including potential future application of the Charting the Life Course (CtLC) framework and CoP platform; and provide AoD with an updated systematic description of the range of CoP organizational structures, practices, and development across CoP states, which include consideration of the degree to which CoP states apply the principles and tools of the CtLC framework.

In 2019, ACL collected information regarding changes made to policies, programs and services, and initiatives, as well as the impact or anticipated impact of these changes. These findings are available in the reports:

Evaluation Findings Brief--Year 1 (2018-2019)

National Community of Practice for Supporting Families Evaluation Annual Report 2018-2019

Preliminary Reporting Tool Findings 

and the presentation:

CoPAnnual Meeting: Presentation of Preliminary Reporting Tool Findings

See individual and state-level success stories based on this work:

 

Evaluation of the Long-Term Care Ombudsman Program (LTCOP)

The Final Report for the Process Evaluation of the Long-Term Care Ombudsman Program (LTCOP) is one of seven products that have been developed as part of the process evaluation of the Ombudsman program. Other products include a report on the relationship between the Ombudsman program and the changing landscape of LTSS and five research briefs. The latter focuses on the following topics: (1) understanding the uniqueness of the Ombudsman program in the context of the aging network, (2) resident complaint handling, (3) systems advocacy and organizational placement, (4) use of volunteers, and (5) promising practices.

The LTCOP Challenges Factsheet and Volunteers Factsheet are based on data from the Research Briefs.

Preliminary results were presented at the 2017 Consumer Voice conference Ombudsman Program Process Evaluation: Progress and Initial Data, at the 2017 HCBS conference Process Evaluation and Special Studies Related to the Long-Term Care Ombudsman Program, at the 2018 HCBS conference The Role of Long-Term Care Ombudsman Programs in HCBS: Overview and Findings from the National Evaluation of the LTCOP, and at the 2019 SLTCO conference Findings from the National Evaluation of the Long-Term Care Ombudsman Program. The research team presented preliminary findings about Challenges and Innovations in Systems Advocacy by the Long-Term Care Ombudsman Program at the 2018 Consumer Voice Conference.

In 2017, started work with NORC at the University of Chicago to measure program results. Specifically, this study will determine the efficacy of LTCOP in carrying out core functions as described in the Older Americans Act, the long-term impacts of the LTCOP’s for various stakeholders, what system advocacy among Ombudsman programs looks like, and effective or promising Ombudsman program practices. The final report from this effort is expected in 2022.

Performance of Older Americans Act Programs

ACL collects information and reports on the performance of Older Americans Act programs through the several data collection systems under its National Aging Program Information System (NAPIS), its national surveys of OAA participants, and its evaluation studies. This is reported in the Program Performance Analysis section of the budget justification submitted to Congress along with the President’s budget each year, as well as through several other mechanisms.

Evaluation Design Projects

Evaluation Design for the Long Term Care Ombudsman Program (LTCOP)

In September 2011 the Administration for Community Living (ACL) awarded contract HHSP233201100500G to NORC at the University of Chicago (NORC) to develop an evaluation study design to better understand and assess the effectiveness of Long-Term Care Ombudsman Programs (LTCOPs). The task included building the evidence base on LTCOPs in order to develop recommendations for a rigorous and comprehensive study design that investigates program efficiency and program effectiveness at multiple levels, including the resident/family, facility, local/state/program, and federal levels. Key tasks of the design process involved the development of a family of four logic models and a set of overarching research questions to guide the evaluation, as well as the identification of data collection tools and data sources that inform those questions. The diversity of proposed activities reflects the ACL’s goals for this evaluation, the commitment to a population health frame of reference, and seven critical LTCOP characteristics that influence design options. The final evaluation design report (PDF, 2MB) was completed in January 2013.

Evaluation Design for a Global Evaluation of Older American Act (OAA) Services

Under this project, the Administration on Aging (AoA) seeks to study the impact of OAA programs and services, specifically services provided under Title III, on key outcomes including HCBS use, health care use, community tenure, and long-term services and supports (LTSS) expenditures. OAA services and programs are diverse, often integrated and/or provided in combination with other services, funded through multiple funding streams, and administered and delivered by different state and local-level agencies with varying data collection capacity. This design focuses on the impact of services provided through OAA funded HCBS programs authorized under Titles III-B, C, D, and E of the OAA. These programs include a range of supportive services, nutrition services, health promotion and disease prevention programs, as well as services for family caregivers. While the mix and type of services offered differ by state and locality, the vast majority of OAA funding is used for the provision of nutritional services. Proposed research questions include:

What is the impact of OAA funded HCBS programs and services on:

  • Community tenure

  • Health care utilization

  • Costs of care for older adults (e.g., LTSS, health care costs)

  • Physical, mental, and emotional health and wellness (e.g., preventive measures) of care recipients and caregivers

  • Unmet needs among older adults

  • Caregivers (e.g., strain, burden, depression, health, etc.)

  • Coordination of services (e.g., care management)?

What is the impact of OAA services alone or in combination with services paid for by other sources?

What is the impact of service mix and intensity on outcomes of interest? [if possible, we will isolate OAA services]

What subgroups had the most favorable outcomes? (e.g., health conditions, demographics, functional status)

A primary recommendation from the final evaluation design report (PDF) is that the Administration for Community Living (ACL) first facilitate an exploratory study to determine the feasibility and limitations of conducting this evaluation with state OAA data. Such a study will assist ACL in identifying actions needed to better position the states for participation in a nationwide evaluation.


Last modified on 09/21/2021


Back to Top