The National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) has been proactive in establishing program performance measures, and developing accountability data systems to manage and track grantee progress. It has worked to ensure the relevance of its research portfolio through its competitive grant awards. NIDILRR research staff spends a significant amount of time each year developing research priorities. The aim is to produce research priorities that reflect and build upon state-of-knowledge, and that are relevant to the lives of people with disabilities.
Once completed, each application undergoes a peer review rating process to ensure that only the highest quality work is funded. When the grant is awarded, NIDILRR staff develops a working relationship with each grantee to ensure that agreed upon outcomes are produced.
NIDILRR conducts formative reviews with a subset of grantees that may be at risk. These grantees are provided with an opportunity to discuss the proposed research methodology with experts in their field. Experts provide suggestions for improving the grantees research plan.
Each grantee also reports on the purpose, progress, and accomplishments of all projects/grants in NIDILRR’s Annual Performance Reporting (APR) system.
Annual Performance Report and Data System
During 2012 to 2013, NIDILRR worked to make the many tables generated from APR data available to all NIDILRR managers and staff. All of NIDILRR can now log on to the RTI website and generate data tables based on the most recent data. The website also contains historical data from 2007-2013.
Through a contract with New Editions, NIDILRR produced its first annual report titled, Summary of 2012 Annual Performance Reports from NIDILRR Grantees (2012 APR Report). This report includes an analysis of the data tables available on the RTI web-site. The report includes information about:
- Overall funding, expenditures, and project information for all grantees
- Research projects by domain, research method
- Development projects by domain, development stage
- Training programs by type of audience
- Model systems
- Products produced by grantees, publications, tools, technology, and informational products
- Data on fellows and graduate students funded by NIDILRR grantees, publications produced, disability status, and race/ethnicity; and
- Grantee staffing formation
In 2014, New Editions developed the second enhanced trend analyses for the 2013 APR Report. The 508-compliant report is now available to all NIDILRR constituents.
In addition to making grantee annual performance report data more accessible to our customers, NIDILRR’s data group has been working with New Editions to identify new analysis that can be used by program managers to improve grants monitoring. The data group identified and developed specifications for five new analyses.
NIDILRR Web-Based Reporting System
A contractor-run secure web-based reporting system is used by NIDILRR and our grantees to meet annual and final grant reporting requirements. From a grantee perspective, the reporting system is a multi-part survey containing NIDILRR-developed questions that grantees must answer.
From NIDILRR's perspective, the web-based reporting system is a data repository containing data used to fulfill various agency purposes. NIDILRR staff is able to extract relevant chunks of data (e.g., accomplishments for reports) by running various online reports available in the reporting capabilities section of our web-based reporting system.
How NIDILRR Grantees Use the Web-Based Reporting System
Every April, the web-based reporting system opens for grantees. Grantees are expected to complete their annual report by July 1 every year. Each annual performance reporting year covers the period from July 1 to May 31.
Only grantees with registered credentials, provided by the contractor, can access the web-based reporting system.
Once grantees gain access, completing their online annual performance report requires them to respond to a series of questions that ask them about the following:
- Their inputs (i.e., staff, equipment, money)
- Their activities (i.e., the research, development, training, and knowledge translation project activities they conduct)
- Their outputs that result from the activities they perform (i.e., publications, tools, technology-related outputs, informational products like fact sheets)
- Any evidence of use and adoption of these outputs by various stakeholders
- Additional information, including noteworthy accomplishments, they wish to highlight
Within 90 days of a grant's end-date, grantees are required to complete the final performance report module, which is also part of our web-based reporting system. The final performance report module asks grantees summary questions intended to capture the big picture results that were produced during the grantee performance period. Examples of summary questions include, but are not limited to:
- For each research and development project conducted during the entire course of this grant award, what are the key findings or discoveries that resulted from it? Please respond using a bulleted format. List the findings or discoveries under the title of each research or development project that generated them.
- For all other types of projects (e.g., Knowledge Translation, Capacity Building, etc.), what are the key contributions of this award? Please respond using a bulleted format. List the key contributions under the title of each Knowledge Translation, Capacity-Building, Training, or Other Project conducted.
- Looking over all of the projects conducted and the outputs produced during the course of the entire grant, what outcomes (changes in learning or knowledge or policy, practice, behavior, or systems capacity) can you say your work helped to bring about? Outcomes do not happen on their own. Therefore, it is important to show: (1) how the research, development, capacity-building, training, and knowledge translation projects carried out relate to the outputs and (2) how the documented use of these outputs by intended audiences resulted in the occurrence of the type of outcome you are claiming.
How NIDILRR Uses the Web-Based Reporting System
NIDILRR staff use the data in the APR for a variety of purposes including, but not limited to:
- Making grant continuation funding decisions
- Tracking progress made on grant activities
- Tracking the production of outputs
- Tracking how the produced outputs are being used and adopted by various stakeholders
- Creating various reports such as the Annual Report to Congress and the briefer Annual Organizational Highlights Report
Need help using the system? Email NIDRRAPR@rti.org or call (919) 541-7083, 8:30 a.m.–5:00 p.m. ET.
Need help responding to the survey questions in the system? Email william.schutz@acl.hhs.gov.
Have a business operations or contract question about the system? Email mary.darnell@acl.hhs.gov.
Performance management: GPRA and Long-Range Plan Measures
NIDILRR has continued to make progress in meeting GPRA performance targets and moving toward providing more current performance data. NIDILRR has proposed reducing the number of measures for 2014 and will forward this proposal to the Budget Office.
In FY 2012 - FY 2013 NIDILRR developed a set of Long Range Plan (LRP) measures which track the progress being made on the objectives of the LRP. These measures are based upon existing data collections, although NIDILRR has asked Abt. Associates, who are developing a 10-year evaluation plan, to propose additional measures and data collection tools where none currently exist.
The LRP measures will be tracked and reviewed by senior management to determine if progress is being made. A subset of these measures will be reported on NIDILRR’s website and in subsequent LRPs.
Evaluation
NIDILRR conducts two broad types of evaluation:
Formative
Formative evaluations, or formative reviews, examine the work of grantees as it happens. The overall goal is to provide feedback to grantees designed to maximize their ability to produce the results described in the funded proposal.
Project Officers (POs) are charged with the responsibility of monitoring the progress of a grantee over the course of several years. Ongoing email and phone correspondence between grantee and PO helps grantees stay on course to deliver their promised results. During the course of their monitoring efforts, POs may realize that a grantee could benefit from the input of two or three expert coaches who are familiar with what the grantee is trying to do. These circumstances are ideal for formative review.
The entire formative review process can be divided into pre-, during, and post-formative review stages.
- Pre-Formative Review
-
The Formative Review Coordinator generates a list of eligible grantees that started within the last 12-18 months grouped by Project Officer. Project Officers then review documents (e.g., grant application, any notes, Annual Performance Reports), email and phone correspondence of the grantees on the list. They then complete a Formative Review Screening Form for only the grantees they believe could benefit from Formative Review. Senior management approves the master list.
Once approval from senior NIDILRR management is approved, the Formative Review Coordinator works with the Project Officer and Formative Review Contractor to: (1) notify the grantees and provide them with preparation instructions; (2) establish dates, teleconference phone lines, and room location; (3) recruit, retain, and orient two or three subject-matter expert reviewers.
The selected grantees prepare presentations that comply with the preparation instructions and provide web addresses where reviewers and NIDILRR can download and review them before the facilitated formative review teleconference.
- During Formative Review
-
During the day of the 2.5-hour formative review teleconference facilitated by the formative review coordinator, the grantee walks the expert reviewers and PO through their presentation; the reviewers and Project Officer ask questions and engage in a dialogue with the grantee.
The grantee can also ask for help or technical assistance in whatever areas they need during this dialogue. The issues raised and discussed during the dialogue are captured by the formative review contractor for later inclusion into a Formative Review Report. The grantee is dismissed and the expert reviewers are given instructions for what they are expected to do during the post-formative review stage.
- Post-Formative Review
-
In this phase, the expert reviewers are asked to jointly generate and compile their observations, implementable recommendations, or helpful suggestions via email. The master list is then emailed to the formative review coordinator and PO for review.
Once approved, the master list is then incorporated into the formative review report template by the formative review contractor. The completed formative review report is then sent to the formative review coordinator and PO for review and approval. Then, the report is modified in accordance with the feedback provided by NIDILRR personnel.
The formative review contractor emails the grantee the approved formative review report and carbon copies the PO and the formative review contractor. The PO uploads the formative review report into the grantee’s electronic TRIM folder so it becomes part of the official grantee record.
Through various email and phone correspondence, the PO and grantee work together to implement whatever feasible recommendations or suggestions they can. The implementation status of these recommendations is documented in their subsequent online annual performance report.
Questions about the formative review process should be emailed to Dr. william.schutz@ed.gov.
Summative
Summative evaluations usually look at what grantees have accomplished once they have completed their work. NIDILRR is always looking for new ways to determine the merit, worth, or value of its program funding mechanisms, grants, and the research, development, training, and knowledge translation projects conducted under the auspices of grants studying a particular disability-related area. For more information on NIDILRR’s summative or external evaluation, go to NIDILRR’s external evaluation page.