SUD Program Evaluation Questions and Measures

Evaluation Questions

Evaluation questions — also known as research questions — are used in program evaluation to guide data collection and analysis.

Process Evaluation Questions

Process evaluation questions focus on whether the program is operating as planned, the services delivered to program participants, and barriers and facilitators to implementation.

Examples of process evaluation questions include:

What substance use disorder (SUD) treatment services (such as medication, individual counseling, group counseling, vocational services, healthcare navigation) are being delivered and to whom?
What barriers affected program implementation?
What helped facilitate program implementation?
What percentage of patients are referred to recovery support services? What types of services are they accessing?
What barriers do patients experience in accessing prevention, treatment, and recovery support services?
What barriers do healthcare providers, law enforcement, emergency medical services, or other relevant groups face in obtaining and administering naloxone?

Data sources for process evaluation questions include program records, patient feedback, and program staff interviews.

Outcome Evaluation Questions

Outcome evaluation questions may focus on improvements in patients' quality of life, changes in substance misuse patterns, and whether health outcomes can be attributed to the program.

Examples of outcome evaluation questions include:

Did patients report reductions in quantity or frequency of substance use?
What policy changes related to naloxone distribution were enacted as a result of the program?
Did the area served experience reductions in alcohol-related motor vehicle crashes?
Has the program shown measurable improvements in health outcomes, including reducing overdose morbidity and mortality?

Data sources include patient health records, legislative changes, and local surveillance databases.

Evaluation Measures

Evaluation measures — also known as performance measures or metrics — can help SUD programs measure program success and effectiveness. Collecting data at baseline and at regular intervals can help document changes within the population served and monitor program progress.

The evaluation measures used by rural SUD programs vary depending on the program model and the goals. Evaluation measures may include demographic information, process measures, and outcome measures.

Demographic Information

Demographic measures focus on describing background information about participants. This information can be important for understanding access, outcomes, and satisfaction among different demographic groups. Examples include:

Age
Educational attainment
Income
Insurance status (Medicare, Medicaid, private, uninsured)

Process Measures

Process measures focus on how services are provided. Examples include:

Number of staff trained
Number of educational sessions held
Number of attendees at program events
Number of coalition meetings held
Number of patients enrolled in and/or served by the program
Number of patients receiving services
Number of referrals to other services
Duration (days or weeks) of patient participation in treatment or services
Number of patient healthcare and/or mental health visits in the past year
Number of clinical and social service providers

Outcome Measures

Outcome measures focus on the program's results and achievements. Examples include:

Rates of SUDs in the patient population or in the community
Rates of hospitalization for SUD or overdose
Rates of substance misuse-related emergency department visits
Rates of drug overdose
Rates of binge or heavy drinking
Rates of underage drinking
Rate of substance misuse-related mortality
Duration (days, weeks, or years) of abstinence from substances such as opioids, alcohol, and stimulants
Percent of patients who have a reduction in drug use
Number of patients discharged to other care settings

Program Setting

Evaluation measures may differ by setting and delivery approach for SUD treatment. Programs may define and measure patient progress and recovery differently, depending on the setting. A program providing SUD services in an inpatient residential setting, for example, may measure the number of patients moving out of the residential setting and into outpatient care management (such as primary care). However, this measure may not apply to a clinic or office-based program, which require different measures tailored to their SUD service delivery approach.

Support Services

When assessing the effectiveness of an SUD program, it is important to also evaluate support and social services, such as counseling, housing, and transportation. For example, in a program that pairs medication for SUD treatment with counseling, changes in frequency of counseling sessions (such as, moving from weekly to bi-weekly sessions) can be a measure of patient progress and program effectiveness. Other measures may include the number of referrals to treatment for support services, and the number of patients who were unhoused who secure or maintain stable housing.

For general guidance for determining evaluation measures, see Evaluation Measures for Community Health Programs in the Rural Community Health Toolkit.

Resources to Learn More

MOUD Performance Measure Reports: A Guide for Developing and Implementing MOUD Treatment Reports to Evaluate Treatment Quality
Document
Highlights key performance metrics for MOUD treatment initiation, engagement, and retention. Describes how performance metrics were developed using data elements from electronic health records and standardized measures from the Pennsylvania Medications for Opioid Use Disorder Technical Assistance and Quality Improvement Expansion Project.
Author(s): Wooten, K., Pieper, D.
Organization(s): University of Pittsburgh School of Pharmacy, Program Evaluation and Research Unit
Date: 8/2023