ROI Measurement Challenges in Rural Settings

Measuring return on investment (ROI) in rural health programs can be challenging. This section talks about when not to conduct ROI analysis and common challenges.

When Not to Conduct ROI Analysis

In some cases, rural program leaders and/or rural programs may not be ready to conduct ROI analysis. Common reasons not to conduct ROI analysis are:

Staff have little experience with quantitative data or research.
There is very limited data to measure program outcomes.
Most benefits are non‑monetary and hard to assign a dollar value.
There are no published or data‑driven measures of program value.
The program is very new and has not yet produced measurable benefits.
Program leaders do not have a clear understanding of what the intervention costs.
The program is so new that current cost data do not reflect long‑term implementation costs.

Challenges Monetizing Benefits

ROI requires monetizing the program's benefits — in other words, translating the program's outcomes into monetary terms like dollars. In rural health settings, this is challenging because many important program outcomes are not easy to assign a dollar value to, such as:

Improved quality of life
Better care coordination
Higher patient satisfaction
Stronger partnerships across sectors
Greater community engagement
Improved workforce retention
Increased community capacity

A strong ROI approach recognizes both financial benefits and social or health benefits. Rural program leaders should translate as many benefits as possible into monetary terms, while still describing some impacts qualitatively. It is also helpful to use additional metrics — not just ROI — to show the full value of the program.

Capturing Benefits Across Sectors

Rural community health programs often create benefits beyond the health sector. For example, they can also improve education, transportation, and social connection. They can also result in new partnerships and greater care coordination. As a result, many different agencies and organizations may save money because of the program. If these savings are not included when calculating ROI, the program may appear less valuable than it really is — even though it has created important benefits for the community.

Weak Counterfactuals

Counterfactual analysis can be challenging. This is an area where rural program leaders will need staff with expertise in statistics and analytical modeling. Potentially weak counterfactuals include a before-and-after comparison that is unrelated to the program or using an average from an unrelated or dissimilar program or setting. There are ways to make counterfactual analysis more reliable. These include collecting more data, using statistical methods to evaluate outcomes, testing results under different assumptions, looking at differences across subgroups, and combining multiple analytic approaches.

Data Limitations

Rural programs often operate with fragmented, incomplete, or non-standardized data systems. This can make it challenging to conduct a high-quality ROI analysis. Rural programs can use the data they do have to produce early ROI estimates. At the same time, they can work on building stronger data collection systems for future ROI calculations.

Measuring ROI in the Short-Term

Rural programs need time to gather data and show results. Because changes in behavior and health outcomes often take months or years, it can be challenging to show strong ROI in the short term.

Uncertainty of ROI Results

The results of an ROI analysis depend on the assumptions made, the data available, and the program context. Because of this, rural program leaders should clearly explain all assumptions and data limitations. Sensitivity analyses can help show how changes in key assumptions affect ROI results and identify which factors may have the greatest influence on the findings. This can help rural program leaders better understand how confident they can be in the results.

Resources to Learn More

Using Return on Investment Analysis to Evaluate Health Promotion Programs: Challenges and Opportunities
Document
Describes the strengths and weaknesses of ROI analysis in evaluating the effectiveness of health promotion and disease prevention programs.
Organization: RTI-UNC Center of Excellence
Date: 2006