Cross-Cutting Evaluation Considerations for AI Programs

Evaluating AI tools in rural healthcare settings is essential to understanding what works. However, limited capacity and resources can be significant barriers. Rural healthcare facilities that are part of a health system and have greater financial resources are more likely to evaluate their AI tools and systems. In contrast, Critical Access Hospitals and rural providers serving under-resourced areas may face barriers evaluating AI tools due to cost, capacity, and resources. Evaluations are needed to strengthen the evidence base for AI use across a range of rural healthcare settings in rural areas.

Need for Independent Evaluations

There is limited evidence for how AI tools work in real-world healthcare settings. AI technologies are often developed and updated more quickly than traditional evaluation timelines allow, meaning that by the time an evaluation is completed, the tool may have already changed. This pace creates challenges for conducting rigorous evaluations and makes it more difficult to generate timely, real-world evidence on AI effectiveness.

While many technology vendors conduct their own evaluations, there is less post-market surveillance and real-world monitoring on how AI tools perform once they are in the field. This increases the importance of independent evaluations of AI implementation, particularly in rural healthcare settings.

Evaluation Focus Areas

Evaluating AI in rural healthcare settings is complex, dynamic, and multidimensional. Meaningful evaluations should consider multiple perspectives and domains, including the extent to which AI tools:

• Save time and reduce workforce burden
• Integrate into clinical workflows
• Increase operational efficiency
• Improve patient and clinical outcomes

Evaluations should be tailored to the specific tasks an AI tool performs, its intended clinical use, and the setting in which it is deployed. Because healthcare providers, staff, and patients may interact with the same AI tool in different ways, evaluations should account for these differing experiences.

Strengthening AI Evaluation in Rural Healthcare

There are several areas that deserve greater attention when designing and conducting evaluations of AI tools in rural healthcare settings, including:

• Real-world implementation studies
• Longitudinal data collection
• Integration and system-level evaluation of AI
• Ongoing effectiveness and safety monitoring
• Assessment of sustainability
• Integration of ethical considerations
• Identification of unintended consequences
• Understanding how effects differ across patient populations
• Strong study designs with appropriate controls
• Engagement and buy-in from clinical leaders, providers, partners, and staff
• Outcome measures that reflect what matters to patients
• Adequate time and resources for clinical studies, including recruitment
• Time and workflow studies to assess efficiency, provider burden, and changes in how providers' time is allocated
• Return on investment studies
• Greater consistency in AI evaluation standards

Addressing these areas can help support more relevant and actionable evaluations of AI in rural healthcare settings.