Evaluation Strategies and Considerations for Telehealth Programs
Rural telehealth programs are complex and varied addressing different types of health issues, healthcare challenges, technologies, populations, outcomes, and stakeholders. Therefore, no single evaluation strategy can be applied to all rural telehealth programs. A number of research and evaluation studies have investigated successful telehealth evaluation criteria. Due to its wide scope and varying reimbursement structures, there are many aspects of telehealth programs that can be evaluated. This range is accompanied with a large set of variables, including clinical applications, characteristics of the information being transmitted, technology functionality, and organizational context.
The California Telehealth Resource Center Telehealth Program Developer Kit defines three key phases of telehealth program development and implementation. These phases can be used to support evaluation efforts:
- Assess and define – Programs should determine service needs and consider their environment, define their program's model, and develop a business case for the telehealth program. Identifying these components helps telehealth programs design and frame their evaluation plans.
- Develop and plan – Programs should focus on creating a detailed project plan while also identifying the evaluation process and metrics. This includes developing a quality improvement process, communication plan, and technology implementation plan. Ensuring evaluation measures and efforts are tailored for each component of the plan will allow programs to obtain an accurate, holistic picture of their program strengths and opportunities for improvement. Programs should consider the data needed, methods for collecting the data, ways to present the data, and the frequency at which these data analytics reports will be generated.
- Implement and monitor – As telehealth programs implement their activities, continuous program monitoring is important to help identify potential points of improvement. Regular evaluation of the program allows the implementation team to identify areas of high and low performance and therefore make changes as needed. This information may be provided in data reports and patient satisfaction surveys.
Evaluation is important for demonstrating the value of telehealth programs, which can be fundamental to justifying continued or expanded investment in telehealth infrastructure and program operations. Strategic planning is important from the start and helps telehealth programs develop plans for evaluations that capture metrics to help show value. Incorporating a robust evaluation plan and tracking the appropriate measures can help programs to show success and build a business case. Thoughtful evaluation plans can also help programs sustain or expand over time.
When designing telehealth evaluation plans, programs should consider the types of measures that are important to key stakeholders. These stakeholders may include providers, payers, health plans, and hospital boards, among others. Research suggests that programs should incorporate flexible multi-method and multi-phase evaluation approaches to allow for easy adaptation and to consider the characteristics and challenges that are unique to individual telehealth programs. Additionally, evaluating process measures can help inform implementation and allow programs to make adjustments as needed to improve processes and outcomes. For innovative programs, tracking quality improvement and quality assurance efforts may be important for ensuring that patients receive the same standard of care that they would through in-person visits.
Rural programs interested in conducting evaluation activities can contact regional Telehealth Resource Centers to receive free technical assistance.
Resources to Learn More
A Standard Telemental Health Evaluation
Model: The Time is Now
Discusses the process and feasibility of developing a new standard evaluation model for telemental health programs that considers multiple factors, such as healthcare costs and savings, patient and provider interactions, telecommunications, and more.
Author(s): Kramer, G.M., Shore, J.H., Mishkind, M.C., et al.
Citation: Telemedicine Journal and e-Health, 18(4), 309-313
Approaches to Evaluating Telemedicine
Builds upon the Institute of Medicine's (IOM) broad framework for telehealth evaluation. Identifies issues that evaluations of telemedicine can focus on and the resulting information they can yield.
Organization(s): Office of the Assistant Secretary for Planning and Evaluation
Considerations for Planning and Evaluating
Economic Analysis of Telemental Health
Summarizes issues and limitations of existing economic analyses of telemental health. Reviews research and methodology considerations in support of a telemental health economic evaluation.
Author(s): Luxton, D.D.
Citation: Psychological Services, 10(3), 276-282
Evaluation Framework for Telemedicine
Using the Logical Framework Approach and a Fishbone Diagram
Summarizes themes from multiple evaluation strategies to describe a comprehensive evaluation framework for telemedicine implementation.
Author(s): Chang, H.
Citation: Healthcare Informatics Research, 21(4), 230-238
How Do I Know If It Works?
Compiles resources on telehealth program evaluation and the evidence base for telehealth efficacy and efficiency.
Organization(s): Mid-Atlantic Telehealth Resource Center
“Real-World” Practical Evaluation
Strategies: A Review of Telehealth Evaluation
Describes current telehealth evaluation strategies, best practices, and challenges to outline approaches and evaluations considerations for telehealth programs.
Author(s): Agboola, S., Hale, T., Masters, C., Kvedar, J., & Jethwani, K.
Citation: JMIR Research Protocols, 3(4), e75
Return on Investment (ROI)
Lists articles and resources for telehealth programs to consider when working to develop a business plan for telehealth.
Organization(s): Mid-Atlantic Telehealth Resource Center
The Business of Telemedicine: Strategy
Describes a structural framework for strategic planning and telehealth program implementation. Provides recommendations to support telehealth programs moving from a grant-funded to profit-centered status for program sustainment.
Author(s): LeRouge, C., Tulu, B., & Forducey, P.
Citation: Telemedicine Journal and e-Health, 16(8), 898-909
CTRC Telehealth Program Developer Kit: A Roadmap for Successful Telehealth Program Development
Describes considerations and resources for healthcare organizations and individuals starting new telehealth programs. Includes measures looking at sustainability and operational considerations and lessons learned from existing telehealth programs.
Organization(s): California Telehealth Resource Center