Conducting Rural Health Research, Needs Assessment, and Program Evaluation
Rural communities and healthcare facilities have limited resources to address many health-related needs.
Research and needs assessments can help determine where and how resources may best be targeted, and program
evaluations can indicate whether a particular intervention or approach works well in a rural context.
Rural stakeholders who understand the purposes of conducting research, needs assessments, and program
evaluations, and who have the tools to undertake such activities, will be better positioned to focus their
efforts where they will have the best result. Likewise, policymakers and funders who understand how to help
rural communities by supporting rural health research, assessments, and evaluations can help build our nation's
understanding of rural health needs and effective interventions to address those needs.
Given the importance of research, assessment, and evaluation to rural interventions and policy, the best rural
health research and community assessments have a member of the community as a member of the research team
helping to ensure that confidentiality is maintained, programs are appropriate, and language is accurate and
reflects the culture of the community.
Identifies the similarities and differences among rural health research, assessment, and evaluation
Discusses common methods, such as surveys and focus groups
Provides contacts within the field of rural health research
Addresses the importance of community-based participatory research to rural and Indigenous communities
Looks at the community health needs assessment (CHNA) requirements for nonprofit hospitals and public
Examines the importance of building the evidence base so interventions conducted in rural areas have the
maximum possible impact
What are the purposes and roles filled by research, needs assessment, and program evaluation
in the rural
Rural health research, assessment, and evaluation are all processes that seek to improve the health of rural
populations through better understanding. These three activities often overlap, share similar methods, and may
be defined differently depending on who you ask.
For the purpose of this guide, we will discuss them as follows:
Research seeks to discover new knowledge or test ways of knowing through investigation.
Researchers seek to answer specific research questions, meet their stated research aims or objectives, or test a
theory or hypothesis.
Needs assessments seek to measure, or assess, the circumstances present in a specific
community, organization, or program. Assessment can be used to identify needs or gaps in available rural health
services, as well as assets and strengths.
Program evaluation focuses on determining how well a program or service is doing in terms of
meeting specific goals and objectives. Ideally, the goals and objectives were identified prior to implementation
of the program, or at minimum, prior to knowing the program results. The focus of program evaluation is to
inform decisions about how to improve the delivery and impact of a program. The subject of the evaluation is
decided based on the program stakeholders' values.
All three activities can be conducted by a community, healthcare facility, government agency, academic
institution, or others, assuming the investigators have the appropriate skills or a mentor to help
develop those skills.
Before you begin an inquiry, determine whether you intend to perform research, assessment, or evaluation.
Clarifying your purpose will help you select the appropriate methods for documenting and sharing what you learn.
A Comparison of Research, Needs Assessment, and Program Evaluation
Intended to answer a question or set of questions
Yes, a desire to understand the state of affairs
Yes, is the implementation going according to plan and is the
its intended effect?
Involves an intervention
May or may not
May or may not. Program evaluations can be done with programs that
are not implementing interventions.
Has measures defined in advance
Yes, could also identify objectives or aims
Includes baseline data
May or may not
Can serve as baseline data or can be compared to previous
Ideally, yes. However, it may be possible without baseline data,
depending on how the goals and objectives are defined and on the program timeline.
May be funded by a grant
Findings may be published in a journal article
Helps build the knowledge base of rural health
What are common data collection methods used in research, assessment, and
Many methods are available to gather evidence. Selecting the appropriate methods will depend on the questions
you have in mind, the resources and expertise available, and time and geographic constraints. While this is not
an exhaustive list, it covers some of the more common tools.
Surveys use sets of questions, which can be asked in-person, by phone, on a paper form, or
online. These questions can be targeted to an individual, such as a rural resident or healthcare professional,
or to an institution, such as a rural hospital. A survey, or questionnaire, might be used by itself for
point-in-time information, or might be targeted to before and after an intervention to see if there is a change
Interviews are discussions between an interviewer and another person, who may represent
themselves individually or represent an institution. Interviews are less structured than surveys. A key
informant interview, for example, is used to find out in-depth information about a community or organization by
asking a stakeholder for their insights.
Focus groups are group interviews. A facilitator leads a gathering of participants, all of
whom share some common characteristic, such as living in the same rural community or belonging to a similar
stakeholder group. Like the interview, a focus group is less structured than a survey. Because of interactions
among the participants, it may offer rich insights that would not be obtained individually.
Observation involves the investigator watching participants do something that is of interest.
For example, a study might observe how healthcare workers interact with a telemedicine program in providing care
to patients. Observation is also common in program evaluation.
Abstraction is the process of taking data from existing records. For example, a study could
involve pulling specific information from a set of medical records to determine if best practices were followed.
Indicators are measurable
information showing the condition or existence of something.
especially useful in evaluation because they are trusted by
administrators or program staff to inform decisions. For example, many people in rural areas will seek
healthcare in an emergency department after clinic hours or during the weekend. These costly visits are an
of another problem: a lack of access to primary care during the evening and on weekends. Care should be taken in
selecting indicators, as some may be time-consuming or expensive to collect.
What are special considerations for keeping collected data private and secure?
Whether you are collecting data for research, assessment, or evaluation, it is your obligation to make sure
that it is managed in a responsible manner, particularly any information that relates to an individual or
family. You should communicate how you intend to maintain the privacy of participants in the cover letter,
information sheet, or focus group agreement.
In rural areas, where people may know more about one another and where various characteristics can make
it easier to pick out a person from the crowd, it is even more important to consider how data is handled and
stored. The National Committee on Vital and Health Statistics Toolkit
Communities Using Health Data includes information on de-identification, data security, and other topics
for protecting data.
Funders, and federal funders in particular, have specific requirements and processes to protect the well-being
and safety of participants. For example, studies that receive funding through the U.S. Department of Health and
Human Services (HHS) and that involve human subjects are subject to review by an institutional review board
(IRB), which is regulated through the Office for Human Research
Protections (OHRP). The purpose of an IRB is to protect the rights, wellbeing, and welfare of research
When should we conduct a needs assessment and plan the program evaluation for our rural community
Needs assessments and program evaluation planning should be included at the start of a rural
Conduct a needs assessment to identify the issues to address and identify helpful resources
Select and adapt, as necessary, a program with goals aligned to meet the identified need and that is
feasible given the available resources
Plan how you will evaluate the program and identify baseline data, if they exist
It can be appealing when you hear about a successful project in another rural community, or a set of activities
that sound useful, to decide that it is the right intervention to use. However, there is no guarantee the
in the other community are similar enough to your own for the same intervention to work.
A needs assessment conducted prior to the beginning of program planning and implementation will be most
effective. Understanding the current circumstances, needs, and strengths can help program planners select
and adapt an intervention that is a good match for their community. Learn more about rural health needs
assessments in the second section of this guide.
Rural communities, in particular, face challenges if they invest resources in a poorly targeted intervention,
Not addressing the intended goals of the program
Not being able to afford an alternative intervention if the first is not successful
Losing the confidence of community members and stakeholders, which may be difficult to renew for future
Losing the confidence of the project's funder, especially in an area where there are likely limited sources
A program evaluation should be designed in the planning stage of a project. This will clarify the rationale for
the program and how the proposed activities target the needs the program intends to address. Planning the
evaluation before you begin will ensure you are measuring needs and circumstances within the direct and
immediate control of the program. You can learn more about rural program
evaluation in a later section of this guide.
Rural Health Research
What kinds of questions does rural health research seek to address?
Rural health research can focus on a wide range of topics important to the delivery of rural medical services,
the viability of rural healthcare facilities, and the health of rural people. It is part of the field of health
services research that looks more broadly at healthcare access, quality, and cost. Rural health research also
examines variables that impact population health, personal health, and health behavior in rural areas, including
social determinants of health.
The Federal Office of Rural Health Policy (FORHP) funds rural health research centers to study issues of
national concern related to rural health. The Rural
Research Gateway lists current
centers, as well as the research projects and findings underway and completed. It also lists a number of
previously funded rural health research centers and their work.
Health services researchers at other organizations also study rural health issues. Some of these researchers
work at academic centers focused specifically on rural health. Others may be part of an academic department,
research organization, or other group that has another primary focus, such as nutrition, tribal issues, or
healthcare workforce, and includes "rural" as an aspect of their research because it relates to their
primary field. Federal agencies that engage in rural health research include the Centers for Disease Control and Prevention, the Centers
for Medicare and Medicaid Services, and the Agency
for Healthcare Research and Quality.
For a selected list of centers that conduct or otherwise support rural health research, please see the Organizations
section of this guide.
What funding is available to support rural health research?
In addition to funding provided by the Health Resources and Services Administration through the Federal Office
of Rural Health Policy and its Rural
Health Research Centers program, these federal organizations also support research on rural health
To browse specific funding opportunities for rural health research, see the Funding
section of this guide.
How do you select an appropriate rural definition for a research study?
A 2005 American Journal of Public Health article, Rural
Definitions for Health Policy and
Research, discusses the importance of using an appropriate rural definition to develop research
findings that offer accurate conclusions. The three key considerations the authors identify for selecting a
rural definition to use in a research project are:
The purpose of the study
Availability of data
Appropriateness and availability of the definition
The authors recommend researchers learn about the various definitions and the pros and cons of each early in
their research process. The paper points out not only the differences between urban and rural areas, but also
across rural areas, depending on their population size, geographic isolation, and other factors. Any definition
will either under- or over-represent rural or urban in some way. Definitions that use only two categories, rural
and urban, with no gradations within the rural category, may miss local issues that impact particular types of
is Rural?, the USDA Economic Research Service describes the various definitions of rural and urban,
noting that “rural and urban are multidimensional concepts” that can be defined by population
density, geographic isolation, population size — each of which can define certain areas as rural or not
rural depending on the metric. The end of the article offers links to data sources from the Office of Management
and Budget and the U.S. Census Bureau. The USDA ERS also maintains Rural-Urban Commuting
Area Codes, which classify census tracts on a scale of 1-10 based on census data including population
density, daily commuting, and urbanization.
A 2013 article in the Journal of Environmental and Public Health,The
Utility of Rural and Underserved
Designations in Geospatial Assessments of Distance Traveled to Healthcare Services: Implications for Public
Health Research and Practice, discusses the impacts of rural definitions on public health outcomes,
noting that the common rural definitions used for health services research were not developed with health policy
in mind and that they should be used cautiously. The article reports on a study comparing eight counties in
Central Texas, using various rural and underserved designations to examine the same health issues as part of a
community assessment. Of the eight counties, only the most urban and the most remote county had consistent
status as urban or rural across the definitions considered, illustrating how the selection of a rural definition
can impact rural health research findings and resulting health policy.
For more details about current rural definitions, see the RHIhub topic guide What is Rural?
Are there concerns unique to rural areas researchers should keep in mind?
Aside from concerns about how to select an appropriate rural definition, additional issues rural health
researchers should consider include:
Challenges related to sample size, given the smaller population pool to draw on. Use of qualitative methods,
such as interviews and focus groups, is one option for addressing this concern. Issues related to smaller
populations may also affect research on health equity in rural areas, so care must be exercised when
combining data on race, age, gender, or other demographics.
The potential need for an on-site healthcare facility collaborator to recruit participants that meet the
Researchers seeking to collaborate with rural facilities should be aware that there may be resource
limitations at these facilities and be sensitive to any time or other burdens the project may cause.
When studying only one or a few rural communities, it may not be appropriate to generalize findings to other
regions or other types of rural communities. The variety of rural communities, from Appalachia to Alaska,
from island communities to mountainous areas, and from tourist-based economies with population fluctuations
to areas with a steadier population, can impact many aspects of healthcare.
Travel time and cost incurred by researchers for studies that require them to be present in-person.
Cultural differences for both rural and tribal communities that may impact the most effective methods for
engaging with study participants.
Skepticism or distrust of academic organizations, concerns about privacy, or concerns about the
implication of the research on policy or funding issues affecting their communities.
A 2018 findings brief from the North Carolina Rural Health Research Program, Range Matters: Rural Averages Can Conceal
Important Information, discusses the importance of looking at highs and lows in data rather than just
relying on averages. It argues that rural health data may include more extreme values than urban data and
provides three rural examples illustrating the benefit of considering the range as well as average.
“an approach to health and environmental research meant to
increase the value of studies for both researchers and the communities participating in a
When using the CBPR approach, community members, healthcare facilities, and other stakeholders work alongside
researchers. This cooperation should begin as early as setting a research agenda and identifying community needs
and continue through dissemination of research findings, implementation of interventions, and on to future
studies. The development of an ongoing relationship between the community and the researchers
is an important piece of CBPR. It helps develop trust and participation and
ensures the work undertaken is relevant to the community and that findings and interventions benefit the
community, not just the researcher’s academic interests.
The CBPR approach has the potential to better meet the research needs of rural minority populations and help
address the health disparities they experience. Historically, Indigenous and other communities of color have
been the subject of research and they have not been considered
active participants in the research process. As research subjects, they have experienced institutional
racism from and mistreatment by research organizations, including cases such as the infamous Tuskegee syphilis study. The long history of research
abuse experienced by American Indian communities is recounted in a 2012 Ethics & Behavior article
No Meaningful Apology for American
Indian Unethical Research Abuses. Indigenous and other communities of color are also less represented in
clinical studies, which is another barrier to identifying effective interventions to address health disparities.
A 2012 Health Services Research article, A Systematic Review of Community-Based
Participatory Research to Enhance Clinical Trials in Racial and Ethnic Minority Groups, found the use of
CBPR improved recruitment of communities of color into clinical trials and helped determine the effectiveness of
interventions for these groups.
American Indian communities have an interest in CBPR, and in particular tribal participatory research
(TPR), as a way to ensure research conducted will contribute to the health of American Indians and
be respectful and appropriate to a community's culture, with power
distributed equally between the tribal government and the researcher. TPR, as defined in Tribal Participatory Research: Mechanisms of a
Collaborative Model, involves:
Tribal oversight and control of the research process
A formal resolution from the Tribal Council identifying the goals and methods of the project
An oversight committee with tribal authority to approve or deny project activities
A tribal research code that governs the research process
A facilitator for meetings between the oversight committee and researchers
Community members hired and trained to work on the project, which is important to build the
tribe's research capacity
Culturally specific methods that are aligned with community values
Given the long history of researchers coming into tribal communities and conducting research that did not
benefit participants, American Indian communities may find CBPR useful as a way to prevent this type of
encounter and to help shape research that meets their community’s needs.
Tribal governments and other organizations are developing infrastructure for American Indian communities to not
only act as research partners with outside entities but also to conduct their own research, including the
support and development of researchers from within the community. For example, in the Upper Midwest, the Collaborative Research Center for American Indian Health is building tribal
research infrastructure through pilot grants, training resources, and an annual summit.
Recognize the historical research experiences of tribal communities and work to improve the image of
Understand tribal sovereignty
Learn about tribal membership and community membership
Understand the differences among tribes
Expect a long review and approval process and allow time to build trust
Be prepared to allocate a large portion of project funding to oversight and planning
Identify key tribal members, such as tribal elders and tribal government representatives
Anticipate turnover in tribal leadership
Learn about and incorporate the tribe's spiritual and philosophical beliefs into the project
Consider the tribal context when interpreting the data
Obtain tribal approval for publication of study results
Recognize increased tribal capacity as a valuable outcome of the project
What is comparative effectiveness research (CER) and how can it help us
understand how well specific
healthcare interventions work for rural residents?
effectiveness research (CER) helps inform healthcare decision-making by comparing different treatment
options. It looks at how well a particular procedure, medication, test, or other healthcare intervention works
compared to other options. CER may focus on how well a treatment works for a particular group based on
race/ethnicity, age, or gender. When focused on rural health, CER examines healthcare outcomes for a
particular treatment or intervention used in a rural setting.
By gathering evidence about the effectiveness of different treatments for rural populations, CER can help
Rural clinicians and healthcare facilities deciding on what interventions to use
Rural patients and their families selecting among treatment options
Health insurance companies and other payers deciding what preventive care and treatments to reimburse and at
Policymakers supporting effective approaches in a rural setting
The Patient-Centered Outcomes Research Institute (PCORI), which was
established in 2010, seeks to help patients and providers make informed healthcare decisions based
on evidence of effective treatment. PCORI funds studies that address populations of interest, including
residents of rural areas, and summarizes findings from those studies.
Public and patient engagement is central to the CER studies PCORI funds, an approach that is influencing others
in health services research. PCORI provides a range
of resources to help researchers and their community partners better understand what the Institute seeks
in applications. This partnership is key in ensuring that potential PCORI-funded studies reflect real world
decisions faced by patients and those who care for them.
“PCORI recognizes that doing research differently will require input and involvement from all
communities across the healthcare continuum. PCORI is interested in investing in stakeholders who believe in
promise of PCOR/CER but may not have had the opportunity to engage actively in research in the past.”
To help communities and researchers better prepare to participate in and disseminate CER findings, PCORI's
Engagement Award program funds projects to increase communities’ knowledge and understanding of CER, as
well as their ability to disseminate PCORI-funded evidence.
What is the role of practice-based research networks (PBRNs) and what are some examples of rural PBRNs?
Practice-Based Research Networks (PBRNs) are networks of healthcare providers who seek to improve community
health by translating research findings into clinical practice. Often PBRNs also include academic researchers
as part of the network. The Agency for Healthcare Research and Quality (AHRQ) provides an overview
of primary care PBRNs.
Because a PBRN's research agenda is shaped by practicing clinicians, it can focus on issues with practical
relevance for patient care. As discussed in the 2014 Journal of Rural Health article Recruiting Rural Participants for a
Telehealth Intervention on Diabetes Self-Management, academic researchers can develop stronger
connections to rural communities and potential research participants by working with local healthcare providers
via PBRNs. Involvement in a PBRN can also help rural clinicians build their local research capacity.
Some of the roles of PBRNs, according to AHRQ, include:
Conducting comparative effectiveness research
Supporting quality improvement within the member primary care practices
Developing an evidence-based culture
AHRQ maintains of list of PBRNs
including several rural PBRNs, such as:
How and where can you share rural health research results?
Rural health research results can be shared in many ways. Two key factors to consider are:
Where your target audience typically seeks information
How quickly you hope to get your results out
The same project may be shared more than once in order to take advantage of the benefits of different formats as
the research project unfolds:
Guidelines for Sharing Rural Health Research Results
Keep in Mind
Reports to the funder and/or stakeholder organizations
Information that may need to be kept private
Projects with no time constraints
Not likely to reach a broader audience.
Does not build the evidence base.
White papers, policy briefs, infographics, and other documents published directly
Practical information and lessons learned
Reaching practitioners and decision makers quickly with useful information
Web-based distribution typically has the broadest reach. Print copies may
be useful for special audiences and events.
Consider sharing in PDF format to ensure integrity of the document.
Media releases and interviews
Reaching a large audience, particularly the general public
Be sure that media outlets pursued can help you reach your target
When working with the media, you give up control of how your results are portrayed.
Reaching a broad audience
Works best to promote a product, such as a white paper or media article
Conference presentations and posters
Practical information and lessons learned
Reaching practitioners quickly with useful information
Sharing early stages of research process
Audience members may share photos or quote you, for example via social
media, so don't share anything you wouldn't want to make public.
Consider sharing in PDF format, to ensure integrity of the content.
Peer-reviewed journal articles
Projects that follow rigorous research protocols
Building the evidence base
The review and publication timeline is often lengthy.
Journals may have concerns or restrictions related to information that is publicly available in
another format, such as a white paper.
Access may be restricted to the journal's subscribers and people with access through a library.
Whatever format you choose, keep in mind that sharing your work in PDF format, with editing permissions
restricted, rather than in the original word processing or presentation slide deck files will make it easier to
protect the integrity of your document's layout and content.
What are some different types of assessments relevant to rural health?
An assessment seeks to understand the current state of affairs in a community, organization, or program. Rural
healthcare facilities and rural stakeholders conduct assessments as a first step in deciding how to strengthen
rural health services. For example, you might assess:
A healthcare facility's attractiveness to potential healthcare providers by identifying
facility and community strengths and weaknesses, for example through the Community Apgar
The need for a particular healthcare service in a community
The health status of a particular rural population, such as a racial/ethnic group or
Health needs related to a particular health condition in a rural community
Rural healthcare workforce supply and demand, for one or many professions
Social determinants of health that impact overall community health
What are the main steps in planning for and conducting an assessment?
Rural communities planning to conduct an assessment will benefit from thinking about the resources available in
their particular community that can support the assessment activities. They can leverage existing relationships
and networks, both formal and informal, to gain support and partners in the assessment process. These networks
include a wide range of people and programs that impact health, such as clergy, schools, police, Head Start,
WIC, and others. They can make use of the hospital or public health newsletter, local newspaper, radio station,
and other local media to inform and engage community members. Already scheduled local events, such as regular
meetings of service organizations, are another opportunity for weaving the assessment activities into community
While the details of an assessment will vary depending upon its purpose, the broad steps are similar:
Gather the interested parties or stakeholders. Think about and network with other organizations in the
community or region. Who else could benefit from the information being gathered? Who should be at the table
to address the suspected issues or concerns? Who would be willing to contribute resources?
As a group, define what is being assessed. What geography? What types of services? Provided to
Identify the goals of the assessment and the purposes for which you anticipate the findings will be used.
Locate sources of existing data that can answer questions suggested by the assessment goals.
Gather information through focus groups, interviews, surveys, and public forums.
Analyze the results, including the development of maps and other types of reports that show what has been
learned. Consider including recommendations for next steps.
Share the findings.
Consider making assessment a recurring activity, for example repeating annually or every 3-5 years, rather
than a one-time task.
The Community Toolbox, a free service from the Center for Community Health and Development at the
University of Kansas, includes a section on Assessing
Community Needs and Resources that describes specific steps in the assessment process.
The COVID-19 pandemic has impacted the methods for conducting assessments and gathering community input,
shifting from in-person focus groups and interviews to surveys and online focus groups. In addition, COVID-19
may raise awareness about issues that were not previously visible to communities, which may necessitate
additional assessments to those conducted before the pandemic.
What are the requirements for hospitals to conduct Community Health Needs Assessments (CHNAs)?
On an annual basis, hospitals must provide a description of the actions taken during that taxable year to
address the needs. The implementation strategy should seek to address the needs identified in the CHNA. In
setting forth how it plans to address identified needs, a hospital should:
Explain what actions it will take and the anticipated impact of the actions,
Identify the programs and resources it plans to commit, and
Describe any planned collaboration between the hospital and other facilities or organizations
How can rural hospitals and public health agencies work together to conduct community assessments?
The IRS requirements of rural nonprofit hospitals are similar to the requirements that local public health
agencies must meet to pursue accreditation. In rural communities, the stakeholder groups for each assessment
overlap and the financial resources to undertake a study are limited. It makes sense for the hospital and public
health agency to work together.
The large geographic service areas of many rural hospitals and public health agencies may overlap
Each organization's influence with different groups of stakeholders
Sharing staff expertise and in-kind resources
Similar missions and responsibility to community health
Commitment to collaborate on similar activities
More likely to have institutional flexibility and history of working together
The PHI report also discusses how public health and nonprofit hospitals can work together to assess
community health and plan for improvement. Some of the issues highlighted include:
The need to build "shared ownership of community health"
Addressing differences in the service areas of the hospital and public health agency
The benefit of collaborating in data collection, which can be resource-intensive
The roles/involvement of the hospital, public health, and stakeholders in setting and addressing
Collaboration in evaluating community benefit program results, with hospitals potentially benefitting from
the evaluation expertise of public health staff
Jointly sharing findings and progress on improvement plans with the community
What CHNA tools and resources are available for rural facilities?
There are many different CHNA processes available, including examples that have worked well in rural areas.
Rural hospitals should consider a range of options to identify an approach that is a good match for their
The National Center for Rural Health Works offers a Community
Health Needs Assessment Toolkit developed in 2012 using a national advisory team and tested in several
rural communities. This toolkit, while designed for rural hospitals, can also be modified for use by other types
of organizations conducting similar assessments. The introduction of the toolkit addresses the legislative
requirements, the main sections walk through a step-by-step process, and the appendices include examples and key
contacts. The toolkit is also available in
additional background information on the CHNA. Other resources for learning about and conducting CHNAs:
What are the different purposes that evaluation can serve?
Program evaluations can serve many purposes, all with the goal of continuously improving programs to ensure
resources invested in the program and in future programs are well spent. Key purposes are:
Process Improvement Process improvement goes by many names, including process evaluation and formative
evaluation. Evaluating the processes and activities are critical to understand if a program is
implemented as intended. If it isn't, then there is less chance the program will be effective in meeting its
goals and objectives and achieving program outcomes. Each step of how the program is supposed to be
delivered should be written down. The evaluation then monitors whether each step happened as planned and, if
not, provides suggestions for improvement. Rural communities, in particular, often must adapt programs that
have not been implemented in a rural area. Evaluating these implementation processes can help fine-tune
those adaptations for the rural setting.
A funder often has some minimal data they would like the program to track, such as the number and type of
participants and the number and type of activities being conducted. This information doesn't say
whether the program is being delivered according to plan or is having its intended impact, but does give the
funder a general sense of whether the things they expect to be done are actually done.
Impact or Effectiveness Outcome evaluation focuses on the results the program achieved. The results of an outcome
evaluation will help determine what difference the program makes in the long term—its impact. The
information gathered can help an organization decide whether it is worthwhile to keep offering the program
and help funders decide whether to support the program. The impact of a program will also determine whether
other rural communities are likely to want to try a similar intervention.
Logic models, also called program roadmaps, are a graphic way to ensure the program assumptions are linked to
the activities you are implementing and the outcomes you expect the project to achieve or to which your project
is contributing. For example, a project intended to address obesity through a walking club might start with
Program Assumption – physical activity reduces obesity
Activity – walking club
Outcomes – Increased availability of walking clubs, people join and participate in
walking clubs, greater frequency and intensity of walking, increased fitness.
Logic models don’t usually include the measures, or indicators,
used to show progress. The logic model is a starting point for what you’re trying to achieve, and the indicators
are ways you measure and show progress. Having good process and outcome measures is critical.
There are many types and guides for developing logic models. This is a typical logic model layout:
Tearless Logic Models, a featured tool in the
December 2011 issue of the Global Journal of Community Psychology Practice provides a simplified
approach to developing a logic model by answering a series of questions.
Why is it important to evaluate rural health programs?
Rural communities face many challenges related to healthcare delivery and population health, but also have
resources to address these challenges. Program evaluation can help ensure the investment of staff time,
organizational will, and other resources are well directed. Program evaluation can also help demonstrate to
funders that a program is a worthwhile investment.
Another important reason to evaluate rural interventions is to show what works in a rural setting. Many of the
evidence-based approaches that federal programs or foundations may request applicants use were demonstrated to
work in a non-rural setting. Seeing how well these interventions work, or what adjustments might be needed to
make them effective and practical in a rural setting, is a key purpose for evaluating rural health programs. It
is also important to share the results of program evaluations through journal publications or other venues, such
as those listed in How and where can
you share rural health research results? so that other programs can refer to and learn from them as
evidence-based practices or promising practices.
Federal programs may want to work together with rural stakeholders when defining what to measure. For example,
the National Quality Forum’s
Measure Applications Partnership assists the U.S. Department of
Health & Human Services in selecting appropriate measures. Its
MAP Rural Health Workgroup focuses on
measurement challenges in rural healthcare.
How can rural programs plan for and conduct efficient and practical program evaluations?
Wilder Research, a nonprofit organization that conducts program evaluations, offers guidance for organizations
with limited resources in its fact sheet Evaluation on a
Shoestring Budget. Australian program evaluator Patricia Rogers offers evaluation tips in her Evaluation on a Shoestring blog post
from the BetterEvaluation.org website. Recommendations from these sources include:
Plan your evaluation at the outset of the program
Prioritize the evaluation's focus based on stakeholder needs
Limit the scope of the evaluation
Select approaches that are easy, appropriate to staff skills and time available, and fit your budget
Collaborate on the evaluation with project partner organizations
Consider seeking assistance from a local college or university
What tools are available to help rural grantees learn about program evaluation?
The National Rural Health Resource Center hosted several rural health program evaluation workshops and shares
workshop materials, which cover topics that include the steps of the evaluation process and evaluation
best practices. The materials also include adaptable templates and a list of additional
Offices of Rural Health
offer program evaluation technical assistance to rural grantees. Depending on your funding source, you may also
be able to get technical assistance through your grant program or funding organization.
Some other sources for learning about program evaluation include:
How can funders ensure their grantees' experiences help build the collective
understanding of what is effective
in addressing rural health issues?
Funders can encourage and support their grantees to share their project results and help other rural communities
learn about what is effective. Requests for proposals should include requirements and funding to support program
evaluation. Funders can offer financial support, learning opportunities, and technical assistance to grantees
regarding program evaluation.
They can also include funding or provide encouragement to program staff to write up their program results for
publication. Publication in a peer-reviewed journal is an important step in making a program or model part of
the evidence base. The AcademyHealth guide, Writing
Articles for Peer-Review Publications: A Quick Reference Guide for PHSSR, provides guidance on how to
develop and submit an article about a health services project for publication.
Funders are in a great position to identify projects that were particularly successful. They can recognize those
programs themselves through an award or honor or recommend the projects to other sources that identify model
programs. RHIhub, for example, is a resource for sharing and finding programs and approaches that rural
communities can adapt to improve the health of their residents. Many of the programs featured in RHIhub's Rural Health Models & Innovations and Evidence-Based
Toolkits were funded by the Federal Office of Rural Health Policy.
Another way that funders can make the most of their investments is to work strategically with each other,
their grantees' experiences and results. Communication and coordination can also help funders plan and
prioritize the projects they choose to support. For example, the Rural
Health Public-Private Partnership (RHPPP) is an
ongoing effort led by the Federal Office of Rural Health Policy, the National Rural Health Association, and
Grantmakers in Health. The collaboration brings together rural federal programs and rural-focused trusts and
foundations to work together on key rural health issues. In addition to identifying funding priorities, the
group develops evidence-based metrics to ensure the funded projects are making a difference. Read more about the
work of this partnership in the June 2015 Rural Monitor article, Government
and Philanthropies Join Forces for Rural Health.