Evaluation Design
There are different designs that can be used to evaluate programs. Given that each program is unique, it is important to choose an evaluation that aligns with:
- Program goals
- Evaluation research questions
- Purpose of the evaluation
- Available resources
Your evaluation should be designed to answer the identified evaluation research questions.
To evaluate the effect that a program has on participants’ health outcomes, behaviors, and knowledge, there are three different potential designs:
- Experimental design: Used to determine if a program or intervention is more effective than the current process. Involves randomly assigning participants to a treatment or control group. This type of design is often considered to be the gold standard against which other research designs are judged, as it offers a powerful technique for evaluating cause and effect. Fully experimental designs are unusual in evaluation research for rural community health programs, but may be feasible.
- Quasi-experimental design: Does not have a random assignment component, but may involve comparing a treatment group to a similar group that is not participating in the program. Quasi-experimental methods are used to estimate the effect of a treatment, policy, or intervention when controlled experiments are not feasible.
- Non-experimental design: Does not involve a comparison group. Non-experimental designs may include pre- and post-intervention studies with no control or comparison group, case study approaches, and post-intervention-only approaches, among others. The key feature of a non-experimental design is the lack of a control group. While non-experimental evaluation studies are likely to produce actionable findings regarding program outcomes, best practices, and performance improvement, they cannot control for extraneous factors that could influence outcomes, such as community contextual factors or selection bias.
Other frameworks that have been used to evaluate rural initiatives or programs include:
- Process Evaluation: Process evaluation is a systematic, focused plan for collecting data to determine whether the program model is implemented as originally intended and, if not, how operations differ from those initially planned. It seeks to answer the question, “What services are actually being delivered and to whom?” This framework also gathers information on stakeholders' perceptions of the program.
- Outcome Evaluation: Outcome evaluation examines how well a project achieved the outcomes it set at the beginning. It is generally a summative evaluation of the program which can be used to make recommendations for future program improvements.
- Impact Evaluation: Impact evaluation reviews the effect that a program had on participants and stakeholders of the project. It measures the outcomes, but also the changes that resulted from those outcomes.
- Performance Monitoring: Performance monitoring is ongoing evaluation of the program to have data at the baseline and at key milestones in the work plan. This provides continuous, real-time feedback on program progress so that changes to the program can be made to better align with the program objectives and goals.
- Cost-benefit Evaluation: Cost-benefit evaluations study the cost-effectiveness of the program by reviewing the relationship between the project costs and the outcomes (or benefits) from the program. Data collected is used to determine whether the program outcomes were worth the investment in program development and operation.
Resources to Learn More
Framework for Program Evaluation in Public
Health
Website
Guided steps and standards for program evaluation and information on how findings from research will lead to
program plans that are clearer and more logical; stronger partnership; integrated information systems will
support more systematic measurement; and lessons learned from previous programs.
Organization(s): Centers for Disease Control and Prevention
The
Community Tool Box: Chapter 36 - Section 1. A Framework for Program Evaluation: A Gateway to Tools
Website
Describes why evaluations are important, frameworks for program evaluation, and standards for developing a
strong program evaluation.
Organization(s): University of Kansas Work Group for Community Health and
Development
Non-Researcher's
Guide to Evidence-Based Program Evaluation
Document
This guide is an overview of program evaluation and each of the main steps, including evaluation goals, types of
evaluation, and evaluation designs. It clearly walks through each of these steps and provides examples
throughout.
Organization(s): Substance Abuse and Mental Health Services Administration(SAMHSA)
Date: 7/2012
The Step-By-Step Guide to Evaluation: How to Become Savvy Evaluation
Consumers
Document
This evaluation guide provides a framework, detailed information, and tips for employing evaluation to inform
and track progress of strategies.
Organization(s): W.K. Kellogg Foundation
Date: 2017
Selecting
an Appropriate Evaluation Design
Document
This webpage provides background information on designing project evaluations, with descriptions of various
design-types and factors to consider when choosing an evaluation design.
Organization(s): Prevention Solutions@EDC
Date: 8/2015