Monitor and Evaluate Your Program

Monitoring occurs continually during a program. Evaluation typically occurs at 1 point, usually the midpoint or an end point.

Monitoring helps you:

  • Investigate how well a program adheres to implementation standards.
  • Learn how participants changed from the start of the program to the end.
  • Discover how program staff can make improvements to the program delivery.
  • Show funders how program implementation has progressed in relationship to the contract or expectations associated with funding.

Prevention researcher and program developer Andrea Taylor believes establishing a connection with the funding agency is important for any program developer.

One of the things I’ve always wanted any funding agency to do is to require a letter from the developer  of the model programs so, when they apply for funds, the funding agency knows that they’ve had a conversation. I have found that those people who called me in advance as they were writing their proposals a) submitted much better and stronger proposals and b) if they [got] funded, had a much easier time in implementing the program.

Evaluating helps you:

  • Learn about the program’s impact in relation to its original goals and objectives, as well as in relation to a meaningful comparison group.
  • Support program replication, if the findings show positive impact.
  • Discover how to modify the program, if the findings show negligible or harmful effects.
  • Provide results that can be shared and used to generate interest from other funders.

Note that a process evaluation assesses how well a program was put into place. This type of evaluation can be an important part of implementing and sustaining a program.

For example, by employing a research team from Temple University, Dr. Taylor found her program was meeting its original goals.

 

 

Table 1. Differences Between Program Monitoring and Evaluation

Type of Questions Answered

Program Monitoring

Program Evaluation

Target Population

What were the characteristics of the people served by the program?

How many people were served by the program?

How did the provided services differ for different subgroups in the program?

How well did the program reach the targeted population? What types of individuals were overrepresented or underrepresented?

How well did the program engage and track the desired number of participants? Was attrition high? Was the amount of missing data low? What was the overall level of participation?

Did attrition, missing data, or program participation vary for different subgroups?

Implementation Activities and Process

How was the program implemented in relation to the proposed design, implementation standards, and contractual specifications?

How did program implementation vary for different sites, target populations, phases, or time periods?

What can be learned from program implementation to improve service delivery?

What are the implications of program implementation for the analysis of program outcomes?

How did variation in program implementation affect outcomes achieved by participants?

Based on the degree and type of program implementation, what effects were achieved by participants in relation to a meaningful comparison group?

Desired Outcomes for Participants

What outcomes did the program achieve? Did the program achieve its goals? Were there any unintended consequences?

How well did the program produce the desired changes for program participants?

How did program outcomes vary by level of participation? For various subgroups?

How did effect sizes vary for the outcomes measured, in relation to a meaningful comparison group?

What are the implications of the results that were demonstrated by the program—that is, how should the effect sizes be interpreted?

What factors may explain differences achieved in desired program outcomes for all participants? For different subgroups?

Cost–Benefit Analysis

What were the costs per participant in relation to the original funding estimate? Did the program stay within budget?

What is the dollar value of the benefits and outcomes achieved by program participants (e.g., reduced hospitalization cost, reduced treatment cost, or increased cost savings from reduced incarceration)?

In relation to the standard care group, what benefits were obtained by program participants and for what cost?

Compared with the dollars spent serving the control group and the outcomes achieved, what was the degree of impact for program participants?

Did the benefits of the program outweigh the costs? How did costs and benefits vary for different subgroups?

Last Updated: 06/15/2017