top of page
This site was designed with the
.com
website builder. Create your website today.Start Now

Measuring Success: Evaluating Program Outcomes

The most effective programs are able to track and demonstrate measurable improvements in public safety, mental health, substance use, school engagement, and any number of other outcomes related to child and family well-being. Through the use of ongoing data collection and evaluation, jurisdictions can see which programs have the greatest impact on youth and, therefore, should be sustained, expanded, replicated or discontinued.

Step 1: Determine the Purpose of the Evaluation

The type of evaluation will vary depending on the program's goals and objectives, needs of the participants, or requirements of the funder.  

  • What do you hope to learn from the evaluation? 

  • Does the program budget allow funds for hiring an evaluation expert? (This is highly recommended). 

  • Is there a someone on staff who possesses the specific skills to manage all phases of the evaluation? 

     

Step 2: Define Program Outcomes and Indicators

Outcomes: Describe what the program was designed to achieve (1). They must be well-defined, observable, and logically connected to the program's goals and objectives. Outcomes focus on short-term changes that occur after the program has been completed. 

Know the Terms

Outcome Evaluations: Assess the effectiveness of a program in producing the desired change. This method is used to determine what happened to participants and how much of a difference the program made for them. They also attempt to eliminate the possibility that changes were the result of something other than the program. 

 

Process Evaluations: Describe how a program's activities, policies, and procedures are implemented. Process measures, or "output" data, describe who received the services, what they received, and "how much" of the service was provided. 

 

Sabatelli, R., Anderson, S. and LaMotte, V. (2005)

Impacts: Address long-term improvements in the quality of life of participants or others. Generally speaking, immediate outcomes are much easier to document than are long-term impacts. Researchers generally design evaluations to measure the following types of outcomes (2). 

 

Outcome indicators: Refer to the specific measurement that will be used to quantify each targeted outcome. In selecting which indicators to measure, it is important, to begin with a baseline – the indicators each youth possessed before the program started - and demonstrate how these have changed as a result of the program.

​

Knowledge: What new information did participants learn?

Attitudes: Did the program change the way participants think or feel?

Beliefs: Did the program change the values participants hold? 

Behaviors: Did the program change the way participants act towards others or help them develop new skills?

Step 3: Collect, Analyze, and Report the Data

Data collection is the process of tracking the indicators. The way data is collected will differ based on the goals of the program and the type of evaluation being conducted. 

 

  • In what ways do you plan to collect data? Personal interviews, document or records review, questionnaires, or observer ratings. 

  • Does your program design include pre-tests and post-tests? (Pre- and post-tests compare indicators before the intervention began and after it was completed.)

  • Does your program design compare an experimental group (those involved in the program) to participate in a control group not involved in the program? (This is the most rigorous type of evaluation design.)

 

Data analysis is the process of interpreting the data. The analysis allows the evaluator to “tell the story” of the program by compiling all of the data and removing identifying information about particular people.

 

  • Are the outcomes a direct result of the activities of the program?  How do you know that other factors were not major contributors to the outcomes or that the changes did not happen by chance?

  • How can the cost-benefit ratio of the program be measured?

 

Data reporting is the process of sharing information in a clear and concise format among key stakeholders. Generally, this includes the preparation of a final report.

 

  • Who are the audiences that will be interested in receiving an evaluation report? How will this evaluation be shared with youth, family, and staff?

  • What is the most useful format to share information with these audiences? How frequently will information be shared? Monthly, quarterly, annually?

Step 4: Use Data to Drive Program Improvements

Data-driven decision-making is widely recognized as a critical element of evidence-based practices.  By correctly understanding the data, practitioners can confirm the needs of the target population, have rationale for proposed changes, and demonstrate the importance of ongoing funding.  It is important to establish a plan for how the data will be used to drive program improvements.

 

  • Who is responsible for reviewing the data and recommending program improvements?

  • What process is in place to help decide when to expand, replicate or discontinue the program? How are changes communicated to staff and participants?

  • How are you using feedback from youth, families, and staff to drive program improvements?

Step 5: Develop a Plan for Sustainability 

The true test of any project is to ensure that its successes are achievable over the long run. When planning for sustainability, most people immediately think of funding needs. While funding is important, it is only one aspect to long-term sustainability.

How have you incorporated evidence-based principles into current policies and procedures?

This will ensure that staff becomes accustomed to these techniques simply as the way you do business.

 

To what extent have you increased partnerships with state government and other service sectors in order to share and leverage resources?

Collaborative endeavors may lead to opportunities to apply for additional federal and private funds with partners in the fields of workforce development, mentoring, education, and mental health.   

 

In what ways have you reinvested cost-savings into prevention and community-based services?

Because community-based programs cost less than out-of-home placement, jurisdictions should realize a net cost-savings by increasing community options.  Reinvesting the savings into prevention or in-home care services will further drive down costs.

 

What systems-level improvements have you made in support of the program?

Purchasing an automated case management system, upgrading an information management system, or establishing information-sharing agreements help establish an infrastructure for supporting community-based programs.

 

To what extent have you built community support and “buy in?”

Community members will want to know if your program is improving outcomes related public safety, education and mental health functioning. Don’t be afraid to publicize your success and build support for your program by distributing a fact sheet to your county commission or notifying your local news.

 

What are other sources of funding available to support the sustainability of your program?

Additional fundraising will likely be necessary; but it will be bolstered by being able to present successful program outcomes, positive feedback from youth and families, and support from community partners

1. Sabatelli, R., Anderson, S. & LaMotte, V. (2005, September). Assessing Outcomes in Child and Youth Programs: A Practical Handbook, Revised Edition. University of Connecticut School of Family Studies Center for Applied Research and State of Connecticut Office of Policy and Management. Retrieved from http://www.ct.gov/opm/LIB/opm/CJPPD/CjJjyd/JjydPublications/ChildYouthOutcomeHandbook2005.pdf

2. Ibid.

bottom of page