Program Theory
As mentioned before, The Friendship Circle has created a program called Lessons for life. This program’s goal is for students to learn and practice social, emotional, and cognitive life skills needed to meet the standards set in subjects established by federal and state mandates. Following the "If and then" statements described by Wilder Research (2009). The program theory for The Lessons For Life program evaluation is:
-If students learn and practice social, emotional, and cognitive life skills in a controlled, safe, and realistic environment during the Lessons for Life program, then they will lead to the desired outcome of succeeding and functioning in real life community settings.
-If teachers and trainers modify the lessons to improve the individuals objectives for each student, then students will learn the life skills they need to succeed.
-If students practice and learn in at The Friendship Circle, then they will be able to translate the concepts to real life situations.
Overall Program Theory
If students with special needs who have social, emotional, and cognitive life skills goals on their IEP complete the 17 lessons in the Lessons for Life curriculum, then students will be able to transfer the skills and practice what they learn in a controlled, safe, and realistic environment to a real functioning community setting.
Impact: If students complete the Lessons for Life program, then they will see improvements in their behavior, self-esteem, life skills, and social skills.
Data: Improvements in these skills will be tracked through their IEP goals as well as a third party data-driven evaluation to measure the learning that takes place at The Friendship Circle.
Logic Model
Additional Comments
​
Assumptions:
-Parents will help support and practice these life skills at home with their students after the program ends.
- Parents will be interested in attending information sessions, student growth meetings, and workshops for their students.
-When students transfer their skills or practice their skills in real life situations, other people will respect their failures if they occur.
- Students will maintain new cognitive, social, and emotional skills throughout adolescence and adulthood.
External Factors:
-
Preconceived notion that students will be able to successfully transfer skills to real life situations right away.
-
Students will have behavior issues that staff members can not control.
-
Unexpected change in funding for the program.
Theory of Change and Action Theory
-
Action theory: this explains how interventions are constructed to activate their theory of change in terms of the activities that will be undertaken and what level of success will be needed for each result to produce the final intended impact.
-
"A ‘theory of change’ explains how activities are understood to produce a series of results that contribute to achieving the final intended impacts. It can be developed for any level of intervention – an event, a project, a programme, a policy, a strategy or an organization.
​
Source: https://www.betterevaluation.org/en/node/5280
​
​
​
Model from Chen, 2005
Evaluation Approach
"Summative evaluation is concerned with designing and using evaluation to judge a program's merit (Shadish et al., 1991). Chen (1996, 2004) proposed a taxonomy of program evaluation, one built around the program stage that is the desired focus of the evaluation, as well as around the desired function of the evaluation (either improvement or assessment)."
​
(Chen, 2005 page 47)
Outcome Stage. The fourth stage of program growth is known as the outcome stage. Following a period of program maturity, stakeholders inside and outside the program want to know whether the program is achieving its goals. An evaluation at this point can serve any of five primary evaluation needs.
1. Stakeholders may rely on evaluators to determine if a program is ready for outcome evaluation. It may not be and, if it is not, evaluators may be asked for help in building the program's “evaluation capacity.”
2. Stakeholders may want to monitor their clients’ progress.
3. Stakeholders may ask for information on what the program would be achieving if it existed in the ideal environment. (Such information can also help stakeholders decide if a program should be expanded to other people or settings.)
4. Stakeholders may seek to know in detail the program's effects in its real-world setting because these, obviously, have a direct bearing on practice.
5. Some stakeholders may ask evaluators to go beyond traditional evaluation and its single-minded focus on assessment. They may want an evaluation that serves both accountability and program improvement needs.
​
(Chen 2005 page 51)
“Outcome evaluations (or impact evaluations) focus on the questions that ask what happened to program participants and how much of a difference the program made for them” (Boothroyd, Roger A.). Outcome evaluation is important because the feedback provides evidence for if the program was successful or not which can result in opportunities for more program funding from the stakeholders.
In order to evaluate a program using the outcome approach it is crucial to have clear objectives. This means identifying the specific changes that are being assessed for effectiveness. Meaningful indicators are needed to measure and show progress toward or away from the outcome and reliable data is needed to observe changes in the indicators (NSW Department of Education).
Summative Evaluations: Summative evaluations answer questions about program quality and impact for the purposes of accountability and decision making. They are conducted at a project’s or program’s end and usually include a synthesis of process and impact or outcome evaluation components.
I believe a summative evaluation approach for the Lessons for Life program would be best. A summative evaluation asks the question "Did it work?" It looks at the quality of the program and impact of the goals. Since this program is not going anywhere, established, and have been running for a while, an outcome or summative evaluation is appropriate to show the stakeholders and donors that this program is effective.
​
As Chen describes, there are five primary evaluation needs. Based on those needs of the steak holders, a Summative Evaluation to determine the success of the program is the best approach.
​
After speaking to the Lessons for Life program director, he explained that in 2015/2016 school year they started tracking data through a third party company. This is another reason for a summative evaluation because we can collect data to show and prove the clients progress and effectiveness of the program by using the new data tracker.
​
​
​
​
​
​
​
​
​
​
​
​
​
Finally, a summative evaluation would be able to show the stakeholders that the program is succeeding at The Friendship Circle and how they could expand the program (for example: teaching the use of technology since it is a norm now).
​
Data that will be collected for my summative evaluation will be quantitative (or numerical data) based off the third party testing program used and the number of IEP goals achieved from the program. Qualitative data will be also collected based off parent and student feed back. For example, surveys and feedback on how students feel transitioning their skills into the real world or success stories!
​
Data Collection and Plan for Program:
The program will collect short term data after each student exits the program as a success or failure. Since this is a summative or outcome evaluation, data will be reviewed at the end of one full calendar year. This will be able to give the stakeholders numerical data to show that the program is effective and hopefully help continue funding! All stakeholders and participants will be aware of the objectives, interventions, and outcome goals. Here is how data will be collected for each goal:
​
​
-If students learn and practice social, emotional, and cognitive life skills in a controlled, safe, and realistic environment during the Lessons for Life program, then they will lead to the desired outcome of succeeding and functioning in real life community settings.
This will be tracked by using the third party data program and data from completed social, emotional, and cognitive life skills IEP goals.
A comparison group will be used to help measure the changes in the program. A Comparison group will provide evidence that the program or intervention did in fact work.
-If teachers and trainers modify the lessons to improve the individuals objectives for each student, then students will learn the life skills they need to succeed.
This will be tracked by looking at the number of social, emotional and cognitive life skills IEP goals completed and number of students served.
-If students practice and learn in at The Friendship Circle, then they will be able to translate the concepts to real life situations.
Data will collected by parent and student surveys, feedback, and success stories for students transferring their skills to real life settings. (Example:
​
​
​
Sources:
-
Huey-Tsyh, C. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage Publications.
-
Program Theory and Logic Models: http://www.evaluatod.org/assets/resources/evaluation-guides/logicmodel-8-09.pdf
-
A Review of Program Theory and Theory Based Evaluations http://www.aijcrnet.com/journals/Vol_1_No_3_November_2011/10.pdf