Data Collection Methods and Analysis Strategies
As a reminder, a summative evaluation will be done to show the stakeholders that the program is succeeding at The Friendship Circle and how they can expand the program (for example: teaching the use of technology since it is a norm now).
​​
I will be using both quantitative and qualitative data to assess the Lessons for Life program. Based on the Rainbow Network Better Evaluation, I will be gathering both qualitative and quantitative data at the same time, also referred to as Parallel Data Gathering. My purpose for collecting and combining both types of data is, “Enriching: using qualitative work to identify issues or obtain information on variables not obtained by quantitative surveys” (Better Evaluation, 2019). This chart should give you a quick understanding of how I will be collecting and retrieving my data, my sources, and my type of data.
​
​
My evaluation questions and corresponding data collection methods are:
-
How many students successfully transfer out of the “Lessons for Life” program each year? We will examine the official statistics (number) of how many students enter the program and transfer out with completed IEP goals.
-
How is success being defined or evaluated for this program? We will look at the number of IEP goals each student is completing when they are entered in the Lessons for Life Program.
-
What type of training do the volunteers go through to teach the curriculum? This data will be collected by asking the program for their training manual and feedback from the volunteers on if they feel prepared and supported. (Although this may not lead to the success of an outcome goal, it will help us determine where the program can improve!)
-
Do families and students feel they can transition these skills into the real world once they transition out of The Friendship Circle "Lessons for Life" program? What are the positive and negatives of the program? We will gather this data through non-participant observation and parent/participant feedback.
-
What is the average cost to run this program at The Friendship Circle? This data we will gather from existing records from the Friendship Circle!
-
What other local programs are competing with “Lessons for Life?” This data we will gather from surveys to discover if there is other programs in the area. We will also complete our own research in finding out this data.
-
How can this program reach or serve more students and families in the community? We will collect this data through surveys and interviews.
How will I analyze the data I collect?
Once my data is collected, I will summarize it and look for patterns. Since I have both quantitative and qualitative data, I will be using a few different analysis approaches based off the Rainbow Network Better Evaluation to report my findings.
​
1. Frequency tables: a visual way of summarizing nominal and ordinal data by displaying the count of observations (times a value of a variable occurred) in a table.
I will use Frequency Tables to show my quantitative data. Frequency tables will show how many IEP goals were met, how many students successfully transferred their skills to a real life setting, how many students successfully transferred out of the "Lessons for Life" program. This data will visually be presented in an Icon Array.
​
​
2. Summary statistics: providing a quick summary of data which is particularly useful for comparing one project to another, before and after.
This method for analyzing data will be used to to compare students who completed the program to the comparison group (students in that school district who did not complete the program). The data analyzed will also be put into a Deviation bar graph​ to compare the shape of data sets from both groups.
​
​
3. Content analysis: reducing large amounts of unstructured textual content into manageable data relevant to the (evaluation) research questions.
A content analysis will be done to use interviews, surveys, questionnaires, and observations to help answer the evaluation questions and support the outcome goals.
Statement on how I will ensure that my evaluation will have an impact on the program.
If students with special needs who have social, emotional, and cognitive life skills goals on their IEP complete the 17 lessons in the Lessons for Life curriculum, then students will be able to transfer the skills and practice what they learn in a controlled, safe, and realistic environment to a real functioning community setting.
​
With the program theory in mind, my summative evaluation will have an impact on the program because the data collected and analyzed will show an increase in students social, emotional, and cognitive life skills improving. To ensure that my evaluation proves this statement, quantitative data will be reported to show that students who participate in the Lessons for Life program successfully complete more IEP goals than students who do not complete the program. Quantitative data will also show that students are transferring the skills learned in the program to a real life situation. Qualitative data will report that students, parents, and community members are finding this program to have a positive impact on students social, emotional, and cognitive life skills. The data collected and reported will help to ensure that stakeholders continue to donate to the program and continue funding it. The evaluation will also prove to the community that this program is effective and the program will increase its number of students reached. The data collected from this evaluation will also help increase the buy-in of more volunteers because they will see the impact that this program has on students. Overall, an evaluation on the Lessons for Life program will ensure that community members, stakeholders, and participants know that this program is achieving its outcome goals and effective!
Reporting Stategies and Methods
As previously stated, I will be analyzing my data and using visual methods to report my findings (see above!). The visual method will allow the community and stakeholders to quickly and visually read the data.
Picture from: Step-by-Step Guides for Conducting a Summative Evaluation In Evaluator's Handbook
Based on the Rainbow Framework Better Evaluation article "7 Strategies to improve evaluation use and influence" by Patricia Rogers (2018), 3 strategies will be used to improve evaluation and influence.
​
1. Anticipate barriers to use. "Many barriers to use have been identified, including the credibility and perceived relevance of the evaluation report(s), the resources and authority to make changes in response to findings, and the openness to receiving negative findings (that a program doesn’t work or isn’t being implemented as intended)."
​
2. Choose appropriate reporting formats and ensure accessibility. "Making the right choices can increase the likelihood that they will know about the findings and understand what they mean and why they are important."
​
3. Actively and visibly follow up what happens after the evaluation
​
Program Evaluation Standards
The Lessons for Life Program Evaluation will adhere to the Standards for Program Evaluation based off The Joint Committee on Standards for Educational Evaluation as listed below:
Utility Standards
“The utility standards are intended to ensure that an evaluation will serve the information needs of intended users.”
The stakeholders at the Friendship Circle will be identified and their needs for evaluating the Lessons for Life program will be addressed. I will conduct the evaluation in a trustworthy and honest fashion so that the data reported is credible. The information and data collected will address the program questions and outcome goals. The program evaluation will report the findings with clear judgments of all perspectives. The report will be clear and easily understood explaining the context, purpose, procedures, and findings. The report will be presented in a timely manner.
Feasibility Standards
“The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.”
The interventions and procedures during the evaluation will not cause a disruption to other programs and events occuring at the Friendship Circle. I will focus on only collecting the data needed for the evaluation. The Lessons for Life program evaluation will plan for the bias opinions and positions of all interest groups. The intervention, tools used, and program evaluation will be cost effective and resources used to produce the evaluation will be justified.
Propriety Standards
“The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.”
All parties involved in the evaluation will sign a formal agreement so that participants are to adhere to the interventions during the evaluation. All positive and negatives of the evaluation will be completed to help build the program and also strengthen it. All conflicts of interest will be dealt with openly so it doesn’t not hinder the results of the evaluation. All allocations and spending that take place during the evaluation will be appropriate and ethically responsible.
Accuracy Standards
"The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated."
The Lessons for Life program will be described and documented clearly as well as the program outcomes. These will be identified, as well as, the purpose and procedures. The program will undergo an intense evaluation so influences on the program can be identified. All information gathered and used will be in its proper form and interpreted for its intended context. Errors in information will be identified, found, and corrected. All quantitative and qualitative information will be analyzed and used appropriately so the evaluation questions are answered. The stakeholders will be able to assess the conclusions and all results will be reported regardless of personal feelings and biases. This will reflect a fair and accurate evaluation.
Source:
​The Program Evaluation Standard - Summary of Standards
Drawn from: The Joint Committee on Standards for Educational Evaluation, James R. Sanders, Chair(ed): The Program Evaluation Standards, 2nd edition. Sage Publication, Thousand Oaks, USA, p.23-24; 63; 81-82;125-126 (see www.wmich.edu/evalctr/jc/)