The COVID 19 Pandemic has significantly impacted the delivery of programs and services for all children, especially children with disabilities. Moreover, it has impacted our ability to thoroughly monitor and evaluate the effectiveness of programs in meeting children’s needs. ECA has reconceptualized program evaluation to fit the changing demands imposed by the COVID-19 pandemic and our belief that a good evaluation draws from the best of implementation science.
If the COVID-19 crisis has taught us anything, it is that robust and credible monitoring and evaluation systems are critical to our ability to carry out high quality, effective intervention in the wake of unprecedented challenge. They provide tracking, real-time feedback, course correction and learning, all of which are vital to ensuring that programs deliver on their goals of improving the lives of our most vulnerable populations.
Teachers, schools, and systems of support will face unprecedented challenges when schools eventually reopen after pandemic-related closures. One of the central challenges will be figuring out how to meet the individual needs of students who had dramatically different experiences while schools were closed, and who will need different academic and nonacademic interventions and supports as schooling resumes.
Inclusive programs face unique challenges in structuring learning environments and implementing interventions and strategies that support the needs of all children. It is not always evident how the IEP accommodations and interventions for each child are integrated in classroom routines and activities and the extent to which they are effective in supporting students’ participation and engagement.
The value of monitoring and evaluation lies in the information it provides about the performance of a program: what works, what does not, and why. Implementation science is a viable way to identify and evaluate processes that must change and the nature in which change must occur to continue to support children’s needs.
Evaluations of inclusive programs vary in quality and often fail to answer the most pertinent and pressing questions school districts and other programs have regarding eligibility, service delivery and program placement, program implementation and how and why a program works or does not work for individual children with disabilities and for all children in inclusive settings.
It is impossible to thoroughly examine how a program is implemented and the degree to which children have access to and can engage in the program’s experiences and activities without being in the classroom to observe on multiple occasions. Harnessing technology to conduct observations both meets an immediate need during the COVID 19 epidemic and has important benefits beyond our current crisis. Between challenges imposed by the COVID-19 pandemic and those associated with quality evaluation of inclusive programs, now is the time to build back better and leverage technology and implementation science to reshape program evaluation as we know it.
ECA’s Approach
Our approach examines how interventions are implemented in real classroom conditions and how these practices influence program and child outcomes despite restraints imposed by the COVID-19 pandemic. It looks more closely at how programs are structured, interventions are implemented, and how programs align children’s individual IEP plans with best practices in educating young children.
ECA leverages technology to uniquely examine these elements using Torsh TALENT, a platform that combines observation, evaluation tools, and professional learning into one robust package without intrusive in-classroom visits.
Torsh TALENT enables us to capture and catalog all evaluation data and create digitized portfolios for managing and the entire evaluation process, including:
- Documents describing the program, its policies, and guidelines.
- Administrative surveys on strengths and challenges of the program.
- Teacher, specialist, paraprofessional and team chairperson surveys to capture perceptions of program implementation and impact on students.
- Observations of classrooms and other aspects of service delivery in action. Teachers record classroom videos, upload them, and share with the evaluator through an app on a mobile device such as an iPhone or iPad. The platform is both secure and complies with FERPA. The videos are time synced allowing evaluators to flag evidence of effective practices using a flagged text or audio comment rather than relying on a hazy recollection of onsite observation events. Ratings on an observation tool or rubric are based on the flagged text and can be used for classroom reflection.
- Evaluator ratings of programs using such tools as the Early Childhood Education EnvironmentIndicators of High-Quality Inclusion.
- Ratings on an Alignment checklist to determine the degree of alignment between IEP modifications and interventions and classroom practices for a select group of students enrolled in each classroom.
- Focus groups with teachers, leaders and other staff through virtual sharing circles or forums on specific evaluation topics.
Utilizing Torsh TALENT allows for seamless data tracking, analysis at classroom, program, school, or district level, and compelling data visualization in report presentation. Together, these features support the team in providing optimal technical assistance and recommendations for program improvement.
We are here to help your district ensure that the needs of all children in your inclusive programs are being met.
Please contact info@earlychildhoodassociates.com for further information.
We look forward to working with you!
About Early Childhood Associates
Early Childhood Associates, Inc. (ECA) is a nationally recognized, women-owned social science research firm located near Boston. ECA’s team includes researchers with PhDs and in-depth experience in advanced quantitative and qualitative research and evaluation techniques. ECA has expertise in virtually every mode of research and evaluation methodology applied to education, from child and adult assessments in a wide range of ethnic, cultural, linguistic, and socioeconomic populations, to environment and classroom observations, focus groups, interviews, online surveys of varying scale, longitudinal studies and webinars.