UMass Donahue Institute

The UMass Donahue Institute’s Applied Research and Evaluation Unit has released a policy briefing summarizing findings of its three-year evaluation of the Massachusetts Department of Elementary and Secondary Education (ESE) Commonwealth Pilot Schools Initiative. The Initiative was a whole school reform initiative active from school years 2007-2008 through 2009-2010 for which the Institute served as statewide evaluator.

Intended to introduce substantive reform into schools struggling with persistently low student achievement, the Commonwealth Pilot Schools Initiative comprised a total of five schools in two cohorts. Each school had been in underperforming status for four years or more prior to its entry into the Initiative.

The policy brief presents critical Initiative-level findings and discusses their potential implications for the design, implementation, and management of similar whole-school reform initiatives. Major findings highlighted in the document include the following:

• Evidence collected through the evaluation suggests that the Initiative facilitated improvements in vision, culture and practice at participating schools, although the extent and scope of these outcomes varied considerably. Gains were typically incremental in nature and not sufficiently large to leverage dramatic changes in student achievement outcomes in the short-term. Although improvement was observed in MCAS results at some schools, when set in the context of trends at other underperforming schools, progress at Commonwealth Pilots was unremarkable at all but one school.

• These findings are not surprising when considered in light of implementation progress. Schools encountered varying degrees of success in the implementation of their Commonwealth Pilot redesign plans, and even where things went well, change proved a gradual process. In nearly all cases, intended changes with regard to curriculum, instruction, and assessment were still in the process of being refined and implemented at the time of the Initiative’s phase-out. A number of contextual factors appeared to influence the extent of implementation progress.
• The experiences of schools participating in the Commonwealth Pilot School Initiative emphasize the nature of change as a multi-step process that is highly customized, open-ended in its duration, and subject to contextual factors not entirely within the school’s control. This suggests the need for a phased and nuanced approach to school reform that extends from an objective assessment of school needs and contextual factors and allows for the development of the capacities, tools, processes needed to leverage meaningful and sustainable gains in student achievement.

Also available are interim evaluation reports, including those related to implementation and preliminary outcomes, as well as an in-depth examination of student achievement trends through the Initiative’s third and final year.  These may be found under the “Commonwealth Pilot Schools” heading of the Research Reports by Topic page of ESE’s Office of Strategic Planning, Research, and Evaluation site, located at http://www.doe.mass.edu/research/reports/topic.html.