New report documents progress by charter, pilot schools in closing student achievement gap

Boston – The Boston Foundation and the Massachusetts Department of Elementary and Secondary Education released a groundbreaking report by a team of Harvard and MIT researchers at an  Understanding Boston  forum today that effectively compares student performance at charter and pilot schools against students attending traditional public schools in Boston. The report uses an innovative two-part research design. One part uses observational data from every charter and pilot school in Boston, and the other examines the experience of a subset of students who took part in lotteries for admission. The result is, for the first time nationally, a rigorous, direct comparison of student performance in three different kinds of schools—charter, pilot and traditional.

According to the findings, charter school students consistently outperform their peers at pilot schools and at traditional schools. First launched in the Commonwealth in the wake of historic education reform legislation passed in Massachusetts in 1993, charter schools now stand as the first example of new educational strategies that have made a demonstrable improvement in the academic performance of their students.

The new study, titled Informing the Debate: Comparing Boston’s Charter, Pilot and Traditional Schools , was researched and prepared by a team led by Thomas Kane, Professor of Education and Economics at the Harvard Graduate School of Education, and Joshua Angrist, Ford Professor of Economics at MIT. Data used in the research was used provided by the Massachusetts Department of Elementary and Secondary Education.

“Fifteen years ago, the charter school idea was launched to see if new school models could lead to gains in student performance,” said Kane. “The results suggest that charter schools are making a significant difference.”

Pilot School student performance was more ambiguous, and varied according to research design.

Key findings
For elementary Pilot School students, a significant impact was seen in English Language Arts scores, but not for math scores.  In middle school, the observational results suggest Pilot School students may actually lose ground when compared to their peers in traditional schools, while the lottery-based results showed no difference between Pilot School and traditional school performance. At the high school level, observational results showed significant improvement of performance by both charter and pilot school students, compared to student performance in traditional schools. The lottery-based study, however, showed no significant difference between high school students in Pilot School and high school students in traditional schools.

Among other key findings of the report: the impact of charter schools was particularly dramatic in middle school math. The effect of a single year spent in a charter school was equivalent to half of the black-white achievement gap. Performance in English Language Arts also significantly increased for charter middle school students, though less dramatically. Charter students also showed stronger performance scores in high school, in English Language Arts, math, writing topic development, and writing composition. Students in pilot high schools also made measurable progress.

“This report speaks to the promise of education reform—and to its potential impact,” said Paul S. Grogan, President and CEO of the Boston Foundation. “Innovation in schools can help us to close the achievement gap that has remained, even as Boston’s schools have earned a reputation as among the best large urban systems in the country.”

Two separate research designs were incorporated into the project in part to address the common complaint that the charter school students are significantly different from their peers in the traditional public schools in ways that influence how they perform in school, and that this difference undermined any meaningful comparison between the two. In particular, the use of a lottery-based component enabled the researchers to compare the academic performance of students—one accepted and one rejected by means of an impartial lottery—to reduce differences such as the role that families typically play in their children’s schooling.

The report directly addresses two of the most frequent criticisms leveled at earlier studied of Pilot and Charter schools: that their students are not representative of traditional Boston schools but rather are more likely to succeed; and that charters and pilots tend to shed students who do not perform up to their standards, again creating an elite student body that will inevitably outperform their BPS peers.

“At the time of admission, the only difference between applicants who were offered admission and those who were not was a coin flip,” said Kane. “The fact that there are large differences in subsequent performance suggests that the charter schools were indeed having an impact.”

Findings for students in pilot schools were limited in their impact, compared to charter schools, in part because of the smaller population that fit the experimental model of the report.

Although the researchers employed both observational and lottery approaches, they acknowledge limitations remain. The observational study includes all schools but does not control for many hard-to-measure differences in students' background.  The lottery study controls for all differences in student background, including unobserved differences, but does not include all schools.

In addition, the control variables used in the study are necessarily limited.  For instance, they include only participation in special education and limited English proficiency programs – not the level of need.  It is possible that pilot and charter schools serve different populations within each of these programs; this could be particularly relevant for the observational study.

Finally, the authors stress that this study was designed to respond to the important question of whether different types of schools produce significant achievement gains, and not to explain why or how charters and pilots might have an impact on performance.  The Boston Foundation has announced its intentions of funding future studies that will address these questions.

“This by no means ends the debate about what schools can best serve our young people, but it points the way forward, underscoring the need for further scholarship,” said Grogan. “In the meantime, charters are generating extraordinary results and the Boston Foundation will continue to support Charter and Pilot schools and to support the scholarship that will extend our understanding of why they work.”

Charter/Pilot background
Charter schools are public schools that are chartered by the Massachusetts Board of Elementary and Secondary Education. Each school is managed by an independent board of trustees and is independent of local school committees. Charters come in two forms: Horace Mann Charters and Commonwealth Charters. Horace Mann Charters must be initially approved by a local school committee and by the local teachers union, while Commonwealth Charters apply directly to the state and do not need those local sign-offs. In addition, Horace Mann Charter employees continue to be members of the local collective bargaining unit, accrue seniority and receive, at a minimum, the salary and benefits set by the local unit.

In contrast, Commonwealth Charter employees are not required to be members of the local collective bargaining unit.

Today, 54 Commonwealth Charter Schools operate statewide, as well as seven Horace Mann Charter Schools. They serve a student population of 25,034, while fully 21,312 students remain on charter school waiting lists. Charters are distributed widely, with 16 in Boston, 25 in other urban settings and 20 not located in urban centers.

State law caps the number of charters at 120, which would allow for 48 Horace Mann Charters and 72 Commonwealth Charters.

Pilot schools were created by the Boston Public Schools and the Boston Teachers Union in 1995 to provide an alternative to traditional schools and to the newly created charter schools. They hold considerable autonomy over five areas: budget, staffing, governance, curriculum and the calendar. Each new pilot school must pass an approval process that includes a two-thirds vote by all school employees who are members of the Boston Teachers Union, as well as a vote by the school district. Teachers in a pilot school retain their seniority, and BPS pay scale provides a minimum for teacher pay.

In the current school year, 20 pilot schools are in operation in Boston, with seven new pilots scheduled to open by September of 2009. They serve a total of 6,337 students, 11 percent of the total Boston Public School population. Four pilot schools serve exclusively an elementary school population, four include elementary and middle school students, two are middle schools, one includes middle and high school students, and nine serve high school students alone.

******

The Boston Foundation, Greater Boston’s community foundation, is one of the oldest and largest community foundations in the nation.  In 2008, the Foundation and its donors made close to $79 million in grants to nonprofit organizations and received gifts of $113 million, bringing its assets to $838 million.  The Foundation is made up of some 900 separate charitable funds established by donors either for the general benefit of the community or for special purposes.  The Boston Foundation also serves as a major civic leader, provider of information, convener, and sponsor of special initiatives designed to address the community’s and region’s most pressing challenges.  For more information about the Boston Foundation, visit www.tbf.org or call 617-338-1700.