fact-check check

The NY Times tried to fact-check the mayor’s claims about Renewal test scores. Researchers say its analysis fell short.

PHOTO: Creative Commons / DMC Wilcox
The New York Times building

Reading the coverage of New York City’s Renewal program, it would be easy to conclude that the program isn’t working for most schools.

And on Thursday night, the New York Times continued in that vein, with a story about the turnaround program headlined: “For $582 Million Spent on Troubled Schools, Some Gains, More Disappointments.”

The story is framed as a fact-check on Mayor Bill de Blasio, who said at a press conference that Renewal schools are showing signs of progress since they had outpaced the city’s average growth in English and math scores.

In evaluating that claim, the Times points out that despite gains at some schools, most Renewal schools have actually not made progress closing the gap between their original scores three years ago and the city average. Some of the program’s fiercest critics seized on the analysis.

But according to three academics who study school performance, two of whom have studied the Renewal program’s impact, the Times’ characterization of the program as producing spotty results is problematic for the same reason de Blasio’s original claim of success doesn’t hold water. That’s because comparing Renewal test score data to city averages is poor evidence of whether the program is working.

The Times’ analysis can’t actually establish a causal effect.

The Times frames its analysis this way:

“To track the effects of the program, which gives schools a longer day and access to special services like vision care for students or mental health supports, The New York Times analyzed Renewal school performance on the 2016 and 2017 tests, as compared with the 2015 scores.”

The phrasing suggests that it’s reasonable to infer “the effects of the program” from test score changes, which is simply not possible, according to Thomas Dee, director of Stanford’s Center for Education Policy Analysis.

That’s because establishing a program’s effect depends on a model that can sort out what would happen to test scores without the program at all. One way to isolate that effect could be to compare low-performing schools that didn’t make it into the Renewal program with those that did, and study the difference in scores between the two groups of schools. But neither the education department nor the Times analysis attempted to do that, making those claims about the program’s impact misleading, experts said.

“There’s often this tacit assumption that we’ve learned about the true effect of the program from comparisons like this, and any researcher worth their salt will tell you that’s not the case,” Dee said.

Amy Virshup, a Times metro editor, defended the story’s analysis. “The piece never presumes to judge the success or failure of the Renewal program based on the ELA and math test results,” Virshup wrote in an email. “Judging whether Renewal is working or not would require many more data points and much more analysis.”

Renewal schools may be serving different students than when the program started.

Another reason the test scores could be misleading is that it has been well-reported that Renewal schools have lost a significant share of their students, continuing an enrollment drop-off that has persisted for years.

And since higher-performing students may be more likely to find a new school, it’s plausible that Renewal schools are serving a more challenging student body than when the program started.

“In general, the students that are most [likely to leave] are those who are higher performing,” said Marcus Winters, who wrote a report about the Renewal program for the conservative-leaning Manhattan Institute.

If that’s true, year-over-year test score comparisons wouldn’t be completely fair, since they would simply pick up changes in which students are served by Renewal schools — instead of the program’s real effect.

“The Times really didn’t do anything to ensure that their comparison to other schools was really comparable,” said Teachers College professor Aaron Pallas.

There’s no mention of rigorous research that has attempted to show causal effects.

Two researchers have tried to suss out whether the Renewal program is creating positive academic changes, and have reached different conclusions.

In an analysis that compares Renewal schools to similar ones that didn’t enter the program, Pallas found the program had essentially no effect on graduation rates or test scores. Meanwhile, using a different statistical model, the Manhattan Institute’s Winters found that the program is actually creating meaningful academic benefits.

The Times doesn’t cite either of those analyses, which would complicate the picture. The results of those research efforts suggest the Times’ description of mixed results is certainly plausible, but it isn’t directly supported by the data analyzed in the story.

Still, Winters said, the Times analysis is worth doing, as long as there are caveats, missing in this case, about what it can and can’t explain. “I don’t think it’s the definitive analysis of what’s going on,” Winters said. “But it’s not nothing.”

Highs and lows

Some Colorado schools brace for state intervention, while others cheer their progress

PHOTO: Nic Garcia

Two Colorado school districts and six individual schools failed to show enough improvement to raise their ratings under the state’s accountability system. Unless they successfully appeal their preliminary ratings this fall, they’ll remain on state-mandated improvement plans. Those include the 7,500-student Adams 14 district based in Commerce City, which is likely to face additional state intervention this school year, and Aurora Central High School.

And two new schools also face the potential for state intervention, Central Elementary School in Adams 14 and Minnequa Elementary in the Pueblo 60 district.

One school district and six individual schools that had previously faced state intervention improved enough to get off Colorado’s “accountability clock,” according to preliminary school ratings released Monday.

“Our students are working harder than they ever have, and it’s making a big difference,” said Superintendent Deirdre Pilch of the Greeley-Evans district north of Denver. In that district, two schools on state improvement plans and four others that had been placed on warning did well enough to get out of state scrutiny. “By every measure we use, we are seeing students move in the right direction. I am so proud of people locking arms and coming together to do that work.”

Colorado’s school accountability system rates districts based on achievement on state literacy, math, and science tests, on annual academic growth, and on postsecondary readiness as measured by graduation rates, dropout rates, scores on college entrance exams, and enrollment in college.

Schools go on performance watch or “on the clock” if their rating places them in one of the lowest two tiers – turnaround or priority improvement – and they face state intervention if they don’t move into a higher tier after five years.

The ratings released Monday are considered preliminary. Districts can request that the state reconsider districtwide or school ratings based on, for example, progress in literacy and math in the early grades or measures of high school achievement that don’t show up on state tests.

The deadline to file a request to reconsider is Oct. 15. The ratings will be finalized in December.

The State Board of Education has four options when deciding the future of schools whose performance remains in the two lowest tiers of the five-point scale for years on end. The board can close the school, hand it over to a charter management organization, contract with a third party to help run the school or create an innovation plan that spells out strategies and exemptions from district and state policy to improve student learning.

The options for districts are similar but include drastic and politically challenging steps like pushing for merger with a higher-performing district.

So far, the State Board has taken a collaborative approach and largely approved the plans that schools and districts brought forward. At the same time, state lawmakers, at the request of the Colorado Department of Education, approved changes to require schools to show more sustained improvement to get off performance watch and encourage schools to take action earlier in the process.

Colorado doesn’t have the option of state takeover that’s been exercised in places like New Jersey and Tennessee, with mixed results.

The next few years will serve as an ongoing test of how well this system of carrots and sticks works to help long-struggling districts.

“We are excited to see the progress made in some schools around the state to improve student performance, especially for some of those that have persistently struggled,” said Alyssa Pearson, the Colorado Department of Education’s deputy commissioner for accountability and performance, in an email. “These schools and their districts have had a laser-like focus on the needs of students. They have done this through two to three high-leverage priorities around data-driven instruction, leadership development, and culture to better meet the needs of all students. The specific actions vary, depending on the local context and need.”

Perhaps one of the best examples of the process working is the Greeley-Evans district north of Denver. When Pilch arrived in 2015, 10 schools were on the accountability clock, and three of them ultimately required state-approved improvement plans: Prairie Heights Middle Schools, Franklin Middle School, and Martinez Elementary.

Franklin Middle School got off the clock last year, and with this round of ratings, Prairie Heights and Martinez Elementary were also freed from state supervision. Four additional Greeley-Evans schools came off performance watch this year without outside intervention.

Three Greeley-Evans schools, though, will be on the state’s watch list if their ratings don’t change in appeals.

Pilch said the district poured resources and support into the schools that needed to improve and took advantage of state grants for leadership training and other professional development. The district received extensive reviews of what was and wasn’t working within its schools from state evaluators and took those findings seriously in crafting improvement plans.

“The schools that are on priority improvement and turnaround, they’ve been our priority in terms of support and in terms of ensuring the right resources and the right leaders are in place,” Pilch said. “We’ve been very intentional about maintaining and training quality leaders at these schools. We’ve seen fewer leaders turn over.”

Also showing improvement was the Westminster district, which had one more year to make progress on its plan and is now off the clock. Officials there claimed vindication for the district’s competency-based learning model, in which students are grouped by their understanding of a certain subject and can progress to another level as soon as they show that they’ve mastered that class content.

Results for Aurora schools, where district officials have been using new reforms to intervene in low-performing schools, have been mixed. The district as a whole improved enough to dodge state action last year. This year’s rating stayed at “improvement.”

The school with the most years of bad ratings, Aurora Central High School, failed to improve. The school is already on a state plan for improvement and has a year left to earn a higher rating before it would have to return to face the state again. Previously, state officials essentially blessed a district plan to continue rolling out interventions it was already trying, with extra help from an outside partner. If the school has to return to the state, officials could take more drastic action.

Three schools that had earned the lowest rating of turnaround last year — including Lyn Knoll Elementary — improved. Paris Elementary, which was facing state intervention this year if it didn’t raise it’s Priority Improvement rating, also managed a higher score, to avoid state sanctions.

If final school ratings remain the same, no other Aurora schools this year would be as close to state action. Three schools — Gateway High School, North Middle School and Virginia Court Elementary — would be entering year four, meaning they would have one year to show improvement before being at risk for state intervention.

“Aurora Public Schools continues to see gains and movement in the right direction,” district officials said in an emailed statement. “We have some great momentum that we will continue to build upon. While we recognize that we need to make more improvements at faster rates, we will dig into our data to plan how we will best leverage our strengths and address our challenges.”

District officials said the results from schools in what the district calls its Action Zone – schools that have individual plans for some flexibility from district rules – show the district has the right structure in place to keep improving. However, two schools in that zone, Aurora Central and Boston K-8, did not improve within the state framework and Boston actually earned a lower state rating this year, though it is not on the clock.

The tiny Sheridan district south of Denver, which got off the clock in 2014 after years of effort, was rated in the second-lowest tier, putting the district back on performance watch. Pat Sandos, the new superintendent there, said the news was a hard way to start his tenure, but it also brings a sense of urgency to improving student performance.

“We’re looking at the data really hard and breaking out the content areas,” he said. “We see opportunity for growth with [English-language learners]. It seems like that’s something that’s not just us, but for a lot of at-risk districts.

“Some of the work that we’re already doing is realigning and looking at the curriculum. We’re really focusing hard on that, on establishing a framework that gets teachers focused on what kids need to know at each grade level. And we’re paying a lot of attention to professional development.”

Other schools that got off the clock after facing state intervention include Hope Online Academy Middle School, authorized through the Douglas County school district, Manaugh Elementary School in the Cortez-Montezuma district in southwest Colorado, and Bessemer Elementary School in the Pueblo 60 district.

Two other Pueblo 60 schools still face state supervision: Risley International Academy of Innovation and Heroes Middle School. So does the little Aguilar district in southern Colorado, where the both the district and its joint junior-senior high school have struggled with years of low performance, and Hope Online Learning Academy Elementary School.

The ratings will be finalized by December, and schools that have spent six years or more in turnaround or priority improvement status will face new or ongoing state intervention.

Editor’s note: This story has been updated to include comment from Colorado Department of Education Deputy Commissioner Alyssa Pearson. 

Follow the money

New York City’s finance watchdog demands answers on $600 million school turnaround program

PHOTO: Alex Zimmerman
City Comptroller Scott Stringer.

The city’s top financial watchdog didn’t wait even a week before pressing Chancellor Richard Carranza on whether the “Renewal” school turnaround program is living up to its nearly $600 million price tag.

“While some Renewal schools have shown improvements,” Comptroller Scott Stringer wrote in a letter to Chancellor Richard Carranza, “inconsistent progress across all Renewal schools suggests the need for a more thorough review of the program’s components and their overall impact.”

The letter, sent just three days after Carranza officially took office, asks for a detailed accounting of how Renewal schools spent money on core elements of the program, including teacher training and extending the school day for an hour — as well as any evidence that those efforts are paying off or being monitored. Two independent evaluations by outside researchers suggest the program has produced only mixed results.

Stringer’s letter appears to be motivated at least in part by a recent round of hotly contested school closures. Since the program’s launch in 2014, 16 of 94 original Renewal schools have been merged or closed. (Another 21 schools are slowly easing out of the program after city officials said they made enough progress.)

“With the decision to now close schools that have not made sufficient progress,” Stringer wrote, “I question whether there have been adequate direction and accountability measures in place to ensure that all school received allocations with sufficient time to show progress, and were directing new resources to high impact programs and interventions.”

Stringer’s letter came just weeks before Carranza began raising his own questions about the Renewal program, which gives long-struggling schools extra academic support and social services. In an interview with Chalkbeat, the new schools chief said the Renewal program did not appear to have a single clear “theory of action.”

The comptroller’s probe also comes at a precarious moment for the program: It is without a permanent leader and it’s also unclear whether the city will phase out or reconfigure it. (Carranza told Chalkbeat he is committed to running a turnaround program of some kind.)

Stringer also touched on a number of other aspects of the program that have drawn criticism from school communities, including how the city identifies which schools should be closed and how the education department helps families find new schools.

According to the comptroller’s letter, multiple schools that met the exact same number of city benchmarks received different decisions about whether they should be closed.

While Stringer acknowledged that the city conducts a holistic review in making closure decisions, “the lack of transparency about these additional factors and how school closure decisions are made is breeding needless distrust in communities.”

An education department spokeswoman, Toya Holness, said the department is “reviewing the comptroller’s letter and will provide a formal response.”