fact-check check

The NY Times tried to fact-check the mayor’s claims about Renewal test scores. Researchers say its analysis fell short.

PHOTO: Creative Commons / DMC Wilcox
The New York Times building

Reading the coverage of New York City’s Renewal program, it would be easy to conclude that the program isn’t working for most schools.

And on Thursday night, the New York Times continued in that vein, with a story about the turnaround program headlined: “For $582 Million Spent on Troubled Schools, Some Gains, More Disappointments.”

The story is framed as a fact-check on Mayor Bill de Blasio, who said at a press conference that Renewal schools are showing signs of progress since they had outpaced the city’s average growth in English and math scores.

In evaluating that claim, the Times points out that despite gains at some schools, most Renewal schools have actually not made progress closing the gap between their original scores three years ago and the city average. Some of the program’s fiercest critics seized on the analysis.

But according to three academics who study school performance, two of whom have studied the Renewal program’s impact, the Times’ characterization of the program as producing spotty results is problematic for the same reason de Blasio’s original claim of success doesn’t hold water. That’s because comparing Renewal test score data to city averages is poor evidence of whether the program is working.

The Times’ analysis can’t actually establish a causal effect.

The Times frames its analysis this way:

“To track the effects of the program, which gives schools a longer day and access to special services like vision care for students or mental health supports, The New York Times analyzed Renewal school performance on the 2016 and 2017 tests, as compared with the 2015 scores.”

The phrasing suggests that it’s reasonable to infer “the effects of the program” from test score changes, which is simply not possible, according to Thomas Dee, director of Stanford’s Center for Education Policy Analysis.

That’s because establishing a program’s effect depends on a model that can sort out what would happen to test scores without the program at all. One way to isolate that effect could be to compare low-performing schools that didn’t make it into the Renewal program with those that did, and study the difference in scores between the two groups of schools. But neither the education department nor the Times analysis attempted to do that, making those claims about the program’s impact misleading, experts said.

“There’s often this tacit assumption that we’ve learned about the true effect of the program from comparisons like this, and any researcher worth their salt will tell you that’s not the case,” Dee said.

Amy Virshup, a Times metro editor, defended the story’s analysis. “The piece never presumes to judge the success or failure of the Renewal program based on the ELA and math test results,” Virshup wrote in an email. “Judging whether Renewal is working or not would require many more data points and much more analysis.”

Renewal schools may be serving different students than when the program started.

Another reason the test scores could be misleading is that it has been well-reported that Renewal schools have lost a significant share of their students, continuing an enrollment drop-off that has persisted for years.

And since higher-performing students may be more likely to find a new school, it’s plausible that Renewal schools are serving a more challenging student body than when the program started.

“In general, the students that are most [likely to leave] are those who are higher performing,” said Marcus Winters, who wrote a report about the Renewal program for the conservative-leaning Manhattan Institute.

If that’s true, year-over-year test score comparisons wouldn’t be completely fair, since they would simply pick up changes in which students are served by Renewal schools — instead of the program’s real effect.

“The Times really didn’t do anything to ensure that their comparison to other schools was really comparable,” said Teachers College professor Aaron Pallas.

There’s no mention of rigorous research that has attempted to show causal effects.

Two researchers have tried to suss out whether the Renewal program is creating positive academic changes, and have reached different conclusions.

In an analysis that compares Renewal schools to similar ones that didn’t enter the program, Pallas found the program had essentially no effect on graduation rates or test scores. Meanwhile, using a different statistical model, the Manhattan Institute’s Winters found that the program is actually creating meaningful academic benefits.

The Times doesn’t cite either of those analyses, which would complicate the picture. The results of those research efforts suggest the Times’ description of mixed results is certainly plausible, but it isn’t directly supported by the data analyzed in the story.

Still, Winters said, the Times analysis is worth doing, as long as there are caveats, missing in this case, about what it can and can’t explain. “I don’t think it’s the definitive analysis of what’s going on,” Winters said. “But it’s not nothing.”

Follow the money

New York City’s finance watchdog demands answers on $600 million school turnaround program

PHOTO: Alex Zimmerman
City Comptroller Scott Stringer.

The city’s top financial watchdog didn’t wait even a week before pressing Chancellor Richard Carranza on whether the “Renewal” school turnaround program is living up to its nearly $600 million price tag.

“While some Renewal schools have shown improvements,” Comptroller Scott Stringer wrote in a letter to Chancellor Richard Carranza, “inconsistent progress across all Renewal schools suggests the need for a more thorough review of the program’s components and their overall impact.”

The letter, sent just three days after Carranza officially took office, asks for a detailed accounting of how Renewal schools spent money on core elements of the program, including teacher training and extending the school day for an hour — as well as any evidence that those efforts are paying off or being monitored. Two independent evaluations by outside researchers suggest the program has produced only mixed results.

Stringer’s letter appears to be motivated at least in part by a recent round of hotly contested school closures. Since the program’s launch in 2014, 16 of 94 original Renewal schools have been merged or closed. (Another 21 schools are slowly easing out of the program after city officials said they made enough progress.)

“With the decision to now close schools that have not made sufficient progress,” Stringer wrote, “I question whether there have been adequate direction and accountability measures in place to ensure that all school received allocations with sufficient time to show progress, and were directing new resources to high impact programs and interventions.”

Stringer’s letter came just weeks before Carranza began raising his own questions about the Renewal program, which gives long-struggling schools extra academic support and social services. In an interview with Chalkbeat, the new schools chief said the Renewal program did not appear to have a single clear “theory of action.”

The comptroller’s probe also comes at a precarious moment for the program: It is without a permanent leader and it’s also unclear whether the city will phase out or reconfigure it. (Carranza told Chalkbeat he is committed to running a turnaround program of some kind.)

Stringer also touched on a number of other aspects of the program that have drawn criticism from school communities, including how the city identifies which schools should be closed and how the education department helps families find new schools.

According to the comptroller’s letter, multiple schools that met the exact same number of city benchmarks received different decisions about whether they should be closed.

While Stringer acknowledged that the city conducts a holistic review in making closure decisions, “the lack of transparency about these additional factors and how school closure decisions are made is breeding needless distrust in communities.”

An education department spokeswoman, Toya Holness, said the department is “reviewing the comptroller’s letter and will provide a formal response.”

 



closures ahead

As New York City prepares to close more struggling ‘Renewal’ schools, here’s what we know about ones they’ve shuttered before

PHOTO: Alex Zimmerman
Monroe Academy for Visual Arts and Design, a Renewal school, was closed last year.

In the coming days, struggling schools in the city’s “Renewal” improvement program will learn whether they get more time to mount a comeback — or will be shut down for good.

New York City education officials are expected to announce soon which of the low-performing schools will close at the end of this academic year. The decision will have enormous consequences for students and teachers who will have to find new schools — and will likely rekindle debate about the effectiveness of Mayor Bill de Blasio’s $582 million effort to turn around troubled schools by infusing them with social services and academic support.

As the Renewal program passed its third birthday in November — a date by which the mayor promised to decide which schools aren’t measuring up — officials have been tight-lipped about which schools are on the chopping block.

Chalkbeat analyzed the previous rounds of closures — nine schools out of the original 94 — to understand which schools might be targeted this time. Perhaps the clearest finding is that it’s difficult to predict which schools the city will shutter.

While all the closed schools had very low graduation rates and test scores, so do other Renewal schools that were spared. The analysis shows there are no strict rules about which schools are shut down and which are given more time to turn around.

That said, here are some takeaways from previous Renewal school closures:

Almost all the closed schools struggled to retain students.

Seven of the nine closed schools enrolled fewer students in the year they were shuttered than when they entered the Renewal program in 2014 — and six shed more than a fifth of their students.

Many of the schools had struggled to recruit and retain students even before the program started — once it did, the schools struggled to staunch the flow. The city considers such shrinkage an existential problem; officials have suggested that schools with fewer than 250 students can become unsustainable since school funding is based partly on enrollment.

Six of the closed schools enrolled fewer than 200 students their final year — including Brooklyn’s M.S. 584, which lost about 25 percent of its population since the Renewal program started, leaving it with just 78 students.

Meeting the city’s goals doesn’t guarantee survival.

The city assigned each Renewal school annual goals around attendance, graduation rates, test scores, and other measures. (The goals have been criticized as overly modest.)

In the past, officials have said “all options are on the table” — including closure — for schools that fail to meet their goals. But those the city has actually shuttered have been all over the map.

For instance, the city closed the Bronx’s Leadership Institute the year after it hit 71 percent of its goals — more than most schools in the program. At the other end of the spectrum, it shuttered Monroe Academy for Visual Arts and Design, which met just 14 percent of its goals.

In the past, education department officials have said they consider a range of factors when deciding which schools to close, including their academic performance, feedback from families, staff turnover, and previous improvement efforts.

“When making decisions about school closures we carefully assess each school based on multiple measures,” department spokesman Michael Aciman said in an email. “In every case where we’ve proposed a closure, we’ve prioritized family engagement and guaranteed that every student has higher-performing school options.”

However, the mix of factors means it isn’t clear which schools are most at risk of closure. The fact that some were shuttered after meeting most of their city-issued goals only adds to the mystery.

Small gains in graduation rates and test scores aren’t enough.

A handful of shuttered Renewal high schools had boosted their graduation rates while they were in the program, while some middle schools got more students to pass the state exams.

However, the gains were usually small and the majority of students were still struggling.

At the Essence School, the share of students who passed the reading tests more than doubled since it became a Renewal school. But even with that bump, still only 5 percent passed. Meanwhile, math proficiency barely ticked up to 3 percent.

At the high-school level, every shuttered Renewal school saw an uptick in graduation rates.

The increases ranged from 5 to 17 percentage points. However, because most of the schools enrolled were relatively small, they could boost their graduation rates by several points simply by helping a few additional students earn diplomas.

And the schools with the biggest gains — a 17 point jump at Foundations Academy and a 13 point spike at Foreign Language Academy of Global Studies — came at schools with serious enrollment challenges. In their final year, both served fewer than 100 students.