behind the metrics

IBO: City's school progress reports are flawed but an advance

The system the city uses to award letter grades to schools is complicated and in some ways flawed — but it’s the best system we have.

That’s the conclusion of a report by the Independent Budget Office, the city’s budget watchdog that since 2009 has been charged with scrutinizing Department of Education data. The office examined the city’s progress reports, released annually since 2007, to see whether their underlying metrics produce meaningful results.

The progress reports were meant to radically reorient the way that New Yorkers thought about school performance. Instead of assessing schools simply by the proportion of students passing state tests, the progress reports focus on students’ improvement from year to year. In a precursor to the “value-added” measurements now being used to assess teachers, the reports use a complex and evolving algorithm that controls for student demographics to calculate just how much students have progressed.

The city then assigns each school a letter grade based on its score. The letter grades inform both the city’s decisions about which principals should receive bonuses and which schools should be considered for closure and families’ choices are where to enroll.

The IBO concludes that the progress reports offer a more sophisticated analysis of school performance than ever before — but that there is room for improvement. “The methodology used by the education department is a significant improvement over simply basing measures on comparisons of standardized test scores,” the report concludes. “Still, the School Progress Reports have to be interpreted with caution.”

The IBO looked at three issues: whether the city’s algorithm has successfully controlled for factors outside of schools’ control; whether the reports have reflected long-term shifts as well as short-term changes; and whether minor methodology changes produced outsized score swings.

On the first question, the budget office concluded that overall, progress report scores in a small set of schools, those serving both elementary and middle school students, can be considered “demographically neutral” — or unaffected by student characteristics. But in most cases that was not true, according to the IBO’s analysis.

“All other things equal, elementary, middle, and high schools with a higher percentage of black and Hispanic students were consistently likely to have lower overall scores than other schools,” the report notes. Progress report scores were also lower in high schools with more poor students and more students with disabilities, the IBO concluded.

Confirming previous findings, the IBO also concludes that elementary and middle school scores have been highly volatile, with the majority of those schools receiving three or more different progress report grades since 2007. (High school progress report grades are based on a wider range of variables and have always been more stable.) But the IBO says that changes to the reports’ methodology, particularly around how students’ year-to-year growth is assessed, have made them more stable.

Finally, the IBO’s analysis found that most of the changes made to the progress report methodology in 2010 and 2011 did not affect their overall grade. In general, the office concludes, the city’s reports successfully identified very high- and very low-performing schools under multiple methodologies, but they were less successful at distinguishing among middle performers.

“The distinction between a C and D rating for a school may be the result of the particular methodology that the DOE has chosen, among the many that are possible, rather than the result of school practices or effectiveness,” the report concludes. “Unfortunately, this weakness occurs at precisely the point where high stakes decisions about schools are made.”

Department of Education officials say that while the peer group comparisons do not “completely control” for student characteristics, they do reduce the impact of race and other demographics compared to other measures of school performance. They also point out that the IBO found strong correlations between demographics and school grades in only a handful of the relationships it analyzed. And they note that the IBO’s analysis shows that between half and three quarters of C and D grades issued in 2011 would have been the same using the IBO’s methodology for analyzing score stability.

“As the IBO has recognized, New York City’s progress reports are a huge improvement over other state and district systems for measuring student learning in schools — and for this reason they have become a national model,” said spokesman Matthew Mittenthal in a statement. “Closing the achievement gap in New York City is a core goal of our reform strategy, but as long as it exists we should expect it to show up in school progress reports, which are designed to be an accurate reflection of our schools’ strengths and challenges.”

The complete IBO report is below.

What's Your Education Story?

As the 2018 school year begins, join us for storytelling from Indianapolis educators

PHOTO: Dylan Peers McCoy/Chalkbeat
Sarah TeKolste, right, and Lori Jenkins at a Teacher Story Slam, in April.

In partnership with Teachers Lounge Indy, Chalkbeat is hosting another teacher story slam this fall featuring educators from across the city.

Over the past couple of years, Chalkbeat has brought readers personal stories from teachers and students through the events. Some of our favorites touched on how a teacher won the trust of her most skeptical student, why another teacher decided to come out to his students, and one educator’s call to ramp up the number of students pursuing a college education.

The event, 5:30 p.m. Thursday, Sept. 13, is free and open to the public — please RSVP here.

Event details:

5:30 p.m. to 7:30 p.m.
Thursday, Sept. 13, 2018
Tube Factory artspace
1125 Cruft St., Indianapolis, IN 46203
Get tickets here and find more on Facebook

More in What's Your Education Story?

School safety

Hiring more security officers in Memphis after school shootings could have unintended consequences

PHOTO: Jahi Chikwendiu/The Washington Post/Getty Images

Tennessee’s largest district, Shelby County Schools, is slated to add more school resource officers under the proposed budget for next school year.

Superintendent Dorsey Hopson earmarked $2 million to hire 30 school resource officers in addition to the 98 already in some of its 150-plus schools. The school board is scheduled to vote on the budget Tuesday.

But an increase in law enforcement officers could have unintended consequences.

A new state law that bans local governments from refusing to cooperate with federal immigration officials could put school resource officers in an awkward position.

Tennessee Education Commissioner Candice McQueen recently reminded school personnel they are not obligated to release student information regarding immigration status. School resource officers employed by police or sheriff’s departments, however, do not answer to school districts. Shelby County Schools is still reviewing the law, but school board members have previously gone on the record emphasizing their commitment to protecting undocumented students.

“Right now we are just trying to get a better understanding of the law and the impact that it may have,” said Natalia Powers, a district spokeswoman.

Also, incidents of excessive force and racial bias toward black students have cropped up in recent years. Two white Memphis officers were fired in 2013 after hitting a black student and wrestling her to the ground because she was “yelling and cussing” on school grounds. And mothers of four elementary school students recently filed a lawsuit against a Murfreesboro officer who arrested them at school in 2016 for failing to break up a fight that occurred off-campus.

Just how common those incidents are in Memphis is unclear. In response to Chalkbeat’s query for the number and type of complaints in the last two school years, Shelby County Schools said it “does not have any documents responsive to this request.”

Currently, 38 school resource officers are sheriff’s deputies, and the rest are security officers hired by Shelby County Schools. The officers respond and work to prevent criminal activity in all high schools and middle schools, Hopson said. The 30 additional officers would augment staffing at some schools and for the first time, branch out to some elementary schools. Hopson said those decisions will be based on crime rates in surrounding neighborhoods and school incidents.

Hopson’s initial recommendation for more school resource officers was in response to the school shooting in Parkland, Florida, that killed 17 people and sparked a wave of student activism on school safety, including in Memphis.

Gov. Bill Haslam’s recent $30 million budget boost would allow school districts across Tennessee to hire more law enforcement officers or improve building security. Measures to arm some teachers with guns or outlaw certain types of guns have fallen flat.


For more on the role and history of school resource officers in Tennessee, read our five things to know.


Sheriff’s deputies and district security officers meet weekly, said Capt. Dallas Lavergne of the Shelby County Sheriff’s Office. When the Memphis Police Department pulled their officers out of school buildings following the merger of city and county school systems, the county Sheriff’s Office replaced them with deputies.

All deputy recruits go through school resource officer training, and those who are assigned to schools get additional annual training. In a 2013 review of police academies across the nation, Tennessee was cited as the only state that had specific training for officers deployed to schools.