state wobegon

Looking for the culprits behind tests' dropping standards

What does it mean for tests to get easier? And is that really what happened to New York’s tests?

The analysis that has spurred that idea in the last few weeks actually found something slightly different. The tests aren’t necessarily easier, in the way that a kindergarten spelling bee is easier than the SAT. Instead, between 2007 and 2009, students who hadn’t learned much came out looking like they had.

This is an important distinction because it points to a different culprit behind the dropping standards than simply the individual test items themselves. Instead, Harvard professor Daniel Koretz – the lead author of the analysis commissioned by the state education department — names two possible causes: a phenomenon called “score inflation” and a possible psychometric error tied to an obscure state law.

The actual questions on the test play a role in both, but just as important is the practice of prepping students extensively for tests. Another key is a state law that forces New York to release all test items publicly, making it easier for teachers to practice test prep and making it harder for officials to keep tests consistent over time.

What Koretz found: A dropping hurdle

The question that motivated this week’s scrutiny of the state tests was: Is the increasing number of New York students passing the tests a sign that they know more — or is it a mirage?

In other words, imagine that the passing score of Level 3 out of 4 is a hurdle. Koretz wanted to figure out if more students were leaping over it because more of them could actually jump higher or, alternatively, because the bar had somehow been tugged down.

Maintaining a “proficiency” bar at the same height over time is harder than you might think, because unlike physical height, academic performance is abstract. An entire field of statistics called “psychometrics” exists just to keep the bars at the same height over time.

Likewise, it was a challenge for Koretz to test whether the Level 3 bar for the exam he studied first — the eighth-grade math test — stood at the same height in 2009 as it did in 2007. To compare two abstract things, Koretz needed a stable measurement of students’ raw competence. How much competence did it take to score a Level 3 in 2009 versus 2007? If the hurdle had stuck at the same height, the knowledge needed to clear it would be exactly the same.

To approximate raw competency, Koretz used the NAEP exam, which is the most respect national test and on which — conveniently for Koretz’s purposes — performance was relatively stable overall between 2007 and 2009. Using a mix of national and state test results, he could estimate the rough percentile rank on the NAEP that students had to get to achieve a Level 3 on the New York State test.

The move was like saying, If you gave the New York test to students nationally, what percentage would fail? (Scoring at the 80th percentile on a test means that you have reached a level that 80 percent of people couldn’t.) The national failure rate to match New York’s Level 3 was a rough way of knowing the New York “proficient” students’ raw competency.

If the number stayed the same between 2007 and 2009, then the bar must have stayed put. If raw competency dropped, Level 3 must have sunk, too.

As Koretz put it, “If people have to jump over a similar hurdle, the proportion failing to get there shouldn’t have changed dramatically – because NAEP scores didn’t change very much.”

But this is not what he found. “In fact,” he said this week, “the hurdle had been dropped so much that almost no kids would have failed to jump it.”

In 2007, 12 percent of students nationwide failed to reach the NAEP level equivalent to a Level 2 on the math exam. In 2009, the percentage had dropped to 2. For Level 3, the percentage dropped from 36 in 2007 to 19 in 2009.


Koretz says he can’t yet be certain why the Level 3 hurdle dropped over time, but he has two guesses. The first — and the one he suspects most strongly — is a phenomenon called “score inflation.”

Score inflation’s primary cause, Koretz told me, is what he calls “inappropriate test prep” — coaching students on material that teachers know will be covered on the test to the exclusion of other material covered by state standards, but that for a range of reasons doesn’t get tested. It can also be caused by deliberate attempts to game the tests, like by barring certain students from taking the test.

The result is that students get better at scoring high on tests over time, but they don’t learn more.

The other possible explanation Koretz cites has to do with the test’s makers, who are charged with “linking” tests from one year to another so that a Level 3 holds the same meaning over time.

In New York, linking is especially challenging because of a law we first wrote about last year that requires the state to release all its test items publicly. That prevents the state from following the industry-standard method of linking, which is to hide secret test questions from one year to the next, and use them as benchmarks that stay constant between years. New York instead has to use a less-reliable method called field testing, in which the state gives separate tests each year that aren’t attached to high-stakes.

“The problem,” Koretz explained, “is kids know it’s a field test.” They don’t take it as seriously as they take the state test, and the results, therefore, are compromised.

A failure to “link” properly doesn’t mean that McGraw-Hill, the company that makes the state tests, broke rules. But, said Koretz, “Even though the process that the contractor used was kosher, it doesn’t mean it worked.”

Moving forward

How do you fix score inflation and bad linking? Koretz said it’s not enough simply to raise the score that equates to “proficient.” But he said that, so far, state education officials are taking the right steps to do more.

Though they haven’t yet decreed a ban on test prep (something that would be hard to do), they have asked McGraw-Hill to redesign the tests so that they are less easily gamed. That includes trying to test a broader set of subjects within math and reading, as happened with this year’s math (but not English Language Arts) tests. It also includes making the test less predictable from year to year. (See a story we ran last year showing how the annual math tests repeat themselves.)

Koretz also said that McGraw Hill has performed “complicated psychometric work to reduce the affect score inflation might have on the linking.”

And the tests will be entirely re-written when the national common core standards effort to re-make assessments is completed in the next few years.

It’s all a big departure from what New York State was saying just two years ago, when Koretz first requested permission to analyze the state’s tests. Then, a spokesman for the State Education Department told me:

“All of New York’s tests are checked many times to be sure that a score this year means the same next year… The only way for a student to improve performance is by learning the curriculum — reading, writing, and math.”

The full Koretz five-page memo summarizing his findings so far:

Memo: Evidence about the leniency of 8th-grade standards

the aftermath

What educators, parents, and students are grappling with in the wake of America’s latest school shooting

Kristi Gilroy (right) hugs a young woman at a police check point near the Marjory Stoneman Douglas High School where 17 people were killed by a gunman in Parkland, Florida. (Photo by Mark Wilson/Getty Images)

It’s hard to know where to start on days like this.

The shooting at Marjory Stoneman Douglas High School in Parkland, Florida, that left 17 people dead on Wednesday has elicited both terror and anger — and raised debates that are far from settled about how to keep American students safe.

Here are a few storylines we noticed as the country again grapples with a tragic school shooting:

1. You’re not wrong to think it: There have been a lot of mass shootings, and many recent ones have been especially deadly.

Data on school shootings specifically, though, is notoriously murky. As the Atlantic recently noted, varying definitions can contribute to either “sensationalizing or oversimplifying a modern trend of mass violence in America that is seemingly becoming more entrenched.”

But by NBC News’ count, 20 people have been killed and more than 30 have been injured in school shootings this year. That’s a lot — and more news organizations are now trying to keep a careful tally.

2. The consequences of traumatic events like the shooting at Stoneman Douglas are likely to be felt for some time.

A number of studies have found that violent and traumatic events in and outside of school do real damage to student learning, as we’ve reported — particularly among students who are already struggling. Here are some resources for teachers who need to talk to their students about trauma.

3. The tragedy is already renewing debates over whether or how to arm teachers.

Education Week gathered some of those calls from politicians Thursday. “Gun-safety advocates say that teachers can’t safely and quickly move from the mindset of teaching to being asked to fire a gun at an active shooter,” the story also notes.

This doesn’t even get at the debate about whether anyone should have access to the kind of gun the shooter used. Students from the district, for their part, told Broward schools chief Robert Runcie Thursday “that the time is due for a conversation on sensible gun control,” the Miami Herald reported.

Whether other technology and infrastructure can help keep students safe is a topic of ongoing discussion in communities across the country. Colorado lawmakers are considering a bill to help schools buy communications systems that would allow them to talk directly to police and other emergency responders. Officials from districts that already use this equipment described them as a way to increase safety without “turning our schools into prisons,” even as they also assured lawmakers that the radios were just as useful for serious playground injuries and broken-down buses as for the much rarer active shooter situations.

In Tennessee, one school district near Nashville announced plans to close schools next Monday to review all safety plans with school staff and local law enforcement.

4. In some places, the shooting is unlikely to change the school safety debate at all.

In New York City, for example, conversations about school safety in recent years have revolved around discipline policies and metal detectors (though police have seized an increasing number of weapons from city schools). There’s little appetite there to arm teachers.

5. But all across America, the shooting and others like it have added a frightening tone to what it means to teach and learn in schools today.

“I know you are waking up this morning to a nightmare,” a former educator wrote in a “love letters to teachers” on Teaching Tolerance. “I know you are frustrated, tired and weary of the news. I know you are wearing your coat of bravery today.”

“I’m so, so angry and I’m having a hard time today looking at my students and not thinking about what happens when it’s my school’s turn,” wrote one commenter on the Badass Teachers Association Facebook group.

getting to graduation

New York City graduation rate hits record high of 74.3 percent in 2017

PHOTO: Christina Veiga
Mayor Bill de Blasio announced the 2017 graduation rate at South Bronx Preparatory school.

New York City’s graduation rate rose to 74.3 percent in 2017, a slight increase over the previous year and a new high for the city.

The 1.2 percentage point increase over the previous year continues an upward climb for the city, where the overall graduation rate has grown by nearly 28 points since 2005. The state graduation rate also hit a new high — 82.1 percent — just under the U.S. rate of 84.1 percent.

The city’s dropout rate fell to 7.8 percent, a small decline from the previous year and the lowest rate on record, according to the city.

“New York City is showing that when we invest in our students, they rise to the challenge and do better and better,” Mayor Bill de Blasio said in a statement Wednesday.

More graduates were also deemed ready for college-level work. Last year, 64 percent of graduates earned test scores that met the City University of New York’s “college-ready” benchmark — up more than 13 percentage points from the previous year. 

However, the gains came after CUNY eased its readiness requirements; without that change, city officials said the increase would be significantly smaller. But even with the less rigorous requirements, more than a third of city students who earned high-school diplomas would be required to take remedial classes at CUNY.

Phil Weinberg, the education department’s deputy chancellor for teaching and learning, noted that CUNY’s college-readiness requirements are more demanding than New York’s graduation standards — which are among the toughest in the country.

We will work toward making sure none of our students need remediation when they get to college,” he told reporters. “But that’s a long game for us and we continue to move in that direction.”

The rising graduation rates follow a series of changes the state has made in recent years to help more students earn diplomas.

The graduation-requirement changes include allowing students with disabilities to earn a diploma by passing fewer exit exams and letting more students appeal a failed score. In addition, students can now substitute a work-readiness credential for one of the five Regents exams they must pass in order to graduate — adding to a number of other alternative tests the state has made available in the past few years.

About 9,900 students used one of those alternative-test or credential options in 2017, while 315 students with disabilities took advantage of the new option for them, according to state officials. They could not say how many students successfully appealed a low test score; but in 2016, about 1,300 New York City students did so.

The news was mixed for schools in de Blasio’s high-profile “Renewal” improvement program for low-performing schools. Among the 28 high schools that have received new social services and academic support through the program, the graduation rate increased to nearly 66 percent — almost a 6 percentage point bump over 2016. Their dropout rate also fell by about 2 points, to 16.4 percent, though that remains more than twice as high as the citywide rate.

However, more than half of the high schools in that $568 million program — 19 out of 28 — missed the graduation goals the city set for them, according to a New York Times analysis based on preliminary figures.

Graduation rates for students who are still learning English ticked up slightly to 32.5 percent, following a sharp decline the previous year that the state education commissioner called “disturbing.” City officials argue that students who improved enough to shed the designation of “English language learner” in the years before they graduated should also be counted; among that larger group, the graduation rate was 53 percent in 2017.

Meanwhile, the graduation-rate gap between white students and their black and Hispanic peers narrowed a smidgen, but it remains wide. Last year, the graduation rate was about 83 percent for white students, 70 percent for black students, and 68 percent for Hispanic students. That represented a closing of the gap between white and black students by 0.4 percentage points, and 0.1 points between whites and Hispanic.

Asian students had the highest rate — 87.5 percent — a nearly 2 point increase from the previous year that widened their lead over other racial groups.

Christina Veiga contributed reporting.