state wobegon

Looking for the culprits behind tests' dropping standards

What does it mean for tests to get easier? And is that really what happened to New York’s tests?

The analysis that has spurred that idea in the last few weeks actually found something slightly different. The tests aren’t necessarily easier, in the way that a kindergarten spelling bee is easier than the SAT. Instead, between 2007 and 2009, students who hadn’t learned much came out looking like they had.

This is an important distinction because it points to a different culprit behind the dropping standards than simply the individual test items themselves. Instead, Harvard professor Daniel Koretz – the lead author of the analysis commissioned by the state education department — names two possible causes: a phenomenon called “score inflation” and a possible psychometric error tied to an obscure state law.

The actual questions on the test play a role in both, but just as important is the practice of prepping students extensively for tests. Another key is a state law that forces New York to release all test items publicly, making it easier for teachers to practice test prep and making it harder for officials to keep tests consistent over time.

What Koretz found: A dropping hurdle

The question that motivated this week’s scrutiny of the state tests was: Is the increasing number of New York students passing the tests a sign that they know more — or is it a mirage?

In other words, imagine that the passing score of Level 3 out of 4 is a hurdle. Koretz wanted to figure out if more students were leaping over it because more of them could actually jump higher or, alternatively, because the bar had somehow been tugged down.

Maintaining a “proficiency” bar at the same height over time is harder than you might think, because unlike physical height, academic performance is abstract. An entire field of statistics called “psychometrics” exists just to keep the bars at the same height over time.

Likewise, it was a challenge for Koretz to test whether the Level 3 bar for the exam he studied first — the eighth-grade math test — stood at the same height in 2009 as it did in 2007. To compare two abstract things, Koretz needed a stable measurement of students’ raw competence. How much competence did it take to score a Level 3 in 2009 versus 2007? If the hurdle had stuck at the same height, the knowledge needed to clear it would be exactly the same.

To approximate raw competency, Koretz used the NAEP exam, which is the most respect national test and on which — conveniently for Koretz’s purposes — performance was relatively stable overall between 2007 and 2009. Using a mix of national and state test results, he could estimate the rough percentile rank on the NAEP that students had to get to achieve a Level 3 on the New York State test.

The move was like saying, If you gave the New York test to students nationally, what percentage would fail? (Scoring at the 80th percentile on a test means that you have reached a level that 80 percent of people couldn’t.) The national failure rate to match New York’s Level 3 was a rough way of knowing the New York “proficient” students’ raw competency.

If the number stayed the same between 2007 and 2009, then the bar must have stayed put. If raw competency dropped, Level 3 must have sunk, too.

As Koretz put it, “If people have to jump over a similar hurdle, the proportion failing to get there shouldn’t have changed dramatically – because NAEP scores didn’t change very much.”

But this is not what he found. “In fact,” he said this week, “the hurdle had been dropped so much that almost no kids would have failed to jump it.”

In 2007, 12 percent of students nationwide failed to reach the NAEP level equivalent to a Level 2 on the math exam. In 2009, the percentage had dropped to 2. For Level 3, the percentage dropped from 36 in 2007 to 19 in 2009.


Koretz says he can’t yet be certain why the Level 3 hurdle dropped over time, but he has two guesses. The first — and the one he suspects most strongly — is a phenomenon called “score inflation.”

Score inflation’s primary cause, Koretz told me, is what he calls “inappropriate test prep” — coaching students on material that teachers know will be covered on the test to the exclusion of other material covered by state standards, but that for a range of reasons doesn’t get tested. It can also be caused by deliberate attempts to game the tests, like by barring certain students from taking the test.

The result is that students get better at scoring high on tests over time, but they don’t learn more.

The other possible explanation Koretz cites has to do with the test’s makers, who are charged with “linking” tests from one year to another so that a Level 3 holds the same meaning over time.

In New York, linking is especially challenging because of a law we first wrote about last year that requires the state to release all its test items publicly. That prevents the state from following the industry-standard method of linking, which is to hide secret test questions from one year to the next, and use them as benchmarks that stay constant between years. New York instead has to use a less-reliable method called field testing, in which the state gives separate tests each year that aren’t attached to high-stakes.

“The problem,” Koretz explained, “is kids know it’s a field test.” They don’t take it as seriously as they take the state test, and the results, therefore, are compromised.

A failure to “link” properly doesn’t mean that McGraw-Hill, the company that makes the state tests, broke rules. But, said Koretz, “Even though the process that the contractor used was kosher, it doesn’t mean it worked.”

Moving forward

How do you fix score inflation and bad linking? Koretz said it’s not enough simply to raise the score that equates to “proficient.” But he said that, so far, state education officials are taking the right steps to do more.

Though they haven’t yet decreed a ban on test prep (something that would be hard to do), they have asked McGraw-Hill to redesign the tests so that they are less easily gamed. That includes trying to test a broader set of subjects within math and reading, as happened with this year’s math (but not English Language Arts) tests. It also includes making the test less predictable from year to year. (See a story we ran last year showing how the annual math tests repeat themselves.)

Koretz also said that McGraw Hill has performed “complicated psychometric work to reduce the affect score inflation might have on the linking.”

And the tests will be entirely re-written when the national common core standards effort to re-make assessments is completed in the next few years.

It’s all a big departure from what New York State was saying just two years ago, when Koretz first requested permission to analyze the state’s tests. Then, a spokesman for the State Education Department told me:

“All of New York’s tests are checked many times to be sure that a score this year means the same next year… The only way for a student to improve performance is by learning the curriculum — reading, writing, and math.”

The full Koretz five-page memo summarizing his findings so far:

Memo: Evidence about the leniency of 8th-grade standards

Transparency Tracker

‘No secret agreements’: Newarkers demand details of district-charter enrollment deal

PHOTO: Patrick Wall

This week, the Newark school board approved a lengthy legal agreement spelling out the details of the enrollment system that thousands of Newark families will use to apply to schools for the coming year.

Didn’t hear about it? You’re not alone.

The board OK’d the deal at a hastily arranged meeting Monday that few people in the community knew about or attended. State rules require that any changes to the district’s enrollment system “be publicly and transparently articulated before adoption.”

It’s unclear whether any changes were made — which would have triggered the transparency rules — because the board did not publicly discuss the details of the deal before voting, and the district has not made the agreement public.

Deborah Smith-Gregory, the president of the Newark NAACP, who attended Monday’s meeting, said she was disappointed that the board did not reveal any specifics about this year’s enrollment deal. Now that the district is back under local control after decades of state rule, she said, the elected board must commit to greater transparency.

“They have to do things differently,” she said. “They have to keep in mind that they’re a public entity — and they’re accountable to the community.”

The agreement describes in minute detail the inner workings of the five-year-old enrollment system, called Newark Enrolls, which allows families apply to most district and charter schools using a single application. The district and charter schools that opt into the system must sign the agreement each year.

The Newark Board of Education ratified the deal during a special meeting Monday — when schools and the district’s central office were closed. The meeting was scheduled to accommodate a charter school whose own board planned to vote on the agreement Tuesday. The timeline is tight because the citywide period for applying to schools begins Dec. 3.

The public agenda for Monday’s meeting, which mostly consisted of the board and Superintendent Roger León talking behind closed doors, did not mention the agreement. Just four community members were present for the public portion, when León and a couple board members made general comments about the controversial system, which critics contend funnels students into charter schools.

Then, without any public discussion of the agreement’s details — including a proposed change that León and charter leaders had debated in private — a majority of board members voted to approve it.

John Abeigon, president of the Newark Teachers Union and a fierce critic of charter schools, said both the district and its charter-school partners should disclose the terms of the deal.

“There should be absolute transparency,” he said.

The district’s current leadership is not the first to keep details of the enrollment system under wraps.

León, who began July 1, inherited it from his state-appointed predecessors, Cami Anderson and Christopher Cerf. One of only a handful of systems nationwide that combine district and charter admissions, proponents say it eases the enrollment process for families while helping to more evenly spread high-needs students across schools. Critics say it was designed to steer students and resources into the charter sector.

The system is dictated by the annual agreement between Newark Public Schools and participating charter schools. Apart from the news website NJ Spotlight, which published the agreement when it was first announced in 2013, it does not appear to have been released to the public since then — even as it has doubled in length, filling 20 pages last year.

In 2015, after Anderson touted the agreement at a state legislative hearing, saying it had created “greater equity and consistency” in admissions, several lawmakers asked to see it.

“No one seems to know about it,” said Assemblyman Ralph Caputo during the hearing.

After Anderson resigned, Cerf’s administration continued to renew the agreement each year. In an email, Cerf, who stepped down in February, said, “The document was always publicly available and was frequently discussed publicly.”

But community activists who have long scrutinized the enrollment system said they do not recall the district ever publicizing the agreement.

“I do not remember ever seeing this document, ever seeing it published anywhere, ever seeing it on the [district] website where we could find it, ever even discussing it in a thorough manner,” said Wilhelmina Holder, a longtime activist and critic of the enrollment system. She added that the new administration and school board should release the latest document to the public.

“No secret agreements,” she said. “You voted on it. If you’re discussing it, then why can’t we have a say in it?”

Absent the agreement, families have other ways to learn about Newark Enrolls. The district publishes a thick enrollment guidebook each year with information about every school, and hosts an annual admissions fair. It also maintains an enrollment website featuring a family-friendly video that illustrates how the system works.

But the agreement offers a uniquely detailed look under the system’s hood — and describes features that are not widely known, according to a copy of last year’s agreement that Chalkbeat obtained.

For instance, it alludes to a “third party” that programs the algorithm used to match students with schools based on the terms set forth in the agreement. The district plays “no active role” in the actual assignment of students to schools, the document says.

It also stipulates that the district must send charter schools as many students as they request. In return, charters must admit all students assigned to them — even if that pushes their enrollment above the limit set by the state, according to the 2017 document.

That practice of assigning schools more students than they currently have space for, called “overmatching,” is done to offset attrition that happens as some families inevitably leave the district before the next school year starts. It became a sticking point in recent closed-door negotiations between León’s administration and charter schools.

León wanted to end the practice, despite charter leaders who said it was critical for filling their seats. People in the charter sector said the final agreement still allows overmatching, though León told Chalkbeat that he believes the practice is unnecessary because charters can pull students from their waitlists to replace those who leave.

In an interview after Monday’s vote, León said no major changes were made to this year’s agreement.

Board chair Josephine Garcia, who made no public comments about enrollment during Monday’s meeting, declined to be interviewed immediately afterwards and did not respond to emails later in the week. However, she was overheard saying after the meeting that the district would eventually “rebrand” the enrollment system.

Chalkbeat contacted the district several times after the meeting to request a copy of the agreement. On Thursday evening, an official provided a public-records request form, which Chalkbeat submitted.

As of Friday, the district had not released the document.

Board Approved

Newark will keep universal enrollment for now — even as key dispute between charter schools and city appears unresolved

PHOTO: Patrick Wall/Chalkbeat
Superintendent Roger León gave a forceful defense of universal enrollment Monday before the school board voted to continue it.

Newark will keep its universal enrollment system for at least another year, despite critics who say it poses a grave threat to the district by allowing families to easily opt into charter schools.

The city’s board of education voted Monday to preserve the controversial enrollment system, called “Newark Enrolls,” which lets families use a single online system to apply to most traditional and charter schools. Just two years ago, the board tried to dismantle the system, arguing that it drained students and funding from the district as it fueled the charter sector’s rapid growth.

But, on Monday, the board appeared persuaded by the district’s new superintendent, Roger León, who said it is their duty to make it easy for families to send their children to whatever schools they choose — even private and parochial schools, which León said he hopes to eventually invite into the enrollment system.  

“That families today go through one system and have one application makes their life a lot less cumbersome,” he said. “It’s our responsibility to make sure that whatever they choose, they get.”

However, certain key details — such as how the system will handle “overmatching,” a process in which more students than typically show up are assigned to a school to address possible attrition over the summer — appear to still be the subject of some disagreement.

León’s full-throated defense of school choice is sure to surprise some community members, who had expected the former Newark Public Schools principal to rein in the charter sector after years of swift expansion under his state-appointed predecessors. Yet León has been open about his admiration for some of the city’s high-performing charter schools and his disdain for the district’s previously decentralized enrollment system, which favored families with the wherewithal to wait in long lines for coveted district-school seats or to apply separately to multiple charter schools.

Politics also may have played a role in the current system’s survival. In recent days, charter school advocates asked state Sen. Teresa Ruiz — a Newark power broker who is close to León — to help prevent changes to the system that they oppose, according to people in the charter sector.

Newark Enrolls also may have benefited from its relative popularity. A survey of 1,800 people who used the system this year found that 95 percent were “satisfied” or “very satisfied” with the enrollment process. And a phone survey of 302 Newark voters last month commissioned by the charter sector found that 52 percent of respondents favored the system, while 26 percent opposed it and 21 percent were undecided, according to a summary of the results obtained by Chalkbeat.

Yet charter schools — which now serve about one-third of city students — remain a lightning rod in Newark. Critics say they sap resources from the district while failing to serve their fair-share of needy students. In March, Mayor Ras Baraka called for a halt to their expansion.

Board Member Leah Owens, a former district teacher who is critical of charter schools, argued before Monday’s vote that more was on the line than the fate of the online application system.

“This is about, What is the future of Newark Public Schools going to look like if we continue to legitimize the idea of having privately run public schools?” she said during the meeting. “When we bring these schools into our enrollment system, we are saying that this is OK and that competition will improve the schools.”

Launched in 2014, the so-called “universal enrollment” system allows each family to apply online to up to eight traditional, magnet, or charter schools. A computer algorithm then assigns each student to a single school based on the family’s preferences, available space, and rules that give priority to students who live near schools or have special needs.

In the past, the district has allowed charter schools to specify how many students they want the district to assign them. Most request more students than they have space for to account for the attrition that invariably happens as some families move over the summer or enroll in private schools.

That practice, known as “overmatching,” became a flashpoint in the recent negotiations between León’s administration and charter schools, which must sign an annual agreement to participate in the enrollment system.

León’s side revised the agreement to eliminate overmatching, according to a person involved in the talks. Some charter leaders, worried the change would leave them with empty seats and reduced budgets, considered pulling out of the system.

The threat appears to have worked. The agreement that the board approved Monday still allows for overmatching, according to people in the charter sector. (The district has not made the agreement public, and officials did not respond to a request from Chalkbeat Tuesday to release it.)

“I don’t know anything about how that happened exactly,” said Jess Rooney, founder and co-director of People’s Preparatory Charter School. “All I know is that [León] got the message that that was of great concern, and he did a lot of work to address that concern very quickly.”

Charter leaders celebrated the agreement the school board ratified Monday, which they believe protects overmatching — a process they consider crucial for filling their rosters before classes start.

However, it’s not clear that León shares their interpretation.

In an interview Monday, he said the district would only send as many students to charter schools as they are authorized by the state to serve — even if they request extra students to offset attrition. If charters lose some students over the summer, they can replace them with students from their waitlists, he added.

“The legislature determined that there is a cap that they have,” León said, “and we’re sticking with that.”

Former district officials said that relying on charter schools to fill empty seats with students from their waitlists can disrupt district schools, which may abruptly lose students whom they were assigned. But León said that was not a concern, because charters can only pull students whose top choice had been a charter school.

“They have a right to pull that student because that student is not at their preferred school of choice,” he said. “That’s fine.”

Families can begin applying to schools for next school year on Dec. 3, León said. On Dec. 8, the district will host an admissions fair with representatives from traditional and charter schools.

In the meantime, the board of each charter school that plans to participate in universal enrollment must approve the agreement. Last year, 13 of the city’s 19 charter-school operators signed on.

Michele Mason, executive director of the Newark Charter School Fund, said she would defer to the district on “the implementation question” of overmatching. Other charter leaders insisted that the issue had been settled, and overmatching would continue as it has in the past.

Either way, Mason said she expects the same number of charter schools to join this year. She added that she was heartened by León’s remarks at Monday’s board meeting.

“I really do believe he values the options that charter schools give students and families,” she said.