state wobegon

Looking for the culprits behind tests' dropping standards

What does it mean for tests to get easier? And is that really what happened to New York’s tests?

The analysis that has spurred that idea in the last few weeks actually found something slightly different. The tests aren’t necessarily easier, in the way that a kindergarten spelling bee is easier than the SAT. Instead, between 2007 and 2009, students who hadn’t learned much came out looking like they had.

This is an important distinction because it points to a different culprit behind the dropping standards than simply the individual test items themselves. Instead, Harvard professor Daniel Koretz – the lead author of the analysis commissioned by the state education department — names two possible causes: a phenomenon called “score inflation” and a possible psychometric error tied to an obscure state law.

The actual questions on the test play a role in both, but just as important is the practice of prepping students extensively for tests. Another key is a state law that forces New York to release all test items publicly, making it easier for teachers to practice test prep and making it harder for officials to keep tests consistent over time.

What Koretz found: A dropping hurdle

The question that motivated this week’s scrutiny of the state tests was: Is the increasing number of New York students passing the tests a sign that they know more — or is it a mirage?

In other words, imagine that the passing score of Level 3 out of 4 is a hurdle. Koretz wanted to figure out if more students were leaping over it because more of them could actually jump higher or, alternatively, because the bar had somehow been tugged down.

Maintaining a “proficiency” bar at the same height over time is harder than you might think, because unlike physical height, academic performance is abstract. An entire field of statistics called “psychometrics” exists just to keep the bars at the same height over time.

Likewise, it was a challenge for Koretz to test whether the Level 3 bar for the exam he studied first — the eighth-grade math test — stood at the same height in 2009 as it did in 2007. To compare two abstract things, Koretz needed a stable measurement of students’ raw competence. How much competence did it take to score a Level 3 in 2009 versus 2007? If the hurdle had stuck at the same height, the knowledge needed to clear it would be exactly the same.

To approximate raw competency, Koretz used the NAEP exam, which is the most respect national test and on which — conveniently for Koretz’s purposes — performance was relatively stable overall between 2007 and 2009. Using a mix of national and state test results, he could estimate the rough percentile rank on the NAEP that students had to get to achieve a Level 3 on the New York State test.

The move was like saying, If you gave the New York test to students nationally, what percentage would fail? (Scoring at the 80th percentile on a test means that you have reached a level that 80 percent of people couldn’t.) The national failure rate to match New York’s Level 3 was a rough way of knowing the New York “proficient” students’ raw competency.

If the number stayed the same between 2007 and 2009, then the bar must have stayed put. If raw competency dropped, Level 3 must have sunk, too.

As Koretz put it, “If people have to jump over a similar hurdle, the proportion failing to get there shouldn’t have changed dramatically – because NAEP scores didn’t change very much.”

But this is not what he found. “In fact,” he said this week, “the hurdle had been dropped so much that almost no kids would have failed to jump it.”

In 2007, 12 percent of students nationwide failed to reach the NAEP level equivalent to a Level 2 on the math exam. In 2009, the percentage had dropped to 2. For Level 3, the percentage dropped from 36 in 2007 to 19 in 2009.

Why?

Koretz says he can’t yet be certain why the Level 3 hurdle dropped over time, but he has two guesses. The first — and the one he suspects most strongly — is a phenomenon called “score inflation.”

Score inflation’s primary cause, Koretz told me, is what he calls “inappropriate test prep” — coaching students on material that teachers know will be covered on the test to the exclusion of other material covered by state standards, but that for a range of reasons doesn’t get tested. It can also be caused by deliberate attempts to game the tests, like by barring certain students from taking the test.

The result is that students get better at scoring high on tests over time, but they don’t learn more.

The other possible explanation Koretz cites has to do with the test’s makers, who are charged with “linking” tests from one year to another so that a Level 3 holds the same meaning over time.

In New York, linking is especially challenging because of a law we first wrote about last year that requires the state to release all its test items publicly. That prevents the state from following the industry-standard method of linking, which is to hide secret test questions from one year to the next, and use them as benchmarks that stay constant between years. New York instead has to use a less-reliable method called field testing, in which the state gives separate tests each year that aren’t attached to high-stakes.

“The problem,” Koretz explained, “is kids know it’s a field test.” They don’t take it as seriously as they take the state test, and the results, therefore, are compromised.

A failure to “link” properly doesn’t mean that McGraw-Hill, the company that makes the state tests, broke rules. But, said Koretz, “Even though the process that the contractor used was kosher, it doesn’t mean it worked.”

Moving forward

How do you fix score inflation and bad linking? Koretz said it’s not enough simply to raise the score that equates to “proficient.” But he said that, so far, state education officials are taking the right steps to do more.

Though they haven’t yet decreed a ban on test prep (something that would be hard to do), they have asked McGraw-Hill to redesign the tests so that they are less easily gamed. That includes trying to test a broader set of subjects within math and reading, as happened with this year’s math (but not English Language Arts) tests. It also includes making the test less predictable from year to year. (See a story we ran last year showing how the annual math tests repeat themselves.)

Koretz also said that McGraw Hill has performed “complicated psychometric work to reduce the affect score inflation might have on the linking.”

And the tests will be entirely re-written when the national common core standards effort to re-make assessments is completed in the next few years.

It’s all a big departure from what New York State was saying just two years ago, when Koretz first requested permission to analyze the state’s tests. Then, a spokesman for the State Education Department told me:

“All of New York’s tests are checked many times to be sure that a score this year means the same next year… The only way for a student to improve performance is by learning the curriculum — reading, writing, and math.”

The full Koretz five-page memo summarizing his findings so far:

Memo: Evidence about the leniency of 8th-grade standards

first steps

Superintendent León secures leadership team, navigates evolving relationship with board

PHOTO: Patrick Wall
Superintendent Roger León at Tuesday's school board meeting.

As Newark’s new superintendent prepares for the coming academic year, the school board approved the final members of his leadership team Tuesday and began piecing together a roadmap to guide his work.

The board confirmed three assistant superintendents chosen by Superintendent Roger León: Jose Fuentes, the principal of First Avenue School in the North Ward; Sandra Rodriguez, a Hoboken principal who previously oversaw Newark Public Schools’ early childhood office; and Mario Santos, principal of East Side High School in the East Ward. They join three other assistant superintendents León selected for his team, along with a deputy superintendent, chief of staff, and several other officials.

The three assistant superintendents confirmed Tuesday had first come before the board in June, but at that time none of them secured enough votes to be approved. During last month’s meeting, the board assented to several of León’s leadership picks and to his decision to remove many people from the district’s central office, but it also blocked him from ousting several people.

This week, Board Chair Josephine Garcia declined to comment on the board’s reversal, and León did not respond to a request for comment.

What is clear is that the board and León are still navigating their relationship.

In February, the board regained local control of the district 22 years after the state seized control of the district due to poor performance and mismanagement. The return to local control put the board back in charge of setting district policy and hiring the superintendent, who previously answered only to the state. Still, the superintendent, not the board, is responsible for overseeing the district’s day-to-day operations.

During a board discussion Tuesday, Garcia hinted at that delicate balance of power.

“Now that we’re board members, we want to make sure that, of course, yes, we’re going to have input and implementation,” but that they don’t overstep their authority, she said.

Under state rules, the board is expected to develop district goals and policies, which the superintendent is responsible for acting on. But León — a former principal who spent the past decade serving as an assistant superintendent — has his own vision for the district, which he hopes to convince the board to support, he said in a recent interview on NJTV.

“It’s my responsibility as the new superintendent of schools to compel them to assist the district moving in the direction that I see as appropriate,” he said.

Another matter still being ironed out by the board and superintendent is communication.

León did not notify the full board before moving to force out 31 district officials and administrators, which upset some members. And he told charter school leaders in a closed-door meeting that he plans to keep intact the single enrollment system for district and charter schools — a controversial policy the board is still reviewing.

The district has yet to make a formal announcement about the staff shake-up, including the appointment of León’s new leadership team. And when the board voted on the new assistant superintendents Tuesday, it used only the appointed officials’ initials — not their full names. However, board member Leah Owens stated the officials’ full names when casting her vote.

The full names, titles and salaries of public employees are a matter of public record under state law.

Earlier, board member Yambeli Gomez had proposed improved communication as a goal for the board.

“Not only communication within the board and with the superintendent,” she said, “but also communication with the public in a way that’s more organized.”

The board spent much of Tuesday’s meeting brainstorming priorities for the district.

Members offered a grab bag of ideas, which were written on poster paper. Under the heading “student achievement,” they listed literacy, absenteeism, civics courses, vocational programs, and teacher quality, among other topics. Under other “focus areas,” members suggested classroom materials, parent involvement, and the arts.

Before the school year begins in September, León is tasked with shaping the ideas on that poster paper into specific goals and an action plan.

After the meeting, education activist Wilhelmina Holder said she hopes the board will focus its attention on a few key priorities.

“There was too much of a laundry list,” she said.

early dismissals

Top Newark school officials ousted in leadership shake-up as new superintendent prepares to take over

PHOTO: Patrick Wall
Incoming Newark Public Schools Superintendent Roger León

Several top Newark school officials were given the option Friday to resign or face termination, in what appeared to be an early move by incoming Superintendent Roger León to overhaul the district’s leadership.

The shake-up includes top officials such as the chief academic officer and the head of the district’s controversial enrollment system, as well as lower-level administrators — 31 people in total, according to documents and district employees briefed on the overhaul. Most of the officials were hired or promoted by the previous two state-appointed superintendents, Cami Anderson and Christopher Cerf, a sign that León wants to steer the district in a new direction now that it has returned to local control.

The officials were given the option to resign by Tuesday and accept buyouts or face the prospect of being fired by the school board at its meeting that evening. The buyouts offer a financial incentive to those who resign voluntarily on top of any severance included in their contracts. In exchange for accepting the buyouts, the officials must sign confidentiality agreements and waive their right to sue the district.

Earlier this week, León submitted a list of his choices to replace the ousted cabinet-level officials, which the board must approve at its Tuesday meeting. It’s not clear whether he has people lined up to fill the less-senior positions.

It’s customary for incoming superintendents to appoint new cabinet members and reorganize the district’s leadership structure, which usually entails replacing some personnel. However, many staffers were caught off guard by Friday’s dismissals since León has given little indication of how he plans to restructure the central office — and he does not officially take the reins of the district until July 1.

A district spokeswoman and the school board chair did not immediately respond to emails on Friday about the shake-up.

Some staffers speculated Friday that the buyout offers were a way for León to replace the district’s leadership without securing the school board’s approval because, unlike with terminations, the board does not need to sign off on resignations. However, it’s possible the board may have to okay any buyout payments. And it could also be the case that the buyouts were primarily intended to help shield the district from legal challenges to the dismissals.

León was not present when the staffers learned Friday afternoon that they were being let go, the employees said. Instead, the interim superintendent, Robert Gregory, and other top officials broke the news, which left some stunned personnel crying and packing their belongings into boxes. They received official separation letters by email later that day.

The people being ousted include Chief Academic Officer Brad Haggerty and Gabrielle Ramos-Solomon, who oversees enrollment. Also included are top officials in the curriculum, early childhood, and finance divisions, among others, according to a list obtained by Chalkbeat.

In addition to the 31 being pushed out, several assistant superintendents are being demoted but will remain in the district, according to the district employees.

There was concern among some officials Friday about whether the turnover would disrupt planning for the coming school year.

“I don’t know how we’re going to open smoothly with cuts this deep,” one of the employees said. “Little to no communication was provided to the teams about what these cuts mean for the many employees who remain in their roles and need leadership guidance and direction Monday morning.”