state wobegon

Looking for the culprits behind tests' dropping standards

What does it mean for tests to get easier? And is that really what happened to New York’s tests?

The analysis that has spurred that idea in the last few weeks actually found something slightly different. The tests aren’t necessarily easier, in the way that a kindergarten spelling bee is easier than the SAT. Instead, between 2007 and 2009, students who hadn’t learned much came out looking like they had.

This is an important distinction because it points to a different culprit behind the dropping standards than simply the individual test items themselves. Instead, Harvard professor Daniel Koretz – the lead author of the analysis commissioned by the state education department — names two possible causes: a phenomenon called “score inflation” and a possible psychometric error tied to an obscure state law.

The actual questions on the test play a role in both, but just as important is the practice of prepping students extensively for tests. Another key is a state law that forces New York to release all test items publicly, making it easier for teachers to practice test prep and making it harder for officials to keep tests consistent over time.

What Koretz found: A dropping hurdle

The question that motivated this week’s scrutiny of the state tests was: Is the increasing number of New York students passing the tests a sign that they know more — or is it a mirage?

In other words, imagine that the passing score of Level 3 out of 4 is a hurdle. Koretz wanted to figure out if more students were leaping over it because more of them could actually jump higher or, alternatively, because the bar had somehow been tugged down.

Maintaining a “proficiency” bar at the same height over time is harder than you might think, because unlike physical height, academic performance is abstract. An entire field of statistics called “psychometrics” exists just to keep the bars at the same height over time.

Likewise, it was a challenge for Koretz to test whether the Level 3 bar for the exam he studied first — the eighth-grade math test — stood at the same height in 2009 as it did in 2007. To compare two abstract things, Koretz needed a stable measurement of students’ raw competence. How much competence did it take to score a Level 3 in 2009 versus 2007? If the hurdle had stuck at the same height, the knowledge needed to clear it would be exactly the same.

To approximate raw competency, Koretz used the NAEP exam, which is the most respect national test and on which — conveniently for Koretz’s purposes — performance was relatively stable overall between 2007 and 2009. Using a mix of national and state test results, he could estimate the rough percentile rank on the NAEP that students had to get to achieve a Level 3 on the New York State test.

The move was like saying, If you gave the New York test to students nationally, what percentage would fail? (Scoring at the 80th percentile on a test means that you have reached a level that 80 percent of people couldn’t.) The national failure rate to match New York’s Level 3 was a rough way of knowing the New York “proficient” students’ raw competency.

If the number stayed the same between 2007 and 2009, then the bar must have stayed put. If raw competency dropped, Level 3 must have sunk, too.

As Koretz put it, “If people have to jump over a similar hurdle, the proportion failing to get there shouldn’t have changed dramatically – because NAEP scores didn’t change very much.”

But this is not what he found. “In fact,” he said this week, “the hurdle had been dropped so much that almost no kids would have failed to jump it.”

In 2007, 12 percent of students nationwide failed to reach the NAEP level equivalent to a Level 2 on the math exam. In 2009, the percentage had dropped to 2. For Level 3, the percentage dropped from 36 in 2007 to 19 in 2009.


Koretz says he can’t yet be certain why the Level 3 hurdle dropped over time, but he has two guesses. The first — and the one he suspects most strongly — is a phenomenon called “score inflation.”

Score inflation’s primary cause, Koretz told me, is what he calls “inappropriate test prep” — coaching students on material that teachers know will be covered on the test to the exclusion of other material covered by state standards, but that for a range of reasons doesn’t get tested. It can also be caused by deliberate attempts to game the tests, like by barring certain students from taking the test.

The result is that students get better at scoring high on tests over time, but they don’t learn more.

The other possible explanation Koretz cites has to do with the test’s makers, who are charged with “linking” tests from one year to another so that a Level 3 holds the same meaning over time.

In New York, linking is especially challenging because of a law we first wrote about last year that requires the state to release all its test items publicly. That prevents the state from following the industry-standard method of linking, which is to hide secret test questions from one year to the next, and use them as benchmarks that stay constant between years. New York instead has to use a less-reliable method called field testing, in which the state gives separate tests each year that aren’t attached to high-stakes.

“The problem,” Koretz explained, “is kids know it’s a field test.” They don’t take it as seriously as they take the state test, and the results, therefore, are compromised.

A failure to “link” properly doesn’t mean that McGraw-Hill, the company that makes the state tests, broke rules. But, said Koretz, “Even though the process that the contractor used was kosher, it doesn’t mean it worked.”

Moving forward

How do you fix score inflation and bad linking? Koretz said it’s not enough simply to raise the score that equates to “proficient.” But he said that, so far, state education officials are taking the right steps to do more.

Though they haven’t yet decreed a ban on test prep (something that would be hard to do), they have asked McGraw-Hill to redesign the tests so that they are less easily gamed. That includes trying to test a broader set of subjects within math and reading, as happened with this year’s math (but not English Language Arts) tests. It also includes making the test less predictable from year to year. (See a story we ran last year showing how the annual math tests repeat themselves.)

Koretz also said that McGraw Hill has performed “complicated psychometric work to reduce the affect score inflation might have on the linking.”

And the tests will be entirely re-written when the national common core standards effort to re-make assessments is completed in the next few years.

It’s all a big departure from what New York State was saying just two years ago, when Koretz first requested permission to analyze the state’s tests. Then, a spokesman for the State Education Department told me:

“All of New York’s tests are checked many times to be sure that a score this year means the same next year… The only way for a student to improve performance is by learning the curriculum — reading, writing, and math.”

The full Koretz five-page memo summarizing his findings so far:

Memo: Evidence about the leniency of 8th-grade standards

Busing Ban

As school districts push for integration, decades-old federal rule could thwart them

PHOTO: RJ Sangosti/The Denver Post
Several districts across the country want to use federal money to pay for school buses as part of their desegregation plans. A federal spending restriction could get in the way.

In Florida, officials plan to use federal money to shuttle students across vast Miami-Dade County to new science-themed magnet programs in a bid to desegregate several schools.

In South Carolina, a tiny district west of Myrtle Beach intends to spend federal funds on free busing for families who enroll at two predominantly black schools, hoping that will draw in white and Hispanic students.

And in New York, state officials want to deploy federal school-improvement money to help integrate struggling schools, believing that may be the secret to their rebirth.

But each of these fledgling integration efforts — and similar ones across the country — could be imperiled by obscure budget provisions written during the anti-busing backlash of the 1970s, which prohibit using federal funding for student transportation aimed at racial desegregation. The rules have been embedded in every education spending bill since at least 1974, as Rep. Bobby Scott of Virginia pointed out in September when he tried unsuccessfully to remove the provisions from the latest appropriations bill.

The rules are “a relic of an ugly history when states and school districts across the nation resisted meaningful integration,” said Scott, the top Democrat on the House education committee, during a floor speech where he called the persistence of the rules “morally reprehensible.”

After Scott’s amendment to eliminate the provisions was blocked, advocates are now working behind the scenes to convince members of the Senate from both parties to strike the rules from the latest spending bill during negotiations. More than 40 integration advocates and experts have signed onto a letter to lawmakers calling for the anti-busing language to be removed, and members of that coalition plan to meet with lawmakers in the coming days.

Advocates are especially worried about funding for magnet programs, like those in Miami and the South Carolina district, which rely on special science or art offerings or rigorous academic courses to draw students of different races into the same school — a choice-based approach that has become the primary way districts now pursue desegregation.

This is the first year districts that receive federal magnet-school grants are allowed to spend some of that money on transportation, after Congress changed the rules as part of its education-law overhaul in 2015. Among the 32 districts that received a total of nearly $92 million in magnet grants this year, at least six plan to use some of that money for transportation, according to their applications.

Now, just as those funds are about to flow to busing — which many families insist upon before they will enroll their children in magnet schools across town — the decades-old spending restriction could cut them off, advocates warn.

That could create a major problem for districts like Miami-Dade County.

It hopes to attract students from across the district to three heavily black and Hispanic schools by launching magnet programs that focus on zoology, cybersecurity, and mobile-app development, according to its application. To pull that off, it requested $245,000 for buses next year since, as the application notes, the “most limiting factor” for many families is “the cost associated with transporting their child to the magnet school.”

The district in Lake City, South Carolina wants to pull new families from different neighborhoods into an elementary school and a middle school that suffer from sagging enrollment and intense poverty. Previous recruitment efforts that didn’t provide transportation amounted to “failed attempts,” the district said in its application.

However, if the anti-busing provisions are not removed from the next federal spending bill, they would cancel out the new rule allowing those districts to spend some of their magnet money on transportation (though districts could still use local funds to fill in the gap). As such, magnet-school representatives are pushing hard for lawmakers to remove the provisions during budget negotiations.

“We’re hoping this doesn’t see the light of day,” said John Laughner, legislative and communications manager at Magnet Schools of America, an association of magnets from across the country. He plans to discuss the issue with lawmakers next week.

Beyond magnet schools, other desegregation efforts could be undercut by the anti-busing provision, which was included in a spending bill for fiscal year 2018 that the House approved and one the Senate has yet to vote on.

At least one state — New York — listed socioeconomic and racial integration among the ways it could intervene in low-performing schools under the new federal education law. In addition, New York officials announced a grant program this week where up to 30 districts will receive federal money to develop integration plans.

Advocates fear the anti-busing rule could disrupt any of those plans that require transportation and aim to reduce racial segregation. (New York education officials said they did not want to speculate on the impact of a spending bill that hasn’t been approved.)

A Democratic Congressional aide who has studied the issue said the provision could even block federal funding for planning or public outreach around desegregation programs that involve busing, not just busing itself.

Either way, advocates say the provision could dissuade districts from using the new education law, the Every Student Succeeds Act, to pursue integration — even though research suggests that student achievement on tests and other measures improve when they attend less segregated schools.

“We shouldn’t have this,” said Philip Tegeler, a member of the National Coalition on School Diversity, which is leading the charge to remove the restriction. He added that the provision stemmed from mandatory desegregation busing of an earlier era: “It’s clearly an anachronism that doesn’t really fit any more with what states and districts are doing voluntarily.”

A U.S. education department spokeswoman said Secretary Betsy DeVos would be bound to enforce any funding prohibitions that Congress approves, though she noted that state and local funds are not subject to the same restrictions.

Negotiators from the House and Senate must still agree on a single spending bill, which would go before the full Congress for a vote. Until then, lawmakers have voted to temporarily extend 2017 spending levels through December. It’s possible Congress will pass another extension then, meaning a final deal — and a decision on the anti-busing language — may not arrive until early next year.

In the meantime, advocates are pressing lawmakers like Sen. Lamar Alexander, the Republican chairman of the Senate education committee who helped craft ESSA, with the argument that the anti-busing provision limits the flexibility and local control the law was meant to provide districts.

Margaret Atkinson, a spokeswoman for the senator, would not say whether he is open to removing the provision, but said he would continue working to ensure ESSA “is implemented as Congress intended.”

The anti-busing language — found in two sections of the current appropriation bills — prohibits using federal funds for transportation “to overcome racial imbalance” or “to carry out a plan of racial desegregation,” or forcing students to attend any school other than the one closest to home. (A separate education law contains a similar restriction, but ESSA exempted magnet schools from it.) The provisions emerged in the early 1970s, just after the Supreme Court ruled that busing students to schools outside their own racially isolated neighborhoods was an appropriate tool for school desegregation.

At the time, many white parents raged against what they called “forced busing.” In response, the U.S. House of Representatives passed at least one law annually from 1966 to 1977 meant to curb school integration, according to historian Jason Sokol, and in 1974 the full Congress voted in favor of an anti-busing amendment to an education bill. The restrictions in the current spending bills appear to have originated around the same time.

The attacks on busing reflect how crucial free transportation is to school desegregation, said Erica Frankenberg, a professor at Pennsylvania State University who studies segregation. Busing was included in guidelines outlining how districts should comply with desegregation requirements in the 1964 Civil Rights Act, and later upheld by the Supreme Court, she pointed out.

More recently, studies have shown that non-white parents are more likely to opt into magnet schools when they provide transportation, and that magnets that don’t offer busing are more likely to enroll students of a single race, Frankenberg said. Yet, many politicians remain reluctant to endorse busing for desegregation — which may reflect a deeper ambivalence, she added.

Resistance to busing, she said, “is a very politically acceptable way to be opposed to integration.”

Yes and No

In a first, New York officials reject 2 proposed charter schools, but sign off on 5 for New York City

PHOTO: Geoff Decker
Charter-school advocates staged a rally outside the state capitol building 2015.

New York’s top education policymakers voted Monday to approve five new charter schools in New York City – but, for the first time, rejected two proposed charters.

The moves by the state Board of Regents sent a mixed message on charter schools. While the Regents have approved more this year than at any point since 2013, the rejections suggest they won’t rubber stamp applications – even those, like the two shot down Monday, that have earned the state education department’s blessing.

Four of the approved schools will be based in the Bronx, and one in Staten Island. (Technically, Monday’s vote is preliminary and the board must finalize its decision at Tuesday’s full-board meeting.)

A new charter high school on Staten Island plans to enroll a significant number of students with disabilities — an area of great need in a borough where a quarter of students have some disability. Students will have the opportunity to graduate with as many as 60 college credits through a partnership with St. John’s University.

The Bronx charters include a new elementary school that will serve high-functioning students on the autism spectrum, an all-boys middle school inspired by an Obama-era program aimed at uplifting young men of color, and a high school for students who have fallen behind academically.

The final Bronx school is KIPP Freedom, slated to open in 2018, which will mark the first time the national network has opened a new school in New York City in six years.

“The community has tremendous support for the charter,” said Board of Regents Chancellor Betty Rosa about KIPP, who suggested the school could even help reduce segregation if sited in the right location.

The two schools the board rejected would have been located in districts in Mount Vernon, in Westchester County, and Homer, in upstate New York.

Board members raised concerns about the applications, including that their curriculums were not very innovative. They also worried that the schools would drain resources from their surrounding districts, potentially forcing them to cut extracurricular programs from traditional schools.

Regent Judith Johnson, who represents the Mount Vernon district, expressed concern that the school only planned to serve students grades 6-8, while the district is moving towards a model that keeps children in the same school from kindergarten through eighth grade. She suggested waiting to see how the district’s efforts pan out.

“I would suggest this is premature,” Johnson said. “I’m not going to support this at this time.”

The vote comes as top state officials have been skeptical of charter schools and policies regulating them.

At past meetings, Regents have wondered aloud whether the schools are serving their fair share of high-needs students. And Board of Regents Chancellor Betty Rosa and State Commissioner MaryEllen Elia have been on a warpath against a new policy that will allow some charter schools to certify their own teachers.

However, those concerns have not stopped the Regents from approving new charter schools. During a low point for approvals in 2015, when the state approved only four charters, few applications made it past the education department’s vetting process and to the board for final approval.

Since then, there has been a steady uptick in approvals. The board signed off on seven new schools last year, and is set to approve at least eight this year. (The board, which typically accepts applications in two or three rounds each year, approved three schools earlier this year.)

State education department officials on Monday also presented new ways to evaluate charter schools and decide whether they should remain open, based on proposals that the Board of Regents floated last month.

The additions to the state’s “Charter School Performance Framework” could include measures of student chronic absenteeism, the schools’ suspension rates, and the results of student and staff surveys. In previous meetings, Regents have also suggested surveying families who decide to leave charter schools.

Charter schools are already required to meet certain enrollment and retention targets, or to make “good faith efforts” to reach them. The state also considers the quality of a school’s curriculum and its outreach to families.

At Monday’s meeting, some Regents proposed adding yet another measure: whether charter schools are sharing innovative practices with the district schools.

“If the original intent [of charter schools] was to create opportunity for innovation,” said Regent Johnson, “we have to decide now, after those twenty plus years, did that happen?”