Test experts: Exam for students with disabilities is flawed, shouldn’t count for school A-F grades

This year, it’s not ISTEP that is causing problems for state test administrators.

The problem is with a new test called the ISTAR that was given for the first time this year to Indiana students with significant cognitive disabilities, which is fewer than 2 percent of students across the state.

Flaws in the way the exam was designed are concerning enough that the state’s technical advisory committee is recommending that the Indiana State Board of Education exclude the ISTAR alternative test from 2016 A-F school grades. The committee plans to make its recommendation to the board at its work session tonight in New Albany.

“We want to make sure we’ve got a valid assessment,” said Cynthia Roach, the state board’s testing director. “All students require us to look out for them, but these are our most vulnerable students.”

The trouble with the exam has to do with the way it was split up into three shorter tests that were given throughout the year, with the first test being the easiest and the third being the most difficult.

The scale used to assign scores to students wasn’t properly designed to accommodate the variations in the difficulty levels of the three sections. That could make the scores hard to interpret and could prevent educators from comparing student scores from one year to the next.

The test also might have too few questions to be considered valid. According to Marianne Perie, a University of Kansas test expert who presented to the state’s ISTEP replacement panel last week, a test is considered highly valid if it has 50 or more questions. ISTAR, by comparison, gives fewer than 40 questions over its three parts, which just isn’t enough, Roach said.

“While I understand the desire to have as short a test as possible … if you are not giving (students) a valid assessment at the same time, you are doing them a grave disservice,” Roach said.

As with ISTEP, the English and math sections of the test are given to kids in grades 3-8 as well as in the 10th grade. Kids also take ISTAR for science and social studies in certain grades.

The state plans to administer ISTAR again next year after addressing the problems that surfaced this year, Roach said. She emphasized that none of the test design issues identified this year mean there is anything wrong with the test questions themselves, which will continue to be used on future exams.

The technical advisory committee that identified the problems with this year’s exam is asking the test writing company, Questar, to create student score reports so parents and teachers can still learn about how students performed on the test even if it won’t be used for A-F grades. Questar will also be asked to fix the scoring problem for 2017.

“This memorandum from the Technical Advisory Committee may be slightly premature,” said Samantha Hart, spokeswoman for the Indiana Department of Education, in an email. “It is my understanding that Questar has developed several options to address the concerns outlined in the Technical Advisory Committee’s memo which will be presented to the board.”

Michelle McKeown, general counsel for the state charter board and a member of a panel that is charged with coming up with a new testing program for Indiana students to replace the unpopular ISTEP test and other state exams, said she’s surprised this kind of design error made it through the state’s test design process.

Before tests are administered in Indiana schools, they’re supposed to be thoroughly vetted by the department of education.

“These are the rights of kids with disabilities,” McKeown said. “Did we design some assessment that didn’t meet their needs and provide as meaningful as possible information about their progress this year?”

The state switched to ISTAR for 2016 after using a fairly well-regarded national test from the National Center and State Collaborative in 2015. Roach said the national test incorporated feedback from not just test experts and scientists, but also teachers and teachers of students with disabilities.

Advocates for students with special needs worry that not counting ISTAR scores this year — and scores for students with special needs in general — could set a dangerous precedent. Students with disabilities are already left out when it comes to school services. They’re also sometimes blamed for pulling down school grades, said Kim Dodson, the executive director of the Arc of Indiana.

“It is a slippery slope to start down if you start not counting tests,” Dodson said in an email. “Students receiving special education services are a common argument schools use to explain low test scores or unsatisfactory grades. We need to make sure all (students’ scores) are reported.”