Bumpy runway

Emails reveal months of missteps leading up to Tennessee’s disastrous online testing debut

PHOTO: Chalkbeat Photo Illustration

Tennessee education officials allowed students and teachers to go ahead with a new online testing system that had failed repeatedly in classrooms across the state, according to emails obtained by Chalkbeat.

After local districts spent millions of dollars on new computers, iPads, and upgraded internet service, teachers and students practiced for months taking the tests using MIST, an online testing system run by North Carolina-based test maker Measurement Inc.

They encountered myriad problems: Sometimes, the test questions took three minutes each to load, or wouldn’t load at all. At other times, the test wouldn’t work on iPads. And in some cases, the system even saved the wrong answers.

When students in McMinnville, a town southeast of Nashville, logged on to take their practice tests, they found some questions already filled in — incorrectly — and that they couldn’t change the answers. The unsettling implication: Even if students could take the exam, the scores would not reflect their skills.

“That is a HUGE issue to me,” Warren County High School assistant principal Penny Shockley wrote to Measurement Inc.

Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state's new online assessment.
PHOTO: Grace Tatter
Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state’s new online assessment.

The emails contain numerous alarming reports about practice tests gone awry. They also show that miscommunication between officials with the Tennessee Department of Education and Measurement Inc. made it difficult to fix problems in time for launch.

And they suggest that even as problems continued to emerge as the test date neared, state officials either failed to understand or downplayed the widespread nature of the problems to schools. As a result, district leaders who could have chosen to have students take the test on paper instead moved forward with the online system.

The messages span from October until Feb. 10, two days after the online test’s debut and cancellation hours later. Together, they offer a peek into how Tennessee wound up with a worst-case scenario: countless hours wasted by teachers and students preparing for tests that could not be taken.

October: ‘Frustration … is definitely peaking’

Leaders with the Education Department, local districts and Measurement Inc. all knew that Tennessee’s transition to online tests wouldn’t be easy. So the test maker and the department developed a plan to identify weaknesses: stress tests they called “Break MIST” to tax and troubleshoot the online system.

They all had a lot riding on a smooth rollout. Tennessee was counting on the scores to assess whether students are measuring up to new and more challenging standards, to evaluate teachers, and to decide which schools to close. Districts, even the most cash-strapped, had invested millions of dollars on new technology. And Measurement Inc., a small company headquartered in Durham, was looking to prove that it belonged in the multibillion-dollar testing industry’s top tier.

The first “Break MIST” day on Oct. 1 was a mess — as expected. Students in the eastern part of the state logged on without issue, but the system stumbled as the majority of students started their tests an hour later.

That morning, emails show that Measurement Inc. received 105 calls reporting problems. The company noted particular problems in districts using iPads. Officials from the testing company assured the state that the bugs could be fixed, and the education department passed the message on to the public.

Department officials said nearly 1.5 million practice tests were completed successfully over the course of the fall. But emails show that even on days that weren’t meant to tax the system, problems emerged.

On Oct. 20, students in some districts were taking practice tests when “everything quit,” according to a state official who summarized complaints that local technology coordinators were swapping by email.

“Not very reassuring,” wrote Randy Damewood, the IT coordinator in Coffee County.

“Not good news,” agreed John Payne, director of technology for Kingsport City Schools, who suggested that his own district’s tests were working that day.

“The frustration among teachers and central office staff is definitely peaking,” wrote Eric Brown, a state official.

But there was more frustration to come, much of it behind the scenes at the Education Department.

December to January: Communication falters

Even after Measurement Inc. and department officials worked together to address problems during practice tests, the department still wasn’t confident in the online system. They weren’t sure whether problems were due to local infrastructure or something bigger. Officials planned two more “Break MIST” days in January to find out.

But they didn’t involve Measurement Inc. in the planning, at least according to company officials who wrote to the department to say they learned of those plans only after being copied on an email sent to local superintendents by Education Commissioner Candice McQueen.

That message was one of many in which officials with the state or the testing company expressed frustration about communication in the weeks leading up to the testing period.

One tense exchange dealt with the problems faced by students taking practice tests on iPads. “Will the iPad platform be ready for primetime in the spring?” Assistant Commissioner Nakia Towns asked Measurement Inc. officials on Dec. 3. “I feel like we need to be honest on this one.”

The test maker did not email a response, and Towns raised the issue again a month later and indicated that she was still waiting for an answer. “I had asked the question very directly in December,” she wrote Measurement Inc. on Jan. 6. “We urgently need an update.”

It took five more days, until Jan. 11, for her to get an answer. A reply from a Measurement Inc. testing expert blamed the problem on Apple but suggested the company had a “workaround.”

The next day, 504 students in Dyer County, about 80 miles north of Memphis, attempted to take the exam, many of them using iPads. Not one was able to complete the test because questions took too long to load, according to a report from Measurement Inc.’s call center. (Another half-million tests were completed successfully during January, according to department officials.)

Henry Scherich
Henry Scherich

In an interview this week, McQueen told Chalkbeat that Measurement Inc. never fixed the iPad problem and that state officials called Apple themselves looking for a solution. She was still looking for an answer on Jan. 21, when she tried to speak directly with Measurement Inc. President Harry Scherich.

“She is wondering if there is any way for you to find even 15 minutes today for a call,” McQueen’s chief of staff wrote. “Commissioner will make herself available. We need to speak to someone who would be able to make a decision concerning technology in an effort to get communication to directors of schools today.”

Scherich, who was in Michigan meeting with that state’s education department, initially said he did not have time to speak with McQueen. (Measurement Inc. is one of two companies producing Michigan’s new exam.) Later that day, he agreed to speak.

McQueenEmail

McQueen said she and her team came to a conclusion the next day: The test wouldn’t work on iPads. They emailed and called districts that had purchased tablets for testing and recommended a switch to paper.

February: A last-minute warning gets too little attention

Even as tensions mounted and glitches piled up, both the department and Measurement Inc. projected confidence about what would happen on Feb. 8, when the test would go live for most Tennessee schools. State officials even invited reporters to Department of Education offices on Feb. 3 to say they were optimistic about the rollout.

But behind the scenes, they were preparing for the worst. McQueen asked the test maker’s call centers to prepare for a major outage, something a Measurement Inc. employee told her was “very unlikely.”

She also emailed districts telling them they should consider switching to paper tests if their students were waiting too long for questions to load. She gave them three days to decide.

Just 15 of Tennessee’s nearly 150 districts took her up on the offer, McQueen told Chalkbeat.

But emails show that the state knew that most districts were having difficulties. When one district’s technology coordinator asked the state for a list of districts ready for the online exam, officials came up short.

“I don’t think I can answer that with any confidence,” the department’s top technology officer wrote.

Five days later, on Monday, Feb. 8, the test officially began. Again, the system handled the first set of test takers but broke down when the rest of the state’s students logged on.

As students stopped being able to connect or saw their tests freeze, emails show that technology directors began frantically contacting each other.

“Has anyone else had MIST drop out on them?” the director from Houston County Schools asked. A chorus of technology directors from other districts replied in the affirmative.

Within hours, Tennessee had ended its foray into online testing. First, McQueen told districts to suspend the exams, then directed them to give up on the online platform altogether.

“We are not confident in the system’s ability to perform consistently,” she wrote in an email to school superintendents that afternoon.

McQueen told Chalkbeat that officials started the day “in good faith,” with an assumption that Measurement Inc. had resolved problems adequately. Scherich told Chalkbeat that he’s still unconvinced that the problems were the company’s fault. He suggested that Tennessee’s decision to cancel testing came too soon.

Either way, the department’s top technology official put it simply when he emailed McQueen on the day of the failure. “It appears that greater procedural and operational rigor could have prevented the network outage,” Cliff Lloyd wrote to McQueen.

The debacle was just what Ravi Gupta, the CEO of a Nashville-based charter school, was worried about when he pressed the state in January for more transparency about the status of the online platform.

“It would be a betrayal of our students’ hard work if adult technical failures stood in the way of their success,” Gupta wrote to McQueen.

In the end, that’s exactly what happened.

Clarification (June 28, 2016): This story has also been revised to clarify the impact of the department’s communications on district testing decisions. It has also been updated to include new information about successful practice tests.

Academic Accountability

Coming soon: Not one, but two ratings for every Chicago school

Starting this month, Chicago schools will have to juggle two ratings — one from the school district, and another from the state.

The Illinois State Board of Education is scheduled to release on October 31 its annual report cards for schools across the state. This year, for the first time, each school will receive one of four quality stamps from the state: an “exemplary” or “commendable” rating signal the school is meeting standards while an “underperforming” or “lowest performing” designation could trigger intervention, according to state board of education spokeswoman Jackie Matthews.

A federal accountability law, the Every Student Succeeds Act, requires these new ratings.

To complicate matters, the city and state ratings are each based on different underlying metrics and even a different set of standardized tests. The state ratings, for example, are based on a modified version of the PARCC assessment, while Chicago ratings are based largely on the NWEA. The new state ratings, like those the school district issues, can be given out without observers ever having visited a classroom, which is why critics argue that the approach lacks the qualitative metrics necessary to assess the learning, teaching, and leadership at individual schools.

Patricia Brekke, principal at Back of the Yards College Preparatory High School, said she’s still waiting to see how the ratings will be used, “and how that matters for us,” but that parents at her school aren’t necessarily focused on what the state says.

“What our parents usually want to know is what [Chicago Public Schools] says about us, and how we’re doing in comparison to other schools nearby that their children are interested in,” she said.

Educators at Chicago Public Schools understand the power of school quality ratings.  The district already has its own five-tiered rating system: Level 1+ and Level 1 designate the highest performing schools, Level 2+ and Level 2 describe for average and below average performing schools, respectively, and Level 3, the lowest performance rating, is for schools in need of “intensive intervention.” The ratings help parents decide where to enroll their children, and are supposed to signal to the district that the school needs more support. But the ratings are also the source of angst — used to justify replacing school leaders, closing schools, or opening new schools in neighborhoods where options are deemed inadequate.

In contrast, the state’s school quality designations actually target underperforming and lowest-performing schools with additional federal funding and support with the goal of improving student outcomes. Matthews said schools will work with “school support managers” from the state to do a self-inquiry and identify areas for improvement. She described Chicago’s school quality rating system as “a local dashboard that they have developed to communicate with their communities.”

Staff from the Illinois State Board of Education will be traveling around the state next week to meet with district leaders and principals to discuss the new accountability system, including the ratings. They’ll be in Bloomington, Marion, O’Fallon, Chicago, and Melrose Park. The Chicago meeting is Wednesday, Oct. 24, at 5 p.m. at Chicago Public Schools headquarters.

Rae Clementz, director of assessment and accountability at the state board said that a second set of ratings reveals “that there are multiple valid ways to look at school quality and success; it’s a richer picture.”

Under auspices of the Every Student Succeeds Act, the state school report cards released at the end of the month for elementary schools are 75 percent based on academics, including English language arts and math test scores, English learner progress as measured by the ACCESS test, and academic growth. The other 25 percent reflects the school climate and success, such as attendance and chronic absenteeism.

Other measures are slated to be phased in over the next several years, including academic indicators like science proficiency and school quality indicators, such as school climate surveys of staff, students and parents

High school designations take a similar approach with English and math test scores but will take into account graduation rates, instead of academic growth, and also includes the percentage of  9th graders on track to graduate — that is freshmen who earn 10 semester credits, and no more than one semester F in a core course.

Critics of Chicago’s school rating system argue that the ratings correlate more with socioeconomic status and race than they do school quality, and say little about what’s happening in classrooms and how kids are learning. Chicago does try to mitigate these issues with a greater emphasis on growth in test scores rather than absolute attainment, school climate surveys, and including academic growth by priority groups, like African-American, Latino, ELL, and students in special education.

Cory Koedel, a professor of economics and public policy at the University of Missouri, said that many rating systems basically capture poverty status with a focus on how high or low students score on tests. Chicago’s approach is fairer than that of many other school systems.

“What I like about this is it does seem to have a high weight on growth and lower weight on attainment levels,” he said.

Morgan Polikoff, a professor at University of Southern California’s school of education, said that Chicago’s emphasis on student growth is a good thing “if the purpose of the system is to identify schools doing a good job educating kids.”

Chicago weights 50 percent of the rating on growth, but he’s seen 35 to as low as 15 percent at other districts. But he said the school district’s reliance on the NWEA test rather than the PARCC test used in the state school ratings was atypical.

“It’s not a state test, and though they say it aligns with standards, I know from talking to educators that a lot of them feel the tests are not well aligned with what they are supposed to be teaching,” he said. “It’s just a little odd to me they would have state assessment data, which is what they are held accountable for with the state, but use the other data.”

He’s skeptical about school systems relying too heavily on standardized test scores, whether the SAT, PARCC or NWEA, because “You worry that now you’re just turning the curriculum to test prep, and that’s an incentive you don’t want to create for educators.”

He said the high school measures in particular include a wide array of measures, including measures that follow students into college, “so I love that.”

“I really like the idea of broadening the set of indicators on which we evaluate schools and encouraging schools to really pay attention to how well they prepare students for what comes next,” he said.

Miseducation

In Colorado’s high-poverty schools, many teachers are just starting their careers

PHOTO: Yesenia Robles/Chalkbeat
A first-grade student reads in Spanish in a biliteracy classroom at Dupont Elementary in Adams 14.

This story is part of a partnership between Chalkbeat and the nonprofit investigative news organization ProPublica. Using federal data from Miseducation, an interactive database built by ProPublica, we are publishing a series of stories exploring inequities in education at the local level.

Koli Jamerson’s residency program gave her tools that she uses every day as a teacher, ideas for developing engaging lessons and for working with English language learners.

But it didn’t teach her how to help a student who explodes in anger because the police were at her house the night before on a domestic violence call or who cries all day because she doesn’t know where she’ll sleep that night.

Jamerson, now in her third year of teaching first grade at Altura Elementary in the Aurora school district, is still standing. She remains committed to her profession in large part due to the help of veteran teachers on her team, who provided advice as she found her footing those first couple of years.

“It helps keep things in perspective,” Jamerson said of her conversations with more experienced educators. “Otherwise, I would have been talking to a bunch of other teachers who were also drowning, and we would have drowned together.”

It’s getting hard for new teachers in Colorado to find those support systems, since the percentage of Colorado’s teachers in their first or second year in the classroom is among the highest in the nation. In 2015-16, the most recent year for which federal data is available, 17 percent of Colorado teachers were new to the classroom, compared with 12 percent nationally. Only Tennessee, Arizona, and Washington, D.C., rank higher. As recently as 2011, less than 11 percent of Colorado’s teachers were new to the classroom.

This information comes from a new interactive database from the investigative news organization ProPublica. It draws on data collected by the U.S. Department of Education’s Office of Civil Rights and for the first time allows parents to easily search their school and district and compare it with others in the region. 

The rates of inexperienced teachers are even higher in certain rural districts and in districts where lots of students come from low-income families and face more challenges. Those districts also often have high numbers of students of color. In contrast, whiter, more affluent suburban districts tend to have low rates of inexperienced teachers.

And those numbers have significant ramifications for Colorado’s students: New teachers can bring energy and innovation to the classroom, and some, especially those with support and guidance, manage to thrive early on. But students with inexperienced teachers tend to have lower test scores on average, according to numerous studies, and new teachers often get lower scores in classroom management from their principals.

Most teachers will readily admit it takes several years to get your bearings in a profession for which no amount of classroom learning can fully prepare you.

“In reality, you get second grade one time, you get third grade one time, as a kid,” said David Singer, founder of Denver’s University Prep charter network, which has shown impressive test results even with plenty of relatively inexperienced teachers. “You deserve an excellent educator.”

Chalkbeat reviewed more recent state data that follows the typical federal definition of “inexperienced teachers” — teachers with less than three full years of classroom experience — and found that the broad trends remain true and in many cases are even more severe than they appear in the federal data. Statewide, one in four Colorado teachers was classified as inexperienced. Last school year, 31 percent of Denver Public Schools teachers were in their first three years on the job, compared with just 7 percent of teachers in the more affluent Boulder Valley School District.

The Adams 14 district, based in the working-class suburb of Commerce City, is one of the lowest-performing in the state. Last year, 45 percent of teachers there were considered inexperienced, compared with 8 percent in the south suburban Littleton district. 

In districts with so many new teachers, it becomes inevitable that students there will encounter educators who haven’t yet reached their prime.

“When a teacher is new to the profession, as with any profession, they’re not as effective,” said Allison Atteberry, an assistant professor in the research and evaluation methods program at the University of Colorado’s School of Education. “There’s a really steep learning curve in those first years. That can’t really be avoided. But if there are more of those teachers, then more students will be exposed to those teachers. And if you have districts with more at-risk students, that has major equity implications.”

Atteberry said the numbers don’t surprise her, and they reflect a perfect storm in the state’s teacher corps. Colorado has experienced rapid population growth, increasing the demand for teachers, at the same time that experienced teachers are retiring or changing careers. That means more new teachers in Colorado classrooms, even as fewer students are entering teacher preparation programs.

Colorado’s low teacher pay exacerbates retention problems. Colorado ranks 30th for teacher pay, and when those salaries are adjusted for cost of living, it falls to 44th. The competitiveness of its teacher salaries is the lowest in the nation, meaning that people who go into teaching take a bigger salary hit compared to their peers with similar levels of education. Nationally, 1 in 10 teachers will leave the profession after their first year, and many more never reach the five-year mark.

Districts around the state are asking voters to raise taxes this November in part to raise teacher pay. Better pay for educators is also a major part of the campaign for Amendment 73, a $1.6 billion statewide tax increase for schools that appears on the ballot. But Colorado voters have so far been reluctant to raise statewide taxes for schools, and critics say there is no guarantee the money will make it into teachers’ paychecks.

Atteberry said raising pay would help mitigate these trends. Beyond that, there isn’t a lot of solid research on the best ways to keep teachers in the classroom, she said, but coaching and support from other teachers can make a difference. Denver is trying a new program to ease the transition for novice teachers with more time spent observing and learning from veterans before getting sole responsibility for a classroom. But just as with teacher salaries, providing adequate coaching is expensive. And the more newcomers there are, the harder it is provide meaningful support to novices.

Denver metro area inexperienced teachers

DISTRICT Teachers with less than three years experience Students receiving subsidized lunches
Adams 14 45 percent 87.3 percent
Dougco 39 percent 12.4 percent
Sheridan 33 percent 90.4 percent
27J (Brighton) 33 percent 37.7 percent
Denver 31 percent 67.2 percent
Jeffco 31 percent 31.7 percent
Aurora 29 percent 68.7 percent
Englewood 29 percent 66 percent
Westminster 24 percent 81.4 percent
Mapleton 21 percent 60.6 percent
St. Vrain 15 percent 30.6 percent
Adams 12 11 percent 39.9 percent
Cherry Creek 9 percent 30.0 percent
Littleton 8 percent 16.8 percent
Boulder Valley 7 percent 19 percent

Source: Colorado Department of Education, 2017-18 school year

This year, for the first time, Annalee Peterson has her own fifth-grade classroom in Columbia Elementary in Colorado Springs, where a large portion of the students are homeless or face other challenges. Before starting an alternative certification process, she ran reading groups as a paraprofessional in the same building for four years. And years before that, she dropped out of a Teach for America placement in a Newark high school where she felt alone and unsupported.

Peterson said her certification program includes intensive classroom observation and feedback that has been invaluable as she made the transition from para to teacher. She also has a trusting relationship with her building principal, who encouraged her to become a teacher.

“I think every new teacher should have a mentor,” she said. “I see other teachers come in, and they don’t have it.”

Peterson said she also benefits from her school’s skilled full-time counselor, something many Colorado schools don’t have.

“If we have a kid with a fair amount of trauma, and they get triggered, they have someone they can go talk to,” she said. “And that’s a huge help. They are getting their emotional needs met, and when they come back to the classroom, they’re ready to work and ready to focus.”

The Adams 14 school district, which has spent eight years on a state watchlist due to its low-performing schools, has the highest rate of inexperienced teachers in the Denver metro area. The 7,000-student district has experienced a lot of turnover not just at the classroom level, but at the highest tiers of leadership.

With an urgent need to improve school performance, Mark Langston, the district’s new manager of educator effectiveness, tries to put a positive face on the large number of new educators that arrive each year.

“I’d rather have a phenomenal teacher for one year, than a bad teacher for many years,” Langston said. “Strong systems have a nice blend of experience.”

At the same time, he’s trying to improve the support those new teachers receive by making changes to the district’s five-day induction program to better meet their individual needs. The thinking is that a 40-year-old switching careers after running a business for 20 years might need different training from a 22-year-old recent college graduate. He’s also trying to match new teachers with mentors earlier in the school year.

But sometimes there aren’t enough mentors or he’s had to make exceptions to allow less experienced teachers to become mentors.

“They are mentoring each other,” said Barb McDowell, president of the Adams 14 teachers union, who says the churn takes a toll on teacher and student morale. “There are no veteran teachers there to help.”

Kevin Clark, a senior at Adams City High, said he always felt supported by his teachers in the district, but very few of them are still there as he enters his final year.

“For the seniors, it’s been rough,” he said. “We really value our support systems. The new teachers are trying to adjust and get their footing, but just because you send in a batch of new teachers, doesn’t mean everything is fine.”

The Denver schools with the highest percentages of inexperienced teachers in 2015-16 include a number of alternative high schools, high-poverty district-run schools, and charter schools. Some of the charter schools are part of high-performing networks whose students do well on state tests.

One of them is University Prep. The homegrown Denver network has two elementary schools, one of which posted the most academic progress in Colorado on state math tests in 2017. But in 2015, the network had just one school — and 42 percent of the teachers there were in their first or second year of teaching, according to the federal data.

At University Prep, some first-year teachers have taken part in a teacher residency program or in a program that has college students work as paraprofessionals while earning their degrees.

“When you think about that individual exiting their undergraduate [education] having spent four years in a building with master teachers, getting all the supports they need to grow, they’re ready to teach on Day 1,” said Singer, the network’s founder.

PHOTO: Eric Gorski/Chalkbeat
Students at University Prep Elementary’s flagship school in Denver

Even so, the network provides its first-year teachers with extra support, he said, such as real-time coaching in the classroom, opportunities to observe more experienced teachers, and help with how to plan a lesson or conduct a parent-teacher conference.

Atteberry said successful charter schools with high rates of inexperienced teachers may be doing something different in the hiring process, looking for “spark teachers who really want to make a difference.”

The high rates of new teachers at some charter schools raise questions, though, about how sustainable the work environment is, and some of these same “spark” teachers may never intend to make a lifelong career of it and instead move on to other challenges. Asked about turnover, Singer said some University Prep teachers have left to pursue careers in medicine and law.

Denver metro area data show another exception to the trend in Douglas County. It’s an affluent and sprawling district southwest of Denver where just 12 percent of students get subsidized lunches, a proxy for poverty. But in 2015-16, 31 percent of teachers were in their first or second year in the classroom, and in 2017-18, 39 percent had less than three years experience.

Kallie Leyba, president of the Douglas County Federation, the teachers union there, said Douglas used to be a “destination district” that teachers aspired to work for. But political upheaval, the election of a conservative school board that has since been replaced, and a “market rate” pay structure that remains have caused experienced teachers to leave in droves — some for much higher salaries in nearby Cherry Creek schools.

The Douglas County pay scale means that teachers with the same amount of experience might make very different salaries. Leyba herself faced the prospect of a lower ceiling on her salary when her building principal asked her to switch from a first grade to a second grade classroom because first-grade teachers are more in demand.

“Even though I knew this was a crazy system, it really hurt to feel like my value had gone down in the eyes of my principal,” she said.

What could Colorado do to get more of today’s inexperienced teachers to become tomorrow’s veteran educators?

Money is a big part of the answer. As it stands, Colorado teachers can earn significantly more money by moving to another state, and with teacher salaries less competitive here than elsewhere, teachers also look to other professions that offer less stress along with better pay.

“The No. 1 thing we should do is increase the prestige and value of teachers in society, and the way we signal that in our society is through salary and compensation,” Atteberry said. “That has a huge influence on who goes into the profession and on who stays.

“This is not an easy change because it costs a lot of money, and it also requires us to change how we think about teachers, but it is the policy that would be most effective.”

Chalkbeat reporters Melanie Asmar and Yesenia Robles contributed reporting to this story.