Bumpy runway

Emails reveal months of missteps leading up to Tennessee’s disastrous online testing debut

PHOTO: Chalkbeat Photo Illustration

Tennessee education officials allowed students and teachers to go ahead with a new online testing system that had failed repeatedly in classrooms across the state, according to emails obtained by Chalkbeat.

After local districts spent millions of dollars on new computers, iPads, and upgraded internet service, teachers and students practiced for months taking the tests using MIST, an online testing system run by North Carolina-based test maker Measurement Inc.

They encountered myriad problems: Sometimes, the test questions took three minutes each to load, or wouldn’t load at all. At other times, the test wouldn’t work on iPads. And in some cases, the system even saved the wrong answers.

When students in McMinnville, a town southeast of Nashville, logged on to take their practice tests, they found some questions already filled in — incorrectly — and that they couldn’t change the answers. The unsettling implication: Even if students could take the exam, the scores would not reflect their skills.

“That is a HUGE issue to me,” Warren County High School assistant principal Penny Shockley wrote to Measurement Inc.

Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state's new online assessment.
PHOTO: Grace Tatter
Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state’s new online assessment.

The emails contain numerous alarming reports about practice tests gone awry. They also show that miscommunication between officials with the Tennessee Department of Education and Measurement Inc. made it difficult to fix problems in time for launch.

And they suggest that even as problems continued to emerge as the test date neared, state officials either failed to understand or downplayed the widespread nature of the problems to schools. As a result, district leaders who could have chosen to have students take the test on paper instead moved forward with the online system.

The messages span from October until Feb. 10, two days after the online test’s debut and cancellation hours later. Together, they offer a peek into how Tennessee wound up with a worst-case scenario: countless hours wasted by teachers and students preparing for tests that could not be taken.

October: ‘Frustration … is definitely peaking’

Leaders with the Education Department, local districts and Measurement Inc. all knew that Tennessee’s transition to online tests wouldn’t be easy. So the test maker and the department developed a plan to identify weaknesses: stress tests they called “Break MIST” to tax and troubleshoot the online system.

They all had a lot riding on a smooth rollout. Tennessee was counting on the scores to assess whether students are measuring up to new and more challenging standards, to evaluate teachers, and to decide which schools to close. Districts, even the most cash-strapped, had invested millions of dollars on new technology. And Measurement Inc., a small company headquartered in Durham, was looking to prove that it belonged in the multibillion-dollar testing industry’s top tier.

The first “Break MIST” day on Oct. 1 was a mess — as expected. Students in the eastern part of the state logged on without issue, but the system stumbled as the majority of students started their tests an hour later.

That morning, emails show that Measurement Inc. received 105 calls reporting problems. The company noted particular problems in districts using iPads. Officials from the testing company assured the state that the bugs could be fixed, and the education department passed the message on to the public.

Department officials said nearly 1.5 million practice tests were completed successfully over the course of the fall. But emails show that even on days that weren’t meant to tax the system, problems emerged.

On Oct. 20, students in some districts were taking practice tests when “everything quit,” according to a state official who summarized complaints that local technology coordinators were swapping by email.

“Not very reassuring,” wrote Randy Damewood, the IT coordinator in Coffee County.

“Not good news,” agreed John Payne, director of technology for Kingsport City Schools, who suggested that his own district’s tests were working that day.

“The frustration among teachers and central office staff is definitely peaking,” wrote Eric Brown, a state official.

But there was more frustration to come, much of it behind the scenes at the Education Department.

December to January: Communication falters

Even after Measurement Inc. and department officials worked together to address problems during practice tests, the department still wasn’t confident in the online system. They weren’t sure whether problems were due to local infrastructure or something bigger. Officials planned two more “Break MIST” days in January to find out.

But they didn’t involve Measurement Inc. in the planning, at least according to company officials who wrote to the department to say they learned of those plans only after being copied on an email sent to local superintendents by Education Commissioner Candice McQueen.

That message was one of many in which officials with the state or the testing company expressed frustration about communication in the weeks leading up to the testing period.

One tense exchange dealt with the problems faced by students taking practice tests on iPads. “Will the iPad platform be ready for primetime in the spring?” Assistant Commissioner Nakia Towns asked Measurement Inc. officials on Dec. 3. “I feel like we need to be honest on this one.”

The test maker did not email a response, and Towns raised the issue again a month later and indicated that she was still waiting for an answer. “I had asked the question very directly in December,” she wrote Measurement Inc. on Jan. 6. “We urgently need an update.”

It took five more days, until Jan. 11, for her to get an answer. A reply from a Measurement Inc. testing expert blamed the problem on Apple but suggested the company had a “workaround.”

The next day, 504 students in Dyer County, about 80 miles north of Memphis, attempted to take the exam, many of them using iPads. Not one was able to complete the test because questions took too long to load, according to a report from Measurement Inc.’s call center. (Another half-million tests were completed successfully during January, according to department officials.)

Henry Scherich
Henry Scherich

In an interview this week, McQueen told Chalkbeat that Measurement Inc. never fixed the iPad problem and that state officials called Apple themselves looking for a solution. She was still looking for an answer on Jan. 21, when she tried to speak directly with Measurement Inc. President Harry Scherich.

“She is wondering if there is any way for you to find even 15 minutes today for a call,” McQueen’s chief of staff wrote. “Commissioner will make herself available. We need to speak to someone who would be able to make a decision concerning technology in an effort to get communication to directors of schools today.”

Scherich, who was in Michigan meeting with that state’s education department, initially said he did not have time to speak with McQueen. (Measurement Inc. is one of two companies producing Michigan’s new exam.) Later that day, he agreed to speak.

McQueenEmail

McQueen said she and her team came to a conclusion the next day: The test wouldn’t work on iPads. They emailed and called districts that had purchased tablets for testing and recommended a switch to paper.

February: A last-minute warning gets too little attention

Even as tensions mounted and glitches piled up, both the department and Measurement Inc. projected confidence about what would happen on Feb. 8, when the test would go live for most Tennessee schools. State officials even invited reporters to Department of Education offices on Feb. 3 to say they were optimistic about the rollout.

But behind the scenes, they were preparing for the worst. McQueen asked the test maker’s call centers to prepare for a major outage, something a Measurement Inc. employee told her was “very unlikely.”

She also emailed districts telling them they should consider switching to paper tests if their students were waiting too long for questions to load. She gave them three days to decide.

Just 15 of Tennessee’s nearly 150 districts took her up on the offer, McQueen told Chalkbeat.

But emails show that the state knew that most districts were having difficulties. When one district’s technology coordinator asked the state for a list of districts ready for the online exam, officials came up short.

“I don’t think I can answer that with any confidence,” the department’s top technology officer wrote.

Five days later, on Monday, Feb. 8, the test officially began. Again, the system handled the first set of test takers but broke down when the rest of the state’s students logged on.

As students stopped being able to connect or saw their tests freeze, emails show that technology directors began frantically contacting each other.

“Has anyone else had MIST drop out on them?” the director from Houston County Schools asked. A chorus of technology directors from other districts replied in the affirmative.

Within hours, Tennessee had ended its foray into online testing. First, McQueen told districts to suspend the exams, then directed them to give up on the online platform altogether.

“We are not confident in the system’s ability to perform consistently,” she wrote in an email to school superintendents that afternoon.

McQueen told Chalkbeat that officials started the day “in good faith,” with an assumption that Measurement Inc. had resolved problems adequately. Scherich told Chalkbeat that he’s still unconvinced that the problems were the company’s fault. He suggested that Tennessee’s decision to cancel testing came too soon.

Either way, the department’s top technology official put it simply when he emailed McQueen on the day of the failure. “It appears that greater procedural and operational rigor could have prevented the network outage,” Cliff Lloyd wrote to McQueen.

The debacle was just what Ravi Gupta, the CEO of a Nashville-based charter school, was worried about when he pressed the state in January for more transparency about the status of the online platform.

“It would be a betrayal of our students’ hard work if adult technical failures stood in the way of their success,” Gupta wrote to McQueen.

In the end, that’s exactly what happened.

Clarification (June 28, 2016): This story has also been revised to clarify the impact of the department’s communications on district testing decisions. It has also been updated to include new information about successful practice tests.

First Person

I’m a principal who thinks personalized learning shouldn’t be a debate.

PHOTO: Lisa Epstein
Lisa Epstein, principal of Richard H. Lee Elementary, supports personalized learning

This is the first in what we hope will be a tradition of thoughtful opinion pieces—of all viewpoints—published by Chalkbeat Chicago. Have an idea? Send it to cburke@chalkbeat.org

As personalized learning takes hold throughout the city, Chicago teachers are wondering why a term so appealing has drawn so much criticism.

Until a few years ago, the school that I lead, Richard H. Lee Elementary on the Southwest Side, was on a path toward failing far too many of our students. We crafted curriculum and identified interventions to address gaps in achievement and the shifting sands of accountability. Our teachers were hardworking and committed. But our work seemed woefully disconnected from the demands we knew our students would face once they made the leap to postsecondary education.

We worried that our students were ill-equipped for today’s world of work and tomorrow’s jobs. Yet, we taught using the same model through which we’d been taught: textbook-based direct instruction.

How could we expect our learners to apply new knowledge to evolving facts, without creating opportunities for exploration? Where would they learn to chart their own paths, if we didn’t allow for agency at school? Why should our students engage with content that was disconnected from their experiences, values, and community?

We’ve read articles about a debate over personalized learning centered on Silicon Valley’s “takeover” of our schools. We hear that Trojan Horse technologies are coming for our jobs. But in our school, personalized learning has meant developing lessons informed by the cultural heritage and interests of our students. It has meant providing opportunities to pursue independent projects, and differentiating curriculum, instruction, and assessment to enable our students to progress at their own pace. It has reflected a paradigm shift that is bottom-up and teacher led.

And in a move that might have once seemed incomprehensible, it has meant getting rid of textbooks altogether. We’re not alone.

We are among hundreds of Chicago educators who would welcome critics to visit one of the 120 city schools implementing new models for learning – with and without technology. Because, as it turns out, Chicago is fast becoming a hub for personalized learning. And, it is no coincidence that our academic growth rates are also among the highest in the nation.

Before personalized learning, we designed our classrooms around the educator. Decisions were made based on how educators preferred to teach, where they wanted students to sit, and what subjects they wanted to cover.

Personalized learning looks different in every classroom, but the common thread is that we now make decisions looking at the student. We ask them how they learn best and what subjects strike their passions. We use small group instruction and individual coaching sessions to provide each student with lesson plans tailored to their needs and strengths. We’re reimagining how we use physical space, and the layout of our classrooms. We worry less about students talking with their friends; instead, we ask whether collaboration and socialization will help them learn.

Our emphasis on growth shows in the way students approach each school day. I have, for example, developed a mentorship relationship with one of our middle school students who, despite being diligent and bright, always ended the year with average grades. Last year, when she entered our personalized learning program for eighth grade, I saw her outlook change. She was determined to finish the year with all As.

More than that, she was determined to show that she could master anything her teachers put in front of her. She started coming to me with graded assignments. We’d talk about where she could improve and what skills she should focus on. She was pragmatic about challenges and so proud of her successes. At the end of the year she finished with straight As—and she still wanted more. She wanted to get A-pluses next year. Her outlook had changed from one of complacence to one oriented towards growth.

Rather than undermining the potential of great teachers, personalized learning is creating opportunities for collaboration as teachers band together to leverage team-teaching and capitalize on their strengths and passions. For some classrooms, this means offering units and lessons based on the interests and backgrounds of the class. For a couple of classrooms, it meant literally knocking down walls to combine classes from multiple grade-levels into a single room that offers each student maximum choice over how they learn. For every classroom, it means allowing students to work at their own pace, because teaching to the middle will always fail to push some while leaving others behind.

For many teachers, this change sounded daunting at first. For years, I watched one of my teachers – a woman who thrives off of structure and runs a tight ship – become less and less engaged in her profession. By the time we made the switch to personalized learning, I thought she might be done. We were both worried about whether she would be able to adjust to the flexibility of the new model. But she devised a way to maintain order in her classroom while still providing autonomy. She’s found that trusting students with the responsibility to be engaged and efficient is both more effective and far more rewarding than trying to force them into their roles. She now says that she would never go back to the traditional classroom structure, and has rediscovered her love for teaching. The difference is night and day.

The biggest change, though, is in the relationships between students and teachers. Gone is the traditional, authority-to-subordinate dynamic; instead, students see their teachers as mentors with whom they have a unique and individual connection, separate from the rest of the class. Students are actively involved in designing their learning plans, and are constantly challenged to articulate the skills they want to build and the steps that they must take to get there. They look up to their teachers, they respect their teachers, and, perhaps most important, they know their teachers respect them.

Along the way, we’ve found that students respond favorably when adults treat them as individuals. When teachers make important decisions for them, they see learning as a passive exercise. But, when you make it clear that their needs and opinions will shape each school day, they become invested in the outcome.

As our students take ownership over their learning, they earn autonomy, which means they know their teachers trust them. They see growth as the goal, so they no longer finish assignments just to be done; they finish assignments to get better. And it shows in their attendance rates – and test scores.

Lisa Epstein is the principal of Richard H. Lee Elementary School, a public school in Chicago’s West Lawn neighborhood serving 860 students from pre-kindergarten through eighth grade.

Editor’s note: This story has been updated to reflect that Richard H. Lee Elementary School serves 860 students, not 760 students.

Facilities

These 102 schools failed latest round of ‘blitz inspections’

PHOTO: Tim Boyle / Getty Images
Taft High School is one of 102 schools that will have to be reinspected.

Chicago Public Schools said Tuesday that 102 schools will require reinspection for cleanliness before students return to class in the fall. The district has been conducting “blitz inspections” at schools to help address widespread concerns about filthy conditions, including rats and rodent droppings.

The Chicago Sun-Times reported earlier in the year that complaints of a rodent infestation at a South Side elementary school had spurred an initial round of investigations, and that 91 of 125 schools failed them. The story brought citywide attention to the issue and raised questions about CPS’ decision to transition the work of keeping schools clean to two private contractors: Aramark, which is based in Philadelphia, and SodexoMAGIC, which is a joint venture between the French company Sodexo Inc. and Beverly Hills, California,-based Magic Johnson Enterprises.

Since 2014, the district has spent more than $400 million on contracts with the two companies.

CPS said in a statement Tuesday that it is “committed to carrying out a multi-pronged plan” that includes adding 200 additional custodians who are deep cleaning schools this summer. Of those, 100 custodians will remain with the district once the new school year begins. A district spokeswoman said monthly inspections will continue and that a “stronger facilities services structure” that employs one building manager to oversee janitorial and engineering services at each school will yield better results.

Jesse Sharkey, the vice president of the Chicago Teachers Union, said that the additional custodians do little to make up for the mess. “(Mayor Rahm) Emanuel made a token commitment to increase full-time custodial staff by 100 next fall—about a tenth of the staff that was cut when (he) moved to privatize janitorial and facilities management services for CPS, and a fraction of what’s needed,” Sharkey said in a statement.

Schools that have not yet passed an inspection have received orders for actions, structures, and timelines for improvement, the district said. CPS does not inspect charter, contract, alternative, or options schools that operate outside of district-managed facilities.

Here’s a list of the schools that require reinspection.

ADDAMS
ALCOTT ES
ALDRIDGE
ASHBURN
AZUELA
BARTON
BELMONT-CRAGIN
BENNETT
CAMERON
CANTY
CARDENAS
CARROLL-ROSENWALD
CASTELLANOS
CHICAGO AGRICULTURE HS
CLINTON
COOK
COONLEY
CORLISS HS
CURTIS
DAVIS M
DUBOIS
DUNNE
DURKIN PARK
EARHART
EARLE
ELLINGTON
ERICSON
FAIRFIELD
FORT DEARBORN
FOSTER PARK
FRAZIER PROSPECTIVE
GALLISTEL
GARVY
GOETHE
HALEY
HARVARD
HAUGAN
HEARST
HEFFERAN
HOLMES
HOPE HS
HOPE INSTITUTE
HURLEY
IRVING
JACKSON M
JOPLIN
JORDAN
KENNEDY HS
KERSHAW
KIPLING
LANE TECH HS
LANGFORD
LAVIZZO
Lee Elementary
MARSHALL HS
MASON
MAYS
MCDOWELL
MCKAY
MORGAN PARK HS
MORRILL
MULTICULTURAL HS
NOBLE – COMER
NORTHSIDE LEARNING HS
NORTHSIDE PREP HS
NORTHWEST
OGLESBY
OTIS
OWENS
PARKER
PARKSIDE
PENN
PETERSON
POE
PRITZKER
PULLMAN
REVERE
RICKOVER MILITARY HS
RUDOLPH
RUGGLES
SCAMMON
SKINNER West
SMITH
SOUTH SHORE ES
SOUTH SHORE INTL HS
SPRY ES
SULLIVAN HS
SUTHERLAND
TAFT HS
TARKINGTON
TAYLOR
TELPOCHCALLI
THORP J
URBAN PREP – WEST HS
VOLTA
WASHINGTON H ES
WASHINGTON HS
WEBSTER
WELLS ES
WESTINGHOUSE HS
WHITNEY
WILDWOOD