Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

404 not found

Sayonara, SESIS: New York City to scrap its beleaguered special education data system

PHOTO: Patrick Wall/Chalkbeat

New York City is scrapping a special education data system that has frustrated educators since it launched nearly a decade ago.

The troubles of SESIS, as the city’s Special Education Student Information System is called, are well known. Since its launch in 2011, the system — which required over $130 million to build — cost the city tens of millions of dollars in settlements, at times malfunctioned more than 800,000 times a day, and made it difficult to track whether students with disabilities are getting the services they need.

Education department officials said they have been able to “stabilize” the system in recent years. But they also have concluded that an entirely different system is needed. On Friday, they announced that they would phase SESIS out and replace it with something new — at a cost and on a timeline that is not yet clear.

The announcement comes on the eve of a City Council hearing set for Monday where council members say they will press for more transparency about special education.

“It was originally designed as a document management system,” Lauren Siciliano, the education department’s Deputy Chief Operating Officer, said about SESIS. “Think more of a filing cabinet right now as opposed to being able to follow a student through the process.”

Special education teachers often spent hours navigating a maze of drop-down menus — inputting data such as whether they met with a student and for how long — only to experience error messages that erased their answers.

Megan Moskop, a former special education teacher at M.S. 324 in Washington Heights, said she once encountered 41 error messages in two hours. What’s more, she said, the system didn’t reflect the experiences she had with her students.

“At the end of the day, I would be expected to go in, mark that they are present, mark whether they made progress toward a goal,” Moskop said. “It’s very standardized.”

It is not yet clear how quickly the education department will phase SESIS out. Officials said the city would begin a multistage process of identifying a vendor to create a new system by the end of March, then would ask for more detailed plans by the end of 2019. An official purchasing process would happen after that, Siciliano said, meaning that construction of  a new system will not begin for well over a year. Families and educators would be consulted throughout, officials said.

Linda Chen, the department’s chief academic officer, said a new system would lead to tangible improvements for students with disabilities.

“I do think that if we have clear and reliable visibility into the data it would absolutely allow us to better serve our students,” Chen said.

Flaws with SESIS have made it difficult to know how well the city is serving students with disabilities. Because the system was not set up to communicate with other city databases, city officials have had to manually tabulate data across systems. And the annual reports that show whether students are receiving required services may not be accurate because of the system’s flaws, officials have warned.

The system’s glitches also made the user experience so cumbersome that teachers had to spend time on nights and weekends entering data. An arbitrator eventually ordered the city to pay over $38 million in teacher overtime.

Additionally, the system has sparked legal action. Former Public Advocate Letitia James filed a lawsuit claiming that SESIS was to blame for some children not receiving services as well as lost Medicaid payments. Between 2012 and 2015, according to the IBO, the city collected $373 million less in Medicaid reimbursements than officials projected.

Some advocates said that given SESIS’s troubled history, it makes sense to find alternatives.

“There has to be a strong data system in place,” said Maggie Moroff, a disability policy expert at Advocates for Children, a nonprofit advocacy organization. “We are eager to see a better system to be put in place, but are really worried about that transition period.”

Advocates have also pushed the city to make the data SESIS tracks directly available to parents.

“We will absolutely be looking at that,” Siciliano said.

next steps

Charter schools racing to find new buildings as district ends their leases

PHOTO: Koby Levin
Escuela Avancemos will move to a new building.

At least two Detroit charter schools are racing against the clock to find new buildings for more than 500 students next fall after the city district decided not to renew their leases.

It’s the latest move in an ongoing effort by the Detroit Public Schools Community District to get out of the charter business, and it means another bout of uncertainty for schools that enroll hundreds of children in Detroit.

Leaders of GEE-Edmonson Academy and GEE-White Academy face the daunting challenge of finding new buildings before the start of the next school year. Another school, Escuela Avancemos, already found a new building. More schools, including Rutherford Winans Academy, have leases that expire this year, but their representatives did not return requests for comment on whether their lease was renewed.

Most students at the two schools run by Global Educational Excellence (GEE) walk every day, Superintendent Michael Conran said. If a new building can’t be found in those neighborhoods, the school’s would face new transportation challenges, casting doubt on their ability to maintain their enrollment.

“We were clearly not anticipating that the leases would not be renewed,” Conran said. “That news came pretty late, I believe it was after the New Year. That’s quite a notification to the boards in such a short period of time.”

The challenges for these schools don’t end there. The district could also decline to renew their charters for the GEE schools when they expire in June, potentially forcing them to find new backers as well as new buildings.

More than one charter school has already jumped ship. Escuela Avancemos, a small school in southwest Detroit, will begin the coming year in a new building and with a new authorizer, Central Michigan University. Officials had begun searching for a new building even before they were notified last month that their lease would not be renewed.

“For the protection of our school, we’ve had to take matters into our own hands to guarantee our future,” said Sean Townsin, principal at Escuela Avancemos.

Superintendent Nikolai Vitti made clear soon after he took the helm of the district in 2017 that he believed the district’s resources should be channeled toward its own students, not toward charter schools.

He reiterated that position last year when the district severed its ties with a three-school network of charter schools, forcing it to scramble to find new buildings and a new charter. Parents were forced to choose between an extraordinarily long commute to the new site and making an unwanted switch to another school. Enrollment was cut in half.

Supporters of the move pointed out at the time that those schools had been district schools until they were spun off by state-appointed emergency managers. In a city with lots of school options and few quality schools, they argued, some consolidation is necessary.

Most charters in Detroit are overseen by Michigan’s public universities, but 10 schools are supervised by the Detroit district.

A handful of those schools also rent their school buildings from the district, putting them in a particularly vulnerable position should the district decide that it would rather not support charter schools — its chief competitors for students and state funding.

In a statement about those schools, Chrystal Wilson, a spokeswoman for the district, said the the charter schools could eventually be replaced with district schools.

“Now that we have the leadership to rebuild the district, we need to review and maximize our property assets. This means possibly re-using currently leased schools for new DPSCD schools, replacing older buildings with high repair costs, or adding a school in an area where facility usage and class sizes are high where another traditional public school does not exist. We understand and accept if district charters are leaving for other authorizers.”

No matter the district’s plans, Conran said the Global Educational Excellence schools would continue trying to serve students.

But he asked for transparency from the district and time to plan.

“I’m just simply waiting to hear from DPS any decisions they anticipate making in as timely a manner as we need to make sure we can continue to support these students and their families,” he said.