Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

More autonomy

These Denver schools want to join the district’s ‘innovation zone’ or form new zones

PHOTO: Melanie Asmar
McAuliffe Manual Middle School students at a press conference about test scores in August 2017. The school has signaled its intent to be part of a new innovation zone.

Thirteen Denver schools have signaled their desire to become more autonomous by joining the district’s first “innovation zone” or by banding together to form their own zones. The schools span all grade levels, and most of the thirteen are high-performing.

Innovation zones are often described as a “third way” to govern public schools. The four schools in Denver’s first zone, created in 2016, have more autonomy than traditional district-run schools but less than charter schools, which are publicly funded but independently run.

Denver Public Schools recently released applications for schools to join the first zone, called the Luminary Learning Network, or to form new zones. The school district, which at 92,600 students is Colorado’s largest, is nationally known for nurturing a “portfolio” of different school types and for encouraging entrepreneurship among its school principals.

The district is offering two options to schools that want to form new zones. One option is for schools to apply to form a zone that would be overseen not by the district but by a nonprofit organization. That’s how the Luminary Learning Network is set up.

Another, slightly less autonomous option is for schools to apply to form a zone that would be overseen by the district. “Some additional autonomies would be available to these schools, but many decisions would still be made by the district,” the district’s website says.

One tangible difference between the two: The principals of schools in zones overseen by the district would answer to district administrators, while the principals of schools in zones overseen by nonprofit organizations would be hired and fired by the nonprofits’ boards of directors.

Schools in both types of zones would have more control over their budgets. A key flexibility enjoyed by the four schools in the Luminary Learning Network has been the ability to opt out of certain district services and use that money to buy things that meet their students’ specific needs, such as a full-time psychologist or another special education teacher. The zone schools would like even more financial freedom, though, and are re-negotiating with the district.

The district has extended the same budgetary flexibility to the schools in Denver’s three “innovation management organizations,” or IMOs, which are networks of schools with “innovation status.”

Innovation status was created by a 2008 state law. It allows district-run schools to do things like set their own calendars and choose their own curriculum by waiving certain state and district rules. The same law allows innovation schools to join together to form innovation zones.

The difference between an innovation zone and an innovation management organization is that schools in innovation zones have the opportunity for even greater autonomy, with zones governed by nonprofit organizations poised to have the most flexibility.

The deadline for schools to file “letters of intent” to apply to join an innovation zone or form a new one was Feb. 15. Leaders of the three innovation management organizations applied to form zones of their own.

One of them – a network comprised of McAuliffe International and McAuliffe Manual middle schools – has signaled its intent to join forces with an elementary school and a high school in northeast Denver to form a new, four-school zone.

Three elementary schools – Valdez, High Tech, and Swigert – submitted multiple intent letters.

Amy Gile, principal of High Tech, said in an email that her school submitted a letter of intent to join the Luminary Learning Network and a separate letter to be part of a new zone “so that we are able to explore all options available in the initial application process. We plan to make a decision about what best meets the needs of our community prior to the application deadline.”

The application deadline is in April. There are actually two: Innovation management organizations that want to become innovation zones must file applications by April 4, and schools that want to form new zones have until April 20 to turn in their applications.

Here’s a list of the schools that filed letters of intent.

Schools that want to join the Luminary Learning Network:

Dr. Martin Luther King, Jr. Early College High School
Valdez Elementary School
High Tech Elementary School

Schools that want to form new innovation zones overseen by nonprofits:

McAuliffe International School
McAuliffe Manual Middle School
Northfield High School
Swigert International School
These four schools want to form a zone called the Northeast Denver Innovation Zone.

McGlone Academy
John Amesse Elementary School
These two schools want to form a zone called the Montbello Children’s Network.

Grant Beacon Middle School
Kepner Beacon Middle School
These two schools want to form a zone called the Beacon Network Schools IMO I-Zone.

Schools that want to form a new innovation zone overseen by the district:

High Tech Elementary School
Isabella Bird Community School
Valdez Elementary School
Swigert International School
DCIS at Ford
These five schools want to form a zone called the Empower Zone.

First Responder

Jeffco’s superintendent has some ideas about preventing school shootings — and none of them involve gun control or armed teachers

Jeffco superintendent Jason Glass at the Boys & Girls in Lakewood (Marissa Page, Chalkbeat).

Superintendent Jason Glass of the Jefferson County school district isn’t interested in talking about gun control in the wake of yet another deadly school shooting.

Home of Columbine High School, Jefferson County is no stranger to these tragedies or their aftermath, and Glass doesn’t think calls for restricting firearms will get any more traction this time than they have before. Nor is he interested in talking about arming teachers, a proposal he considers just as much of a political dead end.

“A solution is only a solution if we can actually enact it,” Glass wrote in a blog post published Monday. “We are not able to get either of these solutions passed into law so they have no impact.”

That doesn’t mean there’s nothing to talk about, he wrote. Glass lays out four ideas that he sees as more politically feasible and that might make a difference:

  • Put trained, armed law enforcement officials in every school
  • Increase funding and support for school mental health services
  • Create a federally funded center to study school safety and security
  • Change the layout of and access to school buildings to make them safer, much the way we’ve renovated airports, stadiums, and other public facilities

Glass describes these measures as “proactive, preventative, and reactive steps that would make a big impact in making our schools much safer than they are today.”

Some schools and districts already have an armed police presence on campus or offer mental health services, but Glass argues these efforts need more money, more support, and more cohesion.

“These solutions need to come from the federal level to create the scale and impact we really need,” he wrote. “Congress and the President need to act and now. … Flexibility and deference can be built into these solutions to accommodate differences across states and communities – but we have a national crisis on our hands and we have to start acting like it.”

Of course, even studying something, as Glass envisions this new center on school safety doing, can be political. Since 1996, the federal government, at the urging of the National Rifle Association, has placed tight restrictions on the ability of the Centers for Disease Control to study gun violence as a public health issue.

The blog post provoked a vigorous debate in the comments. Some called on Glass to join the national movement demanding more restrictions on firearms. This is not a time for “half measures,” one woman wrote.

Others said that turning schools into “fortresses” would work against their educational mission and questioned how well school resource officers could be trained to respond appropriately to students with special needs – or how fair the district-level threat assessment process is.

In the wake of another school shooting at Arapahoe High School in 2013, one largely forgotten outside the state, Colorado legislators passed a law that holds schools liable for missing warning signs in troubled students.

In an interview with Colorado Public Radio, Bill Woodward, a former police officer who trains schools in how to prevent violence, said more schools are doing threat assessments. But their success may require schools to take even more seriously the idea that their own students might be dangerous.

“I think the biggest barrier is the climate of the school, because I think sometimes schools are just thinking in terms of working with students, helping students out,” Woodward told CPR. “And sometimes when you’re looking at someone who’s made a threat, you have to change to the Secret Service model.”

Woodward said a more comprehensive solution may involve gun control. Schools can’t afford to wait, though.

“There is no silver bullet, speaking metaphorically, but I think gun law changes may well be needed,” he said. “I just think we have to do what we can do now, and we can do things now.”