Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

words of advice

Here’s advice from a social worker on how schools can support transgender students right now

PHOTO: Getty Images
A flag for transgender and gender noncomforming people is held up at a rally for LGBTQ rights at Washington Square Park.

Soon after news broke that the Trump administration could further roll back civil rights protections for transgender students, one New York City teacher sent an email blast to her fellow educators.

She was searching for materials to use in biology class that reflect people of different gender identities, but couldn’t find anything.

Many city educators may similarly grapple with how to support transgender students after it was reported that the Trump administration is considering whether to narrowly define gender based on a person’s biology at birth — a move that could have implications for how sex discrimination complaints in schools are handled under federal Title IX.

Olin Winn-Ritzenberg — a social worker at the Lesbian, Gay, Bisexual and Transgender Community Center — has some tips for navigating the questions and emotions this latest proposal might surface. He runs a support group for transgender teens and their peers who want to be allies, and says the most important advice is to just be willing to talk and listen.

“I don’t think it’s the kind of thing that you want to wait until somebody is in crisis,” he said. “By bringing it up ourselves, we’re modeling support.”

Here’s what he had to say about recognizing transgender students, the protections that New York City and state offer, and some mistakes to avoid.

This interview has been edited for length and clarity.

What are your tips for how to explain the news to students and young people?

If it’s news like this, that’s hard to maybe pin down what it exactly means (this was a memo, and does it have teeth? What does it mean?) I would look to them for the feeling of it. That’s what’s really important and a lot of what’s going on is just fear mongering, and a denial of trans existence. And that is something our young people will be able to speak to, to no end, and that they’re not strangers to — especially under this administration.

I would want to help ground things and offer some reassurance that a memo doesn’t have teeth and that we can look to our local New York City and state protections — that we’re lucky to live in a place that has such strong protections, especially for students.

What kinds of protections should New York City students expect to have in schools?

A student in New York City could expect to use the facilities that align with their identity, and could expect to possibly see all-gender facilities in their schools — as there are more and more of those being converted. They can expect to be able to file or register a complaint of discrimination against other students or even staff, and can expect to have an LGBT liaison within the Department of Education. They can expect to have their name and pronoun respected and utilized, and come up with a plan with a staff member around, if they’re transitioning socially or in any form at school, how they would like to be supported and how that looks in each unique situation.

It doesn’t always happen. But the fact that we do have it in policy means that there’s a means to pursuing it and that the institution is on the side of the trans or gender non-conforming student and would help to rectify any situation that’s feeling unsafe or unsupportive.

How can teachers and adults show support for their transgender students right now?

I don’t think it’s the kind of thing that you want to wait until somebody is in crisis. It shouldn’t be necessarily on any student to bring it up. By bringing it up ourselves, we’re modeling support. Even though this is a memo and we’re all waiting to see what they’re going to try to do with it, we know the intentions behind it…

I think we can speak directly to that and not make the debate about, ‘Is there or isn’t there a trans experience?’ That’s maybe one of the most powerful things. Yes, we exist. And if you’re an ally: ‘I’m a witness. You exist. You’re valid and as valid as anybody else.’

What would that validation look like in a school setting, say, if you’re a math teacher?

I think that making things visible is powerful. So if there’s a public bulletin board in a hallway and it says, ‘We stand with our trans staff and students,’ and then people have an opportunity to sign it.

I really think it can be an individualized response by a school depending on that school’s culture and if there is leadership by students, say, ‘We would like to be vocal and explicit in our support. You come up with the idea.’ Or, not to put it on them but say, ‘We’d love to be guided or get input from you on how to do that,’ so it is, wherever possible youth and trans-led.

Say, ‘What do you need and what can we provide?’

What should teachers and adults avoid saying or doing at a time like this? What are common mistakes to avoid?

I think a common, misguided mistake — that’s not necessarily hateful, but is really harmful nonetheless — is propping up a debate that’s going to hinge on ‘Do trans people exist?’ Or, ‘Defend or argue against sex being a binary, scientific, biological basis to view narrowly.’  

If a teacher wanted to engage with this but the assignment were more like, ‘What are your thoughts,’ there is so much education that needs to be done first — and that can put a person’s very identity and being up for debate in a classroom setting.

Another really bad thing would be just to ignore it because people are maybe scared of going there or don’t know what to do.

award-winning

Top principal’s ambitious goal: 100 percent at grade level — and her school is close

PHOTO: Aisha Thomas
Aisha Thomas, principal of Zach Elementary School in Fort Collins, won a national school leadership award.

In late September, Aisha Thomas, principal of Zach Elementary School in Fort Collins, got a phone call from a student’s mother. The woman said her daughter had been telling everyone that she wanted to grow up to be a principal just like Thomas.

It was particularly heart-warming because the girl was multiethnic, just like Thomas.

“I have arrived,” Thomas recalled thinking at the time.

Perhaps it was a harbinger of things to come. In early October, Zach Elementary was one of five Colorado schools recognized as a National Blue Ribbon School, and on Tuesday, the U.S. Department of Education announced that Thomas had won a prestigious leadership award.

Thomas is among 11 principals nationwide — all leaders of Blue Ribbon schools — selected for the Terrel H. Bell Award for Outstanding School Leadership.

“I’m floored,” she said. “I just come to work and I do what I do, and I love kids and I love people.”

But there’s more to it than that.

Thomas, who’s in her sixth year at Zach and her 17th in the Poudre School District, steers the school using five-year plans, frequent classroom coaching visits, and an emphasis on teacher collaboration.

It’s critical to know “where you want to take your school and your staff,” Thomas said. And then to be patient.

“It does take five years of churning through the day-to-day and showing up for people,” she said. “It takes time.”

The school’s latest five-year plan includes a goal that 100 percent of students will meet grade-level academic and behavior expectations. The school, situated on the southeast side of Fort Collins, uses a curriculum based on the Core Knowledge sequence.

Close to 90 percent of Zach students already meet academic standards, Thomas said, but it’s not enough. Even if there’s only one child missing the mark, what if that one kid is yours, she asked.

Thomas said school leaders have always tracked serious behavior problems, but this year will begin monitoring smaller classroom disruptions and distractions that affect student learning. The school also recently hired a coordinator who runs student groups on social-emotional learning and coaches teachers on managing student behavior.

Before she came to Zach, Thomas was a middle school counselor and assistant principal in the district. Since then, she’s discovered she loves the elementary age group.

“I love how creative the kids are and they’re just sponges for new information,” she said. “They don’t take themselves too seriously and they’ll tell you if you’re having a bad hair day.”