Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

Payment dispute

Disputes with Tennessee testmakers aren’t new. Here’s an update on the state’s lawsuit with Measurement Inc.

PHOTO: Dylan Peers McCoy

The testing company fired by Tennessee’s education department two years ago may have to wait until 2019 to settle the case, according to documents recently obtained by Chalkbeat.

As the future of the state’s current testing company, Questar, remains uncertain after a series of testing snafus this year, Tennessee continues to build a case against the first company it hired to usher in online testing three years ago.

The $25.3 million lawsuit, filed by Measurement Inc. of North Carolina, says the state owes about a quarter of the company’s five-year, $108 million contract, which Tennessee officials canceled after technical problems roiled the test’s 2016 rollout. So far, the state has paid the company $545,000.

The 2016 test was meant to showcase TNReady, the state’s new, rigorous, online testing program. But the online exam crashed, and the state abandoned it, asking Measurement Inc. to pivot to paper tests. After numerous delays in delivering the paper tests, Education Commissioner Candice McQueen fired the company.

Measurement Inc. filed a lawsuit last June, and the state Department of Education responded in January with a counterclaim saying the company did not fulfill its duties. Now, the state and the company have through spring 2019 to build their cases and call witnesses. (You can view Measurement Inc.’s claims, and the state’s counterclaim below).

The company argues that the state’s decision to cancel online testing and switch to paper was a series of “unrealistic, arbitrary, and changing demands,” and therefore, the state shares blame for the canceled test.

But the state department countered in its January response that Measurement Inc. breached its contract and didn’t communicate truthfully about the status of the online exam.

After Measurement Inc., Tennessee entered into a two-year contract with Minnesota-based Questar to revive the TNReady online exam. In 2017, the state opted to only use paper exams, and testing went smoothly for the most part, outside of delays in returning test results.

But things didn’t go well this spring, when Tennessee tried to return to online testing under Questar. The reasons for the complications are numerous — but different from issues that ruined the online test’s 2016 debut.

Although Tennessee completed its online testing this spring,  it was beset with technological glitches, a reported cyber attack on the testing system, and poor internet connectivity. Many districts are not planning to use the scores in student grading, and teachers can opt out of using the scores in their evaluations.

The state is negotiating with Questar about its $30 million-a-year contract and also is asking Questar’s parent company, Educational Testing Services, to take on the design work of TNReady. McQueen did not offer specifics about either, but any changes must be approved by the legislature’s fiscal review committee.

Questar’s two-year contract ends Nov. 30, and the state either will stick with the company or find its third testing vendor in four years.

You can view Measurement Inc.’s claims, and the state’s counterclaim, in full below:

Measurement Inc.’s June 2017 claim:

The Department of Education’s January response:

Measurement Inc.’s February response:

Future of Schools

Short on students, 3 Indianapolis charter schools are closing. But 6 more will open in the fall

PHOTO: Dylan Peers McCoy

Three Indianapolis charter schools facing financial struggles will close at the end of this school year, underscoring just how difficult it can be for charter schools to create sustainable operations.

As another sign of charter schools’ cash crunch, particularly in the city’s increasingly competitive school choice market, longtime Indianapolis charter network Tindley will merge its all-boys and all-girls middle schools into a single coeducational location.

Still, even as some schools close and consolidate, six more charter schools are poised to open in Indianapolis for the upcoming 2018-19 school year — including two that will be tasked with “restarting” schools within Indianapolis Public Schools as innovation schools.

In many parts of the city, the proliferation of charter schools is pushing the school choice conversation beyond simply providing more options to focusing on the quality of those options.

According to state data, nearly 17,000 students who live in Marion County — almost 11 percent — attend charter schools, which are publicly funded schools that are privately run. Across the state, charter schools are the fastest-growing school option, though they mostly serve urban areas.

Read more: How are Indiana charter schools doing? 9 things to know from the state’s first study

CLOSING: CARPE DIEM NORTHWEST

Among those shuttering schools are two that focused on blended learning. Carpe Diem Northwest, the national chain’s only remaining campus in the city, will shut its doors, the state charter school board said.

According to the Indiana Charter School Board, Carpe Diem’s board voted to close the school in March. The school’s principal and board president did not respond to requests for comment from Chalkbeat.

According to state data, 218 students were enrolled at Carpe Diem Northwest this year in grades 6-12 — an uptick likely due to the chain merging its former three campuses into one location. Consolidation efforts started in 2016, when Carpe Diem closed its Shadeland campus amid low enrollment. The chain’s Meridian Street campus lost its charter last year after struggling with academic problems, low enrollment, and financial instability.

CLOSING: NEXUS ACADEMY

Nexus Academy, which shared a building with the Glendale library branch, will also close this summer after a drawn-out attempt to stay open as curriculum providers pulled out of the school.

The school used blended learning to serve students who sought an alternative school environment, such as students with disabilities, students who didn’t succeed in conventional classroom settings, or students pursuing professional athletics or acting.

Nexus Academy had initially announced it would be closing at the end of the last school year, said board president Debra Morgan, when online K-12 management company Connections decided to close all of its Nexus Academy locations across three states.

But local leaders in Indianapolis wanted to keep the school open, so they began searching for a new management company. They were able to arrange a trilateral agreement with Connections and a new provider, California-based iLEAD Schools.

Still, Nexus Academy principal Jamie Brady said, “It was at the 13th hour, and it was too little, too late.”

Students had found other schools, and teachers had found other jobs. Marketing efforts to increase enrollment fell short, Brady said, and the school re-opened late in the year with too few students.

Earlier this spring, state charter officials deferred renewing Nexus Academy’s expiring charter. But before the school could return to make its case again, Brady and Morgan said iLEAD Schools also decided it could not help Nexus Academy, leading the school of about 25 students to close.

CLOSING: INDIANA COLLEGE PREP

A third school, the highly troubled Indiana College Preparatory School, will close after the mayor’s office ordered the school to shut down. The company running the school had stopped communicating with the mayor’s office, and the entire school board had resigned.

Read more: In debt, with too many unlicensed teachers, Indiana College Preparatory School loses charter

CLOSING: HOOSIER ACADEMY VIRTUAL

Hoosier Academy Virtual Charter School, a statewide full-time virtual charter school that enrolls students from Indianapolis, is also closing after months of scrutiny from the state, dropping enrollment, and poor academic performance.

Read more: After years of failing grades, Hoosier Academy Virtual will close in June

CONSOLIDATING: TINDLEY MIDDLE SCHOOLS

Among Tindley’s local chain of six schools, its two middle schools will drop their single-gender programming to merge into one co-educational school.

Tindley CEO Kelli Marshall said the decision was in part financial, driven by declining enrollment. As charter school competition has increased, she said it was harder to attract students to the all-girls Tindley Collegiate Academy and all-boys Tindley Preparatory Academy.

Families also said the bridge into high school was more difficult for students who went to single-gender middle schools, Marshall said.

The merged middle schools will operate under the Tindley Collegiate name and use Tindley Prep’s building, next door to Tindley Renaissance, its feeder elementary school.

OPENING: ALLEGIANT PREP AND VANGUARD

A pair of charter schools will open on Indianapolis’ westside to focus on students in the Haughville area, each school founded by Building Excellent Schools fellows.

Allegiant Preparatory Academy will grow into a K-8 college preparatory school with a particular focus on literacy, led by Indianapolis native Rick Anderson. The first week of school will be devoted to teaching students about Allegiant’s culture and core values. Students will begin making college visits in kindergarten and first grade, and the school will also work with families on how to support students on their paths to college.

Allegiant is built upon the motto that “it takes a village” to ensure students’ successes.

“We’re all saying that we have our hands on the shoulder of this child, and we are going to ensure that they’re safe, that they’re learning, and that they’re also growing as leaders,” Anderson said.

At Vanguard Collegiate of Indianapolis, school founder Rob Marshall — also an Indianapolis native — wants to incorporate the school with the Haughville neighborhood, with students completing service learning and projects based on the needs of the community.

The school, located in the former IPS School 75 building, is specifically seeking to help low-income students who live nearby, and Marshall said his leadership team is intentionally composed of people coming from backgrounds similar to their students.

“We know these students can achieve,” he said. “They just need the right adult that understands the circumstances and is willing to build the relationships.”

Vanguard will be “unapologetically” college prep-focused, Marshall added, with mandatory tutoring at the end of school that helps students with whatever they may have struggled with in that day’s lessons.

Both schools say they expect to ramp up enrollment efforts this summer.

OPENING: PILOTED SCHOOLS

PilotED started as after-school programs in Chicago, and now it’s turning into a new school in Fountain Square, in the former home of Indiana Math and Science Academy South and IPS School 64.

PilotED is focused on social identity, asking both teachers and students to examine difficult questions about power and privilege. The school incorporates social justice and racial equity into academics.

School co-founder and The Mind Trust fellow Jacob Allen said he hopes the model does more than prepare students academically for college — he wants it to position students to persist and graduate, particularly students of color, students from low-income families, and first-generation college students.

Allen also said the school wants to pay attention to teacher development and perks, including providing a mental health stipend, a staff gym, and co-working space.

OPENING: PARAMOUNT’S SECOND CAMPUS

Paramount School of Excellence is expanding to a second location about two miles away from its flagship eastside campus. Paramount Community Heights will serve students in grades K-4.

TURNAROUND: MATCHBOOK LEARNING

Matchbook Learning, a turnaround operator with a troubled history, will restart IPS School 63 on the westside as an innovation school. The charter school uses software to help teachers track students’ progress, a model that Matchbook founder and The Mind Trust fellow Sajan George hopes will lead to dramatic test score gains.

Read more: Ousted from Detroit and Newark, turnaround operator Matchbook could get a fresh start in Indianapolis

TURNAROUND: URBAN ACT ACADEMY

URBAN ACT, led by The Mind Trust fellow Nigena Livingston, will restart IPS School 14, a downtown school that has long served many students who are homeless. She plans to use “place-based learning,” a philosophy that incorporates the surrounding community into the projects students pursue at school.

Read more: Homeless students found stability at School 14. Now the school faces a big shake-up