How Charter Schools Cook the Books

For a variety of reasons, charter schools have long been the darlings of American news media’s discussions about education reform. For one thing, our media is disproportionately neoliberal and inclined to believe that markets make everything better. For another, our pundit class draws disproportionately from the elite classes, who tend to have attended expensive private schools and who have no particular sympathy – and often outright disdain – for public education. For another, the funding apparatus of our think tanks is heavily bent against public schools and towards charters, as the do-gooding rich types who fund such institutions are often market-focused and antagonistic to unionized public sector employees like public school teachers.

Whatever the reason, the general state of affairs in education reporting is near-total credulity towards charter schools and their advocates, with few in professional media digging in to charter school rhetoric to find the flaws. To understand these failures, it’s important to look at how charter schools achieve good numbers at the cost of fairness and transparency.

Manufacturing Selection Bias

Generating responsible arguments about education is difficult for a variety of reasons. Perhaps the biggest lies in selection bias. Selection bias refers to when inequalities in how samples are gathered – such as “public school students” vs “charter school students” – leads to incorrect assumptions about results. I have argued in the past that selection bias is in fact the single most important phenomenon in educational statistics.

A classic example in selection bias lies in the common assumption that private schools are superior to public. Many parents send their children to private schools without any rigorous investigation about whether those schools are superior to local public schools at all. After all, they might say, look at the star students the private schools graduate. But there is an obvious and immensely important factor missing when we attempt to naively compare outcomes across school types: the incoming student bodies are not remotely the same. Private schools almost universally have more affluent student bodies than traditional public schools, meaning many of the most disadvantaged students are systematically excluded. And many privates also employ entrance exams or grade requirements before enrollment, ensuring that their student bodies will be predisposed to succeed.

Charter advocates tend to speak as if charter schools have demographically and economically similar student bodies to public, and act as though we have true random placement into their schools. Some claim that lottery systems are sufficient to wash out differences in incoming student bodies. Random assignment is extremely important in educational statistics, as it is necessary to ensure that our comparisons are fair.

But we know that charter school student bodies are very often not equal. And we know that many charter schools go to immense lengths to make sure they aren’t. A 2013 Reuters investigation found myriad ways that charters go out of their way to exclude the most difficult to educate:

Students may be asked to submit a 15-page typed research paper, an original short story, or a handwritten essay on the historical figure they would most like to meet. There are interviews. Exams. And pages of questions for parents to answer, including: How do you intend to help this school if we admit your son or daughter?

These aren’t college applications. They’re applications for seats at charter schools.

Charters are public schools, funded by taxpayers and widely promoted as open to all. But Reuters has found that across the United States, charters aggressively screen student applicants, assessing their academic records, parental support, disciplinary history, motivation, special needs and even their citizenship, sometimes in violation of state and federal law.

Note that even requiring parents to opt their children in to lotteries is sufficient to contaminate randomization enough to make drawing responsible inferences impossible. And clearly these schools go far beyond that.

These behaviors are not only important in and of themselves, as indicators of how unscrupulous actors can bend the rules to make charters look better compared to public. They also demonstrate that even among charter officials themselves, there is a strong understanding of just how strong a role student selection plays in outcomes. Otherwise, why would they go to the trouble? For all of their talk about how charter practices are sufficient to help any child succeed, their own behavior demonstrates differently.

In fact, we have a raft of research showing that, when we employ genuinely random distribution, perceived differences in school quality makes no impact on student achievement.

You’d expect charter advocates to be particularly stringent about charters that engage in these practices; if they really believe that charters are better on the merits, they’d want to ensure fair comparisons. But in my experience, reformers are in fact incredibly credulous about even the rosiest numbers that arise from the charter world, almost never engaging in appropriate, productive skepticism.

Refusing to Backfill

An important type of selection bias is survivorship bias. With survivorship bias, we only observe a given characteristic in those examples that make it past some sort of selection procedure. If you ever hear a speech by any successful famous person, they are likely to deliver some sort of bromide about how they kept a positive attitude and never gave up. Which may be true – but there are also plenty of people who kept a positive attitude and never gave up and didn’t succeed, but crucially they never get the opportunity to make speeches about it so we don’t adjust our understanding accordingly. This is survivorship bias.

A common type of charter school chicanery involves the refusal to backfill and in so doing create a type of survivorship bias. “Backfill” refers to schools enrolling more students to fill spaces created through students dropping out, failing out, or being removed for disciplinary problems. Backfill – backfill through random selection, of course – is essential for making fair comparisons. After all, the students most likely to leave are often the ones living the most difficult, most transient lives, and thus those most likely to struggle academically. Refusing to backfill amounts to creaming the best students off the top after the fact.

Who’s guilty of refusing to backfill? Why, Success Academy Charters, the darlings of the charter school set! Aside from the brutal working conditions and army of short-term “tourist teachers” looking for a foothold in New York City, I suspect that this accounts for a large portion of the supposed advantage of Success Academy. If charter advocates are serious about actually wanting real student gains, why have they not led the charge against this kind of practice?

You can also just routinely suspend the most vulnerable students until they drop out or are forced out, which many charter schools already do.

When In Doubt, Cook the Books

Survivorship bias strikes again. The 2013 CREDO study was widely ballyhooed at the time as a vindication of charter schools, showing significant learning gains relative to public. And with the credibility and prestige of Stanford’s CREDO project behind it, the report made serious waves. Unfortunately, few people seemed to dig into the fine print. As a (pseudonymous) writer pointed out at EduShyster, the CREDO report admitted that 8 percent of the charter schools in the initial sample had closed. And which schools are most likely to close? The worst performers! Of course your numbers are going to look good when the worst 8% of the sample simple vanishes into thin air, a vanishing act generally impossible for public schools. Again: why would serious charter school advocates tolerate this kind of thing, if they are genuinely interested in helping children learn?

There are many, many other examples of charter advocates playing fast and loose with numbers in order to attack public schools. For example, when discussing the supposed New Orleans miracle in post-Katrina schooling, charter advocates are prone to trumpet the rise in the number of schools receiving a passing grade from the state since public schools were closed and replaced by charters. They typically neglect to mention that the cut score for passing was lowered in between the rating of the public schools and the rating of the charter schools.

Just Giving Everybody A’s

I like this one the best, because it is the most brazen. At San Diego Metropolitan Career and Technical School, every student is above average. The grades are sterling. The graduation rate is top notch.

The test scores, sadly, are quite bad. Because they seem to be giving out great grades to everybody regardless of performance. Hey, that’s one way to achieve – just lower standards. Reform types love to argue that market forces compel schools to promote student learning, but this is incorrect on its face. Market forces compel charter schools to please parents, which is not at all the same thing. And you can bet if it’s happening at one school, it’s happening at another. There are thousands of charter schools in the country, and yet their advocates constantly talk as though any given school performs identically to the attention-grabbing, high-resource, big-city idiosyncratic schools they love to tout.

If I am hard on the charter school crowd, it’s in part because they’ve spent the last several decades attacking teachers, hundreds of thousands of public servants who make middling wages performing an impossible job. But it’s also because issues like these are simply not discussed by advocates, who tend to adopt a defensive position and refuse to countenance any questioning of charter schools at all. I am currently working on a book about these topics; my day job is in academic assessment; I wrote a dissertation about standardized tests; and I’ve taught students from kindergarten to graduate school in a variety of contexts. I have never found serious attempts to grapple with the profound challenges to charter school numbers that I have laid out here. If charter advocates actually care about improving education, rather than simply winning, you’d think they’d leap at the challenge.