So this may get my Hater Card revoked, but I actually have found a lot to like about the early days of Vox. There have been some really good pieces on there– I liked the Passover explainer, and Yglesias’s distillation of Pikkety’s Capital in the 21st Century was well done, and an example of how that type of piece could function well. I haven’t read the book yet, but the post seemed to do a good job of relaying its central arguments and criticism against them, certainly informing people who were never going to read it, and I doubt there is anyone who was going to read the book who is going to skip it thanks to reading Yglesias’s work. Of course there are also pieces I find of dubious quality, but that’s the nature of the beast. Dylan Matthews failing to mention the loudness war in a piece on the benefits and drawbacks of vinyl is just bizarre to me; he calls dynamic range merely a drawback of records, when in many instances, the limited dynamic range on records is a benefit, since it keeps mixers and engineers from squashing the mix in the race for more and more volume. (Is it insulting to have pieces that ask “what is marijuana?” Sure. It’s a little insulting.) Like any publication, Vox will likely have some good, some bad, and it’s up to readers to be discriminating. So far, I’ve been pleasantly surprised.
That being said: this piece by Kelsey McKinney is really bad, and worse, it’s bad in exactly the way that people have feared about Vox. (People like me.) The legitimate fear, with a site like Vox, is that the urge towards simple explanations and data-driven writing will result in pat, seemingly authoritative answers to questions that are actually far more complex. And that’s exactly what McKinney is guilty of, cooking the books to make the world seem far more orderly and simple than it really is, and doing so in a way that seems transparently designed to reach a particular conclusion. Matt Novak of Paleofuture has an extremely effective, thorough, and utterly fair dissection of McKinney’s piece that really could stand as a model of how to write a rebuttal of this kind of piece. I highly encourage you to read it. There’s a lot of territory that Novak covers, but most damningly, he shows that McKinney’s dates for when prominent technologies were invented are almost entirely arbitrary. Arbitrary dating can sometimes be OK– many technologies, after all, don’t really have a specific invention date– but when your variable of interest is entirely dependent on length of time since invention date, that’s just an existential problem with your research.
McKinney’s graph alone should make us skeptical. Check this out.
That’s almost perfect! Which is enough to make the hairs on the back of my neck stand up. Real data does not look like that. Not DVs, anyway, not data the researcher doesn’t directly control. A stair-step graph of such uniform precision is only going to be found under rare research conditions, ones where we’re dealing with variables that are highly constrained or mechanistically related. And the time to adoption of prominent technologies simply is not that kind of data. This chart should raise red flags. The first question anybody should ask is: what about the technologies that aren’t on this chart? How does the selection of these technologies affect the model? I’m willing to bet we could find 7 technologies that did not fit this pattern, even if we have a more rigorous method for determining invention date.
Now just getting things wrong is okay. (I do it all the time!) After all: Novak wrote that response. As pessimistic as I can be about online journalism and commentary, this is how it’s supposed to work. When unsupportable claims or irresponsible analysis happens, hopefully, somebody else will do the work to refute it. There’s nothing disqualifying about McKinney’s piece. However, Vox apparently heard the criticism and reacted to it– but didn’t come clean. Here’s Novak’s update:
It looks like Vox has heavily edited their original story by creating a new lede, referencing research by the FCC and Pew Research, and correcting factual errors like their conflation of the invention of the internet with the invention of the web. They’ve also swapped out their original “Technology Adoption” graph with a new one and added an additional graph with data from the FCC. None of these edits are noted in the piece. But they appear to be sticking with their original argument anyway, so the critique below still stands.
That’s just bush league, in my opinion. It’s okay to make corrections– better than okay, actually, it’s necessary and responsible. But you have to come out and say you did that by writing a brief section (a paragraph will do) saying “we changed X, Y, and Z, and this is why.” If you don’t, it just looks dishonest, and it risks contributing to a sense of imperiousness that is not a good look. Worse, it gives you less incentive to not make the same mistake in the future, if you just disappear the old problems. There’s an “Updated by” line at the top, but no other information, and for me, that doesn’t do enough. Don’t compound the problems, guys. Just own up to them.
Update: The post in question now has a brief correction at the top, which is progress. As with Novak, I don’t think the correction addresses the fundamental problem.