no, the Brookings Institution study isn’t “garbage”

It’s just research like all research: conducted by humans, with limitations and caveats, but also deserving of a respectful, accurate reading.

There’s way too much student loan debt in this country. (I hold some of it myself.) I’ve written about it at length, and many times, over the course of years. I’ve written about ways to reduce the cost of college, about structural changes to the student loan system, about the desperate need for state governments to reinvest in higher education, and about a potential system of no-frills, low-or-no tuition federal universities that could graduate thousands without loan debt. It’s a big economic drag to have so many young people pouring money into student loans instead of into houses and cars, and it’s a moral travesty for so many people to suffer because they can’t afford to pay for the education they already received.


It’s also the case that people who write and argue about the student loan debt crisis frequently speak about the problem irresponsibly. People throw around fake figures all the time, propose all sorts of doomsday scenarios, and generally represent the problem in ways that are not supportable from data. I can’t tell you how often I’ve had people talk about “every graduate running around with six figure debt!,” when six figure debt is extremely rare. Or when people compare the size of the problem to that of the housing bubble, which is simply not correct. The housing bubble was orders of magnitude bigger than the student loan crisis. Pointing these things out is not a matter of disagreeing that there’s a problem, and it’s not a matter of being callous to those struggling under student loan debt. It’s a matter of being responsible in talking about a problem, under the theory that this is the only way to solve such problems.

So I’m seeing a lot of breathless sharing of this piece by Choire Sicha, attacking the methodology of a Brookings Institution report showing that student loan debt concerns are somewhat overblown. Like all studies, there are issues with the methodology of this one. But Sicha does not do a great job of critiquing this study. I don’t blame him– methodological critiques are really hard and take a lot of work. I’ve taken something like 9 graduate courses in research methodology and statistics, and I still find confronting methods and methodology an imposing task. But there’s lots that’s off, here. And Sicha isn’t mincing words: he’s saying the study is garbage, and people are listening to him. That’s not supportable, at least not from the evidence that he provides, and it’s unclear if he’s angry about the methods or about the conclusion.

Of all the households in that study, only about 1711 have “household heads” that are younger than 40. That’s what they’re extrapolating from. (And, intriguingly, a small number of those have a head of household younger than 18.) This is not a big sample!”

This is something I’ve written about before— people dramatically overestimate the sample size needed to make responsible statistical conclusions. A sample size of almost 2,000 isn’t just big, it’s enormous. The standard error of a sample of this size will be very low. Absent systematic sampling bias (as opposed to error), the odds of the underlying population being significantly different from a sample of this size is tiny. Saying that it’s not a big sample just displays ignorance about the standards applied in statistical research.

One effect of this age spread sample is that it includes college graduates from up to almost 20 years ago. This is literally not at all a study of college graduates of the last five years, or even ten years. We’re talking about people up to the age of 40, well into Gen X.

This is a strange statement. As Sicha acknowledges, the people sampled in the study are from ages 20-40. That certainly includes many people who have graduated in the last five to ten years. It also includes people up the the age of 40, but so what? Why is that disqualifying? The study admits that up front– it’s right there in the methods. And if you’re making the case that aggregating 20 year old debt holders with 40 year old debt holders necessarily invalidates your sample, you have to say why. The notion that there’s some dramatic difference between those who graduated within the last five years and those who graduated fifteen years ago is an empirical point that has to be proved. And it’s not as if tuitions have suddenly exploded; they’ve been rising, at a totally indefensible rate, for decades. (They have actually slowed their growth, but that’s cold comfort.) People who assert that the student loan debt issue is bigger rather than smaller have a burden of proof, as well.

Worse still:

And finally… this survey is, essentially, of rich people. No, literally!

We apply survey weights throughout the analysis so that the results are representative of the U.S.
population of households. The use of survey weights is particularly important in the SCF because
the sample design oversamples high-income households to properly measure the full distribution of
wealth and assets in the United States. This high-income sample makes up approximately 25 percent of
households in the SCF.

Literally what they are saying there is that the information on which they are basing a sweeping assessment of American student loan debt is based on a sample in which 25% of those surveyed were “high-income households.” This is insane.

This is the problem with reading methods sections without having an adequate basis in research methodology. As commenters other than me point out in the comments, this practice is absolutely common, and quite responsible. Given an adequate sample, it’s not at all hard to adjust this kind of imbalance quantitatively. Sicha calls a standard, simple practice “insane,” which demonstrates the degree to which he fails to moderate his claims. I’m afraid he doesn’t know what he’s talking about.

Sicha’s basic problem is a very common one, which it’s understandable but unfortunate: overestimating sampling standards necessary to make responsible generalizations with statistical data. I get it. I remember when my stats prof told me, last fall, that in certain situations an of 30 is adequate to make broad generalizations. I was flabbergasted. But when you pay attention to the processes of generating confidence intervals and standard errors, and when you understand the law of large numbers and the central limit theorem, it makes more sense. I understand why Sicha is put out by the head-of-household distinction, and I agree that you could view that as a limitation. But people asserting that this necessarily invalidates the data are going too far, and they themselves face a burden of proof in establishing that the heads of households are necessarily different enough from the broader population of debt holders to invalidate the sample.

If you’d like a sober, data-supported argument about the size of the debt crisis, I recommend this infographic by Derek Thompson of The Atlantic. It includes some data from Brookings– but also from the NCES, the NBER, the BLS, Harvard, UChicago…. Maybe they’re all in a conspiracy to underestimate this problem! But I doubt it. I instead think that the student loan crisis is indeed a crisis, a moral and practical problem of considerable size. But it’s not the size that most people think it is. And more, it doesn’t change this fact: that despite the endless concern trolling of almost our entire media, the constant tendency for the (college educated) professional writers in our  culture to say that “college isn’t worth it,” a college education remains a very good investment for the large majority of graduates. American college graduates are, by essentially any international standards and in comparison to Americans with only a high school degree, in a very economically secure class. There are tons of exceptions; that’s how averages and aggregates work. Those who are among the exceptions deserve not just sympathy but structural change to improve their lives. I’ll keep saying: there is no reason for the federal government not to engage in broad forgiveness of the vast amounts of loan debt it controls. In a bigger perspective, a universal basic income could solve an enormous number of our social problems. Most certainly including those of recent college graduates.

But I have feeling that this kind of support will not be sufficient. By criticizing the louder and less careful claims about this problem, I’m necessarily going to be placed in the ranks of the bad guys. But that’s OK. I just think being careful matters, and the truth matters, and that solving this kind of problem requires having higher evidentiary standards, rather than lower.


  1. I get it. I run into this problem all the time. As a progressive/socialist/whatever, I’m constantly very annoyed when so-called leftists trot out statistically invalid arguments to try and support a progressive agenda, when those arguments aren’t even needed to make the case.

    Then when I try to correct them, but still support the ultimate goal, I get accused of some nonsense, like being a “secret right-winger” or a “reactionary”, or whatever.

    I find that this is also a very common problem when it comes to women’s issues, as bad statistics are regularly employed there. I keep telling them that using bad statistics only weakens your case because they can be disproven, you are better off making a solid case on real numbers, but of course that never goes well.

    For example the whole 77 cents on the dollar issue, which get’s incorrectly trotted out by everyone from the President on down.

    The stat does not say that women are getting paid 77 cents on the dollar for doing the same work as men, but that’s how everyone who uses the state frames it, even though the research itself doesn’t say this. women are really getting underpaid for equal work more like 95 cents on the dollar. The 77 cents on the dollars comes from a wide combination of factors, mostly the different types of jobs that men and women tend to do and the number of hours worked by men and women.

    Yes, there is still pay discrimination, but pay discrimination isn’t leading to women getting paid 77 cents on the dollar, and that’s a provable fact.

    Another case is minimum wage issues.

    I’ve pointed out that even if you took all of McDonald’s profits and executive pay, you could only give all of it’s workers a raise of less than $1 an hour. This means that if you used all executive pay and all profits to give raises it still wouldn’t get McDonald’s workers to a living wage.

    That doesn’t mean that there isn’t a problem, it means that McDonald’s can’t simply decide to give it’s workers significant raises, it’s going to require economy wide regulation. What’s absurd is these “liberals” who act as if it’s simply a matter of greed and McDonald’s and similar such employers choosing not to pay people money that they easily could.

    In fact I’m making the bigger point that no, it isn’t a simple case of greed, it is a structural failure of capitalism. But many liberals actually don’t want to acknowledge that, they want to believe that without significant changes we can simply shame rich people into paying workers more. Absurd.

    But again, this all results from disregarding careful math and analysis and not understanding statistics.

    1. I don’t think your assertion about McDonald’s could possibly be true; I say this based on the well known comparison of minimum wage today vs 1968, when it was over $3 more per hour in constant dollars. (I presume McDonald’s was a solvent concern in 1968.)

      1. Yes, McDonald’s pays employees in foreign coutnries more too, and they used to pay more in inflation adjusted dollars.

        These things have nothing to do with doing the math. Market conditions are different now in america than under these different scenarios.

        McDonald’s also charges more in Europe and the inflation adjusted price of their food is lower today than it was in the past.

        In a market type economy a single entity can’t just simply decide to pay their workers 50% more than the competition. Doing so, without other major changes to the busienss model, would result in having to raise prices and thus lose market share, ultimately leading to losses.

        That’s why things like rasing wages levels has to be done with regulation.

        Now, let’s “simply” do the math. Note that this isn’t necesarily so simple for McDonalds because of how to count things. There is global revenue vs US revenue, global profits vs US profits, franshise employees vs direct employees, etc. But let look at some numbers. I’ll use global numbers because they are easier to obtain.

        I’ll take what I think is the simplest way to do this.

        Using this I’m using $4.9 billion as the profits for 2013. If you add in executive pay it doens’t really change things much, but let’s do that and round it up to $5 billion.

        So we’ll say that in 2013 McDonalds had $5 billion in revenues that went to profits and exectuve pay which could be used to pay workers.

        Who are the “workers”? They have 440,000 direct workers world wide, but 1.8 million including franchise workers. Since the profits include franchise payemnts, we’ll include the franchise workers.

        $5 billion divided by 1.8 million people = $2,777.78 per worker, meaning that if they took the total $5 billion and payed it out equally to all the workers, each worker would get $2,777.78 for the year.

        Assuming an average of 35 work hours per week, over 50 weeks a year, this gives us a raise of $1.59, so that is more than $1 per hour, but stil it isn’t much. Depending on who you listen to teir average wage about $7.80 an hour, so this would move the wage up to $9.39, which is far short of the $15 an hour that strikers are asking for. And this hourly raise amount changes depending on the number of work hours a year per employee that you use. 40 hours at 52 weeks (true full time) gives you $1.34 per hour as the raise.

        So you are right, it is more than $1, but not much more. And that assumes using 100% of profits and exectuvie pay to fund rasies.

        Of course in the real world higher raises can be made, but they would result in a combination of layoffs and price hikes. This is difficult to do in a market economy without losing market share, which is why rasing the minimum wage would actually work better for McDonald’s than simply giving employees raises on their own. We could raise the minimum wage to $12 an hour and that would hurt McDonalds less than simply rasing their average wage to $9.39 on thier own.

        1. Yeah, that is more than a dollar. Also, McDonalds’ 10K for 2013 lists their net income as $5.586 billion.

          This drives the hourly wage difference up to about $1.77, via your math.

          So already, even doing this “the simplest way,” your statement about the paltry dollar increase isn’t accurate. But I have a feeling if we zeroed in on US operations it would be even less accurate. I’ve heard this for awhile now, even that WalMart’s shitty pay has NOTHING to do with the Walton income, so I’m gonna have to get off my ass and do some real math soon. Suffice it to say this shit smells funny.

          1. I used what they actually returned as profits as opposed to simple net income because some net income can be used for improving the financial position of the company, i.e. putting it into savings, etc.

            Of the $5.5 in net income, they actually returned $4.9 to share holders.

            But this is really splitting hairs. Regardless of what number you use, you don’t get the wages anywhere near the $15/hr that McDonald’s workers are striking for. Under the most generous scenario you get to around $10/hr.

            But again, this assumes zero profits.

            That’s highly unrealistic, so under any REALISTIC scenario, where the company maintains much smaller profits, you end up with around a dollar or less that could be used to give hourly raises.

            If they were to cu their profits in half and use half of the full $5.5b in profits to give raises, that would amount to about 73 cents an hour, and that’s with cutting profits in half, which would be pretty dramatic for an established company like that.

            I’m not necessarily against that type of thing, but many of the people advocating for these raises are. They aren’t communists. unless you are advocating complete restructuring of the US economy or turning McDonald’s into a non-profit entity, or a worker takeover the company or something, then there isn’t any reasonable way, within the capitalist framework for McDonald’s to give out large raises to its workforce.

            At best, the company could stand to give out raises ranging from 50 cents to about $1 per hour to its workers. That’s not going to revolutionize anything or even get these people to a living wage.

            The only way to go beyond that level of raise is though regulation or some kind of more dramatic change, like a worker takeover the company turning McDonald’s into a co-op, allowing collusion among all fast food restaurants so they can raise prices in unison, etc., stuff like that.

            But many liberals have the idea that McDonald’s can simply afford to give their employees significant raises with no significant changes or costs, just by making a handful of rich people less rich. That just simply isn’t the case.

            I’m also for reduced executive pay and for that matter elimination of profits, especially in the “fast food” sector, but unless you are advocating a communist revolution (which I’d support), then don’t claim that McDonald’s can simply give it’s lowest paid workers raises raises from about $7.70 an hour up to $15 an hour.

            McDonald’s can’t pay its employees $15 an hour because of many structural economic problems, not simply because a handful of rich people are greedy. Yes, those rich people are greedy, and they don’t deserve their incomes IMO, but even if we took all of it, it wouldn’t get the workers to $15 an hour.

    2. Interesting you bring up the 77 cents. Aside from what that figure actually describes there is the issue that, to me, is even more important, that of the basic political dishonesty in how its being used. The 77 cents figure has some meaning, but the simple fact is that even if Dems got supermajorities in all branches of government any legislation that they would seriously pursue would only attack a few of those 23 cents. I’m all for societal initiatives that would close much more of that gap, but those initiatives would be attacking the deep structure of the culture, would be decades in realization, and national Dems would be followers rather than leaders on this.

    3. I appreciate your broader point about using statistical evidence responsibly but, like some other commentators here, I think you’re wrong about McDonald’s. First, you assume an average 35-hour work week. That’s probably significantly higher than the real figure, considering that fast food is notorious for relying on part-timers and keeping weekly hours below benefits and overtime thresholds. If so, your calculations significantly understate profits per worker-hour. Second, a substantial fraction of McDonald’s employees already do make more than the minimum wage. The company wouldn’t have to raise the wages of all its workers, so you don’t have to divide the profit pool by 1.8 million. Thus, the denominator of the fraction is smaller and the quotient larger than your calculations suggest. Third, you’ve omitted franchisees’ profits; some of the franchise holders are mom and pop operations but others are pretty big businesses in themselves. Adding them in and you increase the amount that could be divvied up among the low-wage workers too.
      Does this mean all fast food operators could afford to pay all their workers at least $15 an hour (which Seattle’s plan, the most generous around, won’t reach until c. 2020, by the way)? Probably not, but let’s not underestimate resources of American corporations or the potential of organizing to improve the lot of poor workers.

  2. That Sicha tried to reframe the narrative is fine, but when I saw he was trying to argue with the methodology I was very surprised. I know nothing about statistics, but the whole thing seemed to me no different than some lay conservative attacking a study on climate change without nearly the level of expertise necessary, the sort of thing that would be linked to in mockery by Gawker.

  3. I hope you got a decent paycheck from TPM, they’ve got lots of advertising including sponsored content.


    You’re right that Choire’s piece trafficked too much in the common sense view that small and biased samples cannot be corrected for, but despite being wrong in that belief, I think he has the better argument.

    – The focus on heads of households will overweight to those doing well.
    – In parallel, the inclusion only of those not in default will underweight those doing especially badly.
    – The parallelism of private colleges not seeing much tuition rise in recent years, but public colleges seeing significant rise, ignores that fact that public colleges educate 5 times as many students as private colleges.
    – Much of the “college premium” is a result of falling high school wages, not rising college wages.
    – The cost of college is not just rising; the value of our benchmark product, the Bachelor of Arts from a public university, has fallen by 15% in the last 15 years.
    – The comparison with 1992 is odd, since that year was in a viscous recession. There are in fact only *two* previous sample years where debt repayment was worse than this year. One of them was 1992. None were in this century.
    – The emphasis on the *size* of loan repayments does not note that the loans are often repayed over a longer period today, prolonging the effect of debt

    Much of what is in Leonhardt’s piece is a selective reading of the Brookings data, a point Choire, however ineptly, was correct to make. Those last two bullets above, though lead to the one outright falsehood in the piece:

    “…the burden for the people with the most debt is significantly lower today than two decades ago.”

    This is simply not true, and indeed, several grafs earlier, Leonhardt notes that debt has gone up. What he means is merely that annual service costs are below their historic highs (but above their historic lows The actual burden — what people owe overall — HAS gone up, and is simply being repayed over a much longer period, which stretches the opportunity costs further into the future.

    He also offers a misleading quote from one of the Brookings authors:

    “The evidence,” Ms. Akers says, “doesn’t support the notion that student loan debt is creating a drag on the economy.”

    What she means (and he only weakly clarified) is that the evidences also doesn’t support the notion that student debt isn’t creating a drag on the economy. He presents Akers as trafficking in reassurance rather than simply expressing uncertainty.

    In addition, both Leonhardt there and you here present the pieces featuring the outliers with $50K+ in debt as having the pernicious effect of convincing young people not to enroll in college. I am a fairly avid consumer of literature on the contemporary economics of US higher education, and I have never read a piece that seemed to me to suggest that, no matter how many broke barista/art historians it featured. Instead, the plain meaning of those pieces seems to be to suggest that young people should go to cheaper colleges. I wonder what makes you think that warnings about too-high debt convince people to forgo, rather than budget carefully for, a college education?

  5. The New Republic published a much better critique of this study:
    One of their main arguements was that the Brookings study focused on monthy loan burden. I also felt that the Brookings study was biaised because it did not sufficiently account for changes in repayment policy or repayment length over the study period.

    Chingos is also financed by the Lumina Foundation with ties to Sallie Mae…

  6. I left this reply at Talkinpointsmemo, where this essay was also posted:

    No, the study is still “garbage”, because it doesn’t really tell us anything useful about the national student loan situation, and it has the potential to mislead.

    deBoer focuses on what appears to be a misunderstanding of statistical sample sizes by Sicha. But regarding of whether Sicha could pass Stats 101, the Brookings study has at least two major flaws.

    One flaw is that it excludes households headed by someone over age 40. This means that any young person with large student loans who moved back in with their parents is probably excluded. (I say “probably”, because it seems that some young people who move back in with their parents may count as their own “household” for the purposes of this study, but it’s not clear what rules determine this and how carefully those rules are followed.) Since many of the people with the largest student loan debt burdens moved back in with their parents, this could seriously bias the study in a way that has nothing to do with sample size. (I’m one of them- I graduated with approximately $160,000 in student loan debt, and at age 29 I still live with my parents. I’m not sure whether or not I qualify as an independent “household”.)

    The other problem with the study is that it seems to focus on comparing monthly debt burdens of recent student loan debtors with those from the past. While this is a relevant measure if we’re only asking about whether any particular debtor can pay their bills or is about to default, it ignores much of what’s going on. Owing X dollars per month for 10 years is not the same as owing X dollars per month for 25 years. Particularly if after those 25 years the debt is “forgiven” by IBR but then the former debtor suddenly owes a giant tax bill to the IRS, assuming no bailout. The much larger student loan burdens of today- even if the monthly payments are similar- have serious personal, psychological, and macroeconomic effects. Debtors need to save more money and spend less than those from the past, at least if they choose to be responsible, simply because they owe more, regardless of whether their minimum payments are any higher. This reduces household formation and all the demand that goes with it.

    1. The preponderance of the best evidence, not merely from Brookings but from a variety of places, is that most of our student loan borrowers are handling their debt adequately, and are as a class in significantly better shape than those who never attended college at all. Why that claim is so emotional for some, I don’t know.

      1. I don’t believe that anyone is seriously disputing that college graduates, as a class, tend to be better off than those that never attended college. I also don’t believe that the intention of those who write and talk about the problem of student loan debt is to discourage people from going to college.

        The real questions are whether this debt is historically unusual, and whether something needs to be done about growing student loan debt. The Brookings study- and Leonhardt’s NY Times piece about it- tend to mislead, because they imply that nothing has changed, or even that student loan debtors tend to be better off than in the past. If student loan borrowers are only “handling their debt adequately” because they’re on IBR or they live with their parents into their thirties, or both, and this did not happen in the past, that signals that there has been a change. That change implies that perhaps something should be done to address this problem. (And no, that “something” is not discouraging higher education. Nobody is suggesting that.)

        You’ve written that you favor reducing student loan burdens, but by praising a misleading study that will have the effect of making any reduction in student loan debt less likely, you are, in your own little way, making it less likely that anything will be done to help current and future student debtors.

        1. I haven’t praised the study; I have, on the contrary, defended it from a misleading critique. I’ve also argued that the way in which student loan debt is generally discussed is consistently unhelpful, because it exaggerates the problem, exaggerates how many people have truly enormous loan debts, and risks obscuring the fact that those with large loan debts and degrees are generally in better economic shape than those without degrees at all. I’m not interested with what you think my unintended effects are. I’m in favor of broad loan forgiveness and have written thousands and thousands of words speaking out on how to reduce tuition costs and why. The truth has to be good enough.

          1. I totally agree with you. What I don’t understand are these people who don’t understand that credibility makes a stronger case than dramatic hype. It drives me crazy. I see ti all the time on both sides but I worry about it more on “the left”.

            In pretty much every case the dramatic, and incorrect, numbers aren’t needed to make the case being made, but people don’t seem to get that.


          2. There are truths, and half-truths. Half-truths can be just as misleading as lies. I agree that being careful matters, and that the truth matters, but at the least, you should be even-handed in dispensing the truth, because attacking only one side can be misleading.

            Sicha’s critique may have been flawed, but the conclusion it reaches- that the Brookings study is garbage- is accurate. Sicha’s article leads readers to believe that yes, the student loan problem really is that bad, and no, this study minimizing the problems is not any good. Those are accurate conclusions, despite any flaws in reaching those conclusions.

            I’d prefer it if a completely unimpeachable refutation of the Brookings student were published and went viral. (Paul Krugman… are your reading this?) The truth should be enough. But that isn’t what happened here. The choice is between Brookings/Leonhardt and Sicha. Sicha says that student loan debt is a serious problem, and Brookings/Leonhardt say it’s not. Both sides’ arguments had serious flaws. Yet you came down on the side of Brookings/Leonhardt by only criticizing Sicha.

            I’m not sure how, specifically, you believe the problem of student loan debt has been exaggerated. (I don’t have time to read your entire blog right now.) But even if this problem has been exaggerated, the proper response to that exaggeration is not then to understate or minimize the problem the way Brookings/Leonhardt have. It’s to accurately describe the problem. Unless you don’t want anything done to deal with the problem.

          3. “I don’t have time to read your entire blog right now” = I’m talking out of my ass and don’t have the evidence necessary to back up my assertions.

          4. You made an assertion when you wrote “I’ve also argued that the way in which student loan debt is generally discussed is consistently unhelpful, because it exaggerates the problem…” Saying “I’ve also argued”, implicitly referring to your entire blog history, is not support, because it’s unreasonable to expect someone else to be familiar with everything you have ever written, or even your entire blog. When I wrote “But even if this problem has been exaggerated…” that meant that even if you can back up your assertion, my conclusions are unchanged. My argument in that paragraph didn’t rely upon evidence; yours did.

  7. Choire’s critique is weak, but the Brookings paper is still pretty shitty. The major problem is that the 20 – 40 age cohort is way too wide which means trends in monthly payment data reflect the demographic influence of paying down the principle.

    If, as is likely, the ratio of 20 year olds with debt in the 90s was higher than it currently is, then you would see the average monthly payment drop. This is because as people pay off their loan principle, their interest payments are lower. So if you have a sample with more older people you are going to have a lower average payments rate just because of demographics. But this says nothing about the sustainability of the debt, or the burden on borrowers. It just shows that in the past there were more borrowers.

    Combine this with longer payment periods, (which also mean total payment is likely higher) and secularly lower interest rates and it is not surprising that the average payment is lower. But this is very relevant for policy.

Leave a Comment

Your email address will not be published. Required fields are marked *