Campbell’s Law in action

My friend Isaac Butler of Parabasis clued me in to a nice little illustration of Campbell’s Law, which states “The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” Try doing a Google search for any prominent figure, organization, or event in US history, and start typing in “APUSH” after. For most things you’ll see the term pop up along with a bunch of related terms like quiz, timeline, notes, questions, etc. APUSH stands for “AP US History.” The Google suggestions come from students trying to get information for the test.

Now you can certainly make the case that there’s nothing wrong with using the internet to fortify your learning and your research. I’m guessing that a lot of kids are doing this to skip out on reading their textbooks, which their teachers probably wouldn’t be thrilled about, but I’m somewhat sympathetic to that viewpoint. But the bigger issue is this: I doubt there’s very much difference in the thinking of students who go out of their way to legitimately get more information and students who go out of their way to cheat the system. In fact I think experience tells most teachers that, when it comes to tests like the AP US History exam, the distinction is illusory. This students are interested in achieving a particular score on a test that they see as having immediate incentives for them. The challenge of learning US history is merely an impediment to that goal, so they will subvert that challenge if they can and acquiesce to that challenge if they must. By instrumentalizing knowledge of our country’s history, we have placed the priority of scoring highly on a test that contributes to a student’s college applications above the priority of actually understanding history in terms of content and as a methodology.

All of this is eminently predictable, and was in fact predicted by critics. Perhaps the elevation of the test above the material will have the beneficial consequence of more students learning about history, I don’t know. I do know that the inevitable consequence of rampant standardized testing is generations of students whose interest in the tests far exceeds their interest in the material. That’s a rational response to the world they’re living in. The question is whether the policy makers will ask if it’s rational on their end as well.

14 Comments

  1. How would this same criticism apply to the college survey course APUSH is meant to be a high school substitute for? A lot of first and second year course examinations have little at all to do with (in most cases) the stated objectives of the courses they’re for.

  2. I learned a lot more from my AP classes in 12th grade than from most of my (reasonably well-taught) college classes.

    I learned more prepping independently for the MCAT (naw, I never went to med school, but I did well!) than I did from any of my basic science courses.

    The level of articulation of an assessment and its level of congruence to the actually taught curriculum makes an enormous difference in how much people learn.

  3. If a municipal airport put up a metal wind-sock, so that the wind would always be indicated to be in the right direction for planes pointed straight down the runway, they might have a few crashes.

    In American education there are two, and only two, studies of any relevance to policy, for hundreds of millions of dollars spent over possibly eighty years now of serious research.

    There are odds and ends of interesting experiments here and there, e.g. my friend Ben Wood’s 1937 PhD thesis: his friend Tom Watson gave him 4,000 IBM electric typewriters, and he gave them to kids all over Pennsylvania. Lo, the kids wrote reams and reams. “The child’s bandwidth is too wide for the narrow tip of a pencil,” Ben announced. And, I think interestingly, “Kids recognise print, and see their own handwriting as failed print. The typewriter gives them what they think of as the real thing.” Sweet. But only one thesis.

    The two good studies are the Coleman Report, in several volumes, and a private study done by the excellent Tavoulouras family when they were planning on giving a ton of money to the Harlem Academies. The Coleman Report demonstrates that the only policy-relevant thing known to be good for student success, normalized for IQ and parental income, is teachers’ vocabulary, as measured by common tests like the Readers’ Digest “It Pays to Increase Your Word Power.” (Of course the RD title is a lie: what is known is that people with good vocabularies make more money, not that people with lousy incomes and poor vocabularies can improve either or both by studying words.)

    The Tavoulouras’s study showed something similar: the teachers of successful students were significantly better than other teachers at putting together statements of what they were planning to teach. They didn’t necessarily do so, but they showed they could when tested. (When I was elected to my daughters’ School Board in Tokyo, and then elected curricurum tanto-o, roughly “classroom inspector,” within that board, I found this very widely true of the Japanese teachers I talked to: they simply had a precise stance about what they were doing, what the classroom action was going to be.)

    Now here comes the metal wind-sock. When these studies started getting around, in the late 1960s, bureaucrats everywhere ignored James Coleman, because he had shown so many Nice Things to be useless. The Tavalouras study, however, was wonderful: it “proved” that teachers should make lesson plans, and everything would be fine and dandy. America promptly suffered an epidemic of compulsory lesson plan delivery.

    Well big fat giggle. Yes, there is an industry in providing lesson plans, and school supervisors issue plans, and teachers crib plans, — but even if there weren’t, this is still fixing the wind-sock to guide the wind, gluing the compass to steer the ship.

    Campbell: Q.E.D.

    -dlj.

  4. I’m a bit confused about why this is a criticism of standardized tests specifically. Don’t students do this sort of thing for regular old in-class exams? Don’t students read SparkNotes (do those still exist? am I dating myself?) so they can BS in case they’re called on in lit discussions? Standardized or otherwise, I thought in the vast majority of cases most kids, even very bright ones, were interested in the tests and not the material.

  5. The more concise version of the same law (and I didn’t make up this phrasing, but I wish to hell I could remember the source) is: “an incentive to create an effect is also, and always, an incentive to create the appearance of an effect.”

  6. As they say in the military, “You get what you inspect, not what you expect.”

    But then this augurs more for using markets instead of government for delivery, since the subjective preferences of the consumer may map better to the social utility function than the measure that you’re using.

    1. Ethan,

      Social utility?

      How does individuals’ massive popularity of heroin, amphetamines, and codeine — as reflected in their high free market prices — map to the general good?

      -dlj.

      1. Are their high free-market prices unaffected by restrictions on supply?

        Surely you must admit that if you can’t develop good measures for the state to use in deciding how to provide a good, then that’s an argument against state provision, and for market provision. Markets do turn subjective preferences into objective measures, that’s what they’re good at.

        1. Keith,

          You attempt to change the subject, and maunder on repeating your original claims, rather than answering my questions.

          Yes, you are quite right that markets produce prices, which may be said, somewhat tautologously, to be measures of marginal utility to people with tradable assets — rather less than what you claim.

          We have no knowledge of what prices would be in the absence of police restrictions. With narcotics legal you would have both increased demand and commercial marketing, with who knows what effects on price.

          Of course I used this as only one example of the problem: the marginal wants of individuals need not have much to do with social utility.

          One very major effect: in “free” markets people have a strong interest in charging whatever they can. Where something has a negative price in the market, i.e. when you have to pay to get rid of it, as with garbage or CO2 and smoke, people give it away free — overcharging wildly.

          Private interest and social utility meet: 180 degrees in opposition.

          -dlj.

          1. You seem to be the one not “getting it.” Who cares if market prices are perfect? If you can’t decide what measure to use to judge performance for your government-based approach, then you’re still hosed, and yes, market provision could then be better. The parents may indeed know better, and may send better market signals, than a bureaucracy employing your gameable measure.

            This is an irrefutable logical consequence of Fredrik’s point, regardless of whether it hurts your feelings. Please deal with it honestly, rather than arguing about some platonic (im)perfection of market prices. Otherwise, I will have to conclude that are just too emotionally attached to your ideology, and you cannot have a rational conversation.

  7. Poor Keith,

    Nothing about prices being perfect or imperfect. No “government based approach,” your phrase. I simply ask you (now for the third time) to justify that private interests and, your phrase, social utility are aligned. I have suggested, I think convincingly, that sometimes they may be and often they aren’t — with specific examples you have avoided replying to.

    I have already agreed with you that markets produce prices, and prices reflect the interests of those who can buy. There’s no need for you to say it again: I never questioned it, only toned down your rather too universal claims.

    But I have asked you a simple question, and you have once again evaded it.

    -dlj.

    1. Here is my original statement:

      “But then this augurs more for using markets instead of government for delivery, since the subjective preferences of the consumer may map better to the social utility function than the measure that you’re using.”

      That’s what I said in the original post. Ever since then, you’ve been arguing with your own imaginary friend in your own head, about whether prices PERFECTLY reflect social utility, which was never a claim I made. You’re not exactly dazzling anybody, here.

      Nothing I have claimed requires perfect mapping between prices and social utility. It’s merely this, which remains completely true, and which you keep dodging: If the government has poor measures of performance, that augurs for market provision, because the prices (whether perfect representations of social utility or not) may map better to social utility, relative to government provision lacking decent, nongamable, performance measures.

      Now, please tell me what exactly you disagree with in the final paragraph.

      Really, I think you’re just not following the logic, and you’re making up a simpler idea in your own mind to argue against, to preserve your social status in your own mind.

  8. And my point, for the fourth and last time is personal preferences, whether revealed by prices or otherwise, are often utterly contrary to social utility.

    The most important example of this, it seems to me, is things that have negative prices on the free market, e.g. garbage, air pollution, CO2 and so forth.

    And you still haven’t replied to me, just, now for the fourth time, repeated your apparently most loved source of wisdom, yourself.

    -dlj.

Leave a Comment

Your email address will not be published. Required fields are marked *