selection bias: a parable

To return to this blog’s favorite theme, how selection bias dictates educational outcomes, let me tell you a true story about a real public university in a large public system. (Not my school, for the record.) Let’s call it University X. University X, like most, has an Education department. Like at most universities, University X’s Education majors largely plan on becoming teachers in the public school system of that state. Unfortunately, like at many schools across the country, University X’s Education department has poor metrics – low graduation rate, discouraging post-graduation employment and income measures, poor performance on assessments of student learning relative to same-school peers. (The reasons for this dynamic are complex and outside of the scope of this post.)

The biggest problem, though, is this: far too many of University X’s graduates from the Education program go on to fail the test that the state mandates as part of teacher accreditation. This means that they are effectively barred from their chosen profession. Worse, going to another state to work as a public teacher is often not feasible, as accreditation standards will sometimes require them to take a year or more of classes in that state’s system post-graduation to become accredited. So you end up with a lot of students with degrees designed to get them into a profession they can’t get into, and eventually the powers that be looked askance.

So what did University X’s Education department do? Their move was ingenious, really: they required students applying to the major to take a practice version of the accreditation test, one built from using real questions on the real test and otherwise identical to the real thing. They then only accepted into the major those students who scored above benchmarks on the test. And the problem, in a few years, was solved! They saw their graduates succeed on the accreditation test at vastly higher rates. Unfortunately, they also saw their number of majors decline precipitously, which in turn put them in a tough spot with the administration of University X, who used declining enrollment as a reason to cut their funding.

Now here’s the question for all of you: was introducing the screening mechanism of the practice accreditation test a cynical ploy to artificially boost their metrics? Or by preventing students from entering a program designed to lead to a job they ultimately couldn’t get, were they doing the only moral thing?