Validity, as you likely know, typically refers to how well a given experimental or test instrument actually measures the given construct it is meant to measure. If an experiment is meant to test vocabulary knowledge, for example, then its validity is dependent on how well it actually assesses and reports the vocabulary knowledge of its participants. In the social sciences, there is often a great deal of controversy over what constitutes validity in experiments.
One popular concept of validity is ecological validity. This refers to the “real world” nature of tests or research instruments. How well do they approximate the actual conditions under which the assessed construct is encountered in real life? Like many issues within methodology and epistemology, both the term and the concept are debated, not only as a matter of what constitutes ecological validity but also whether we should make ecological validity a criterion of research success at all. Many will tell you that what really matters is just external validity– that is, does the construct assessed in the research port into the context in which we actually care about, regardless of whether the research conditions match real world conditions? So in the vocabulary example, ecological validity would be concerned with how well our research conditions matched real world, natural language production of vocabulary. External validity, in contrast, would just concern whether we could accurately generalize from the findings into a real world context.
So consider the current debates about testing in K-12 education. Many parents, teachers, and concerned parties worry that the rise of endless testing fails because it lacks ecological validity– that is, that so much testing removes learning from the real world contexts that we actually care about in the first place. Some in the world of standardized testing have replied that, in fact, these tests are authentic– they are authentic tests. That is, they are ecologically valid in that they approximate the conditions under which students will surely be tested in their future lives. They argue, not entirely unreasonably, that testing will be a part of a student’s educational life, whether those tests are standardized and enforced from above or not. So a test might not authentically test a given construct in real world application, but it would authentically test conditions that are a given in education.
That’s true, I suppose, but the problem with it seems obvious: once you’ve made that leap, you can justify literally any new educational assessment under the theory that the students will encounter it in the future. It’s tautological: we have to test students in order to prepare them for the tests they’ll have to take.
That thinking, to me, is indicative of a broader problem in the whole school reform debate: the way the snake starts to eat the tail, and we lose sight of just what we cared about in the first place