Evaluating the Relationship Between VST Score and Lexical Diversity in Written Production Among Asian Learners of English

Abstract: Researchers in language learning divide a given learner’s passive vocabulary, or words that can be defined on request, from his or her active vocabulary, or words that are utilized in the production of natural language. Active vocabulary is of greater value for communicative competence and of more interest to language testers, but is harder to adequately assess than passive vocabulary. In the present study, the timed essays of writers from China, Korea, and Japan were assessed for their active vocabulary, operationalized as lexical diversity, using three popular metrics for such assessment. These measures were correlated with each writer’s score on a test of passive vocabulary, the VST, or vocabulary size test. Regression analysis was conducted for all three metrics as well. Across all lexical diversity metrics and language backgrounds, the correlation with VST score was low, suggesting that there is little direct connection between a writer’s passive vocabulary and their active vocabulary as expressed in their writing. This suggests that such tests are of little use for predicting practical vocabulary use in writing and should not be utilized for that purpose.

Read the whole thing here.

standardized tests are tools of inequality, not equality

Consider this fairly recent NYT piece, titled “A Simple Way to Help Low-Income Students: Make Everyone Take SAT or ACT.” The essay talks about research that shows that making these tests mandatory in high schools raises the participation rate (duh) and in so doing identifies high-achieving students who ordinarily would not have taken the test. See, typically those who are screened out of taking a college entrance exam through self-selection are those who are less college ready and perform less well. But this is far from universal, and there are many potential high-scoring testers who are screened out through fees, lack of parental guidance, or a lack of information about when and how to sign up. The research discussed showed that low-income but high performing students are less likely to take these tests than high-income, high-performing students, and that making the tests mandatory will thus lower the relative disadvantage of those students. Mandating the test is thus a tool for equality – it increases the opportunity for students who are typically systematically excluded from college.

Let’s think about things for a second. First, and to be clear, the research does not show that low-income students are more likely to perform well. The opposite is the case:

(Data’s a bit old, but this is a durable outcome.) So having more low-income students taking the SAT will likely mean finding that many low-income students are in fact not prepared for college, to go along with finding those high-performing kids who we wouldn’t otherwise. Still, obviously I think it’s a noble and necessary goal to help identify talented students from poor families. The point is that it’s odd to think of this as a project for increasing equality as such. We’re simply looking for more “diamonds in the rough,” and hopefully helping to pull them out from their peers – who are thus left even further behind.

Here’s a point to stress: the very purpose of educational testing is to identify inequality. That is, we develop and administer tests precisely to better understand how students are not the same. In fact, the most precisely that tests are, the more unequal we understand the tested population to be. A 10 question test likely has less discriminatory power than a 100 question test, and thus the 100 question test is more likely to differentiate between closely-grouped students – that is, to identify how they are unequal. Progress in educational testing stems from designing instruments that are more sensitive to underlying inequality. That’s the very name of the game.

As I’ve said before, we talk about education as fulfilling two functions that are not just in tension with each other but directly contradictory: education is discussed as a tool for creating greater socioeconomic equality, and as a system for identifying excellence and rewarding it with status and economic opportunity. The problems here should be obvious.There’s a much larger conversation about summative equality and equality of opportunity here, which is too directly political to get into in this space. (I will say that I think equality of opportunity is not really a coherent idea when you pull at it a bit.)

But from the standpoint of educational policy, it’s not clear to me that we really know what we want to be doing. Some people tell me that our goal should just be to move everyone up in terms of absolute achievement, raising averages without necessarily changing relative performance. That might have lots of good effects, but it’s by definition not something that could help with inequality, as what’s rewarded by the labor market is relative educational achievement, not absolute. If everyone who ever went to an Ivy League school was sent to the moon, they’d simply look for the next rung down and hire accordingly. If the purpose is instead to shrink the variance, to narrow the range between the top and bottom of the achievement scale, we’d want to talk about limiting resources to the top-performing kids, and we’d still be looking for differences in what individuals can do. And we have no good reason to think that we can achieve either at scale, because while some interventions have helped different groups at different times, the general bell-shaped distribution of overall achievement on any identifiable quantitative metric of academic success has been persistent and unchanging over time.

As long as we use education as a system for sorting students into different tranches of ability, and as long as that sorting system is a key mechanism for placing people into different levels of income and joblessness, we can’t conceive of our system as being an engine of socioeconomic equality. We might sometimes use testing to identify areas where more resources are needed and distribute them accordingly. But 15 years since No Child Left Behind and the testing-heavy era it augured, we have seen almost nothing in the way of convincing proof that testing is a reliable tool for raising standards and increasing either equality or opportunity. Tests are powerful things, and modern test-development can produce exams of extraordinary precision. But they can’t be useful until we have a clear and coherent vision of what we’re testing for. To get that understanding, we have to begin to pull apart our basic assumptions about education and our economy, to ask ourselves if the system can do what we imagine it can do.

transitioning out of freelancing

Hey gang, I’m back from vacation and excited to get back to work here on this project. Lots of cool stuff in the works for here, including hopefully some audio and video content soonish. (Not a podcast, don’t worry. The world has enough podcasts already.)

I wanted to take a moment and explain why I’m going to be moving away from freelance writing. I’ve had a pretty good run lately; I was in the print Los Angeles Times a couple weeks back and the print Washington Post last week. (You can always check out my published writing by clicking the My Work tab above.) I know we’re all supposed to be too cool to care about print these days but, well, I do care. And I have a couple of heavily-researched pieces coming out in some longer form journals in the next several months, and it looks like I might have a regular column-type thing to indulge my political side. But beyond that, I’m not really interested in freelancing anymore. The truth is that I just find the process so aggravating and dispiriting at this point, and the money so bad, that it’s simply not worth it to me.

I just find, at this point, that the process of pitching, composing, shepherding through edits, promoting, and trying to get paid sucks the life out of me. The commercial interests of publications require editors to ask for things that are tied to the news cycle in the most facile way imaginable. I get it, and I don’t blame them personally. But I’m opting out. And it’s increasingly hard for me to explain to editors what I want a piece to do and say without writing the piece. I’m just really not interested in the “beats” of a piece of nonfiction anymore; the argument, in the sense that people traditionally mean, is just about the least interesting aspect of nonfiction writing. So when asked to reduce my own prospective writing to a series of explicit moves, I’m forced to fixate on the parts that I find least interesting or valuable. What I want is to write in a way that is free of precisely the kind of paint-by-numbers literalism that editors require. Again, not a knock on them. It’s just not in my interests anymore.

Meanwhile, the money generally sucks. I am very grateful for the LAT publishing me in their print edition, for example, and I knew what the rate was going in. But writing and editing a thousand-plus word piece for one of the biggest newspapers in the country got me $200. So many younger writers I know think that the higher profile, more established places are where the money is, but often that’s not true. Not anymore. And if I don’t enjoy it and the money’s not good, what’s the point?

I also don’t have a lot of hills to climb anymore in terms of places I want to be published. At this point even my (many) dogged critics can’t really claim that I can’t get published in major magazines or newspapers. And it’s not like they changed their tune once I did, anyway. I started writing for big pubs in part as a way to prove to my detractors that, contrary to what they said, I could get published in respectable places. When I did, they didn’t retract their old insults. They just switched to new ones. So there’s little appeal there, at this point.

And, finally, I’m just exhausted by people not reading. I’m just exhausted. The WaPo piece is an expression of 100% straightforward left-wing values; it’s a critique of corporations and an endorsement of the idea that only the left can guarantee true freedom. I do write my fair share of left-on-left critiques, but this piece really is not that. It’s simply an articulation of basic left principles in a frame designed to make them more appealing to the unconvinced. But the piece has predictably attracted criticism from the left, people insisting that I’m a reactionary even though I’m making a standard left critique of corporate power. Some have claimed that it’s a defense of the Google memo writer, when in fact I explicitly justify Google’s actions in the very first paragraph. The great bulk of the piece was written six weeks ago, before that memo existed, and that situation is tangential to my larger point. Meanwhile, others saw the headline and immediately assumed that this was a defense of the Charlottesville protesters – which would have been remarkable, given that the piece had come out on Friday, before the event. Either of these misconceptions could have been cleared up simply by reading the piece. But this is, increasingly, a bar that many refuse to clear.

This is a long-winded way of saying that I’m happy to have this outlet, where my audience is small and sympathetic and where I can avoid so many of the headaches involved in professional freelancing. Never say never, obviously, and I’ll pop up here and there. But what was always a bad bet has only gotten worse since I started doing this and I just don’t really have it in me to continue the slog. I need to focus on academic writing, book projects, and this website. Thanks for coming along.

throwing out the individual differences baby with the group differences bathwater

Group-level differences in cognitive ability are in the news again, thanks to the quickly-notorious Google memo on the company’s diversity efforts. You can find a lot of stuff written about these differences from qualified people already and I’m not gonna add to the pile, other than to say that from where I’m sitting, if there are any sex differences in intelligence or math ability, they seem not to exist in early childhood and there are plausible cultural and social reasons that they would appear by high school. The science on personality trait differences seems less clear to me but then those constructs are also less concrete. In all of this I’m pretty much in keeping with the liberal mainstream.

But I do want to voice a caution, here, because there’s a natural but unfortunate tendency to make an unjustifiable corollary to arguments of these kind. Regular readers will know that I reject the idea of biological or genetic explanations for academic differences between races. Instead I follow most progressive people in thinking that the differences are socioeconomic and environmental in origin. There, too, I’ve often seen people make the same bad leaps: they tend to reject the idea of innate or genetic differences in individual academic ability or intelligence too. It’s not hard to understand why; talking about genetic differences in intelligence at all may seem like fruit from a poisoned tree, and why not just reject the whole idea altogether? But understanding the difference in group-level claims and individual-level claims is hugely important, both analytically and morally. It’s the difference between contributing to stereotypes that have contributed to marginalization and injustice of vulnerable groups, and accepting the reality that not all individual people are equally gifted in all areas.

And the data here is really, really clear: there are profound differences in individuals in academic or intellectual ability; these differences are generally quite durable over the course of one’s life, although of course there is some variability, as there is in any measurable psychological trait; and there is very strong evidence that a major portion of this difference comes from genetics. I don’t think that boys are smarter than girls or that black kids are less intelligent than white. I do think, and think both the empirical record and common sense shows, that not all people are equally talented in different intellectual domains, and that if you believe that the brain is the product of evolution, we should expect a significant amount of that difference to be genetic in origin, which is in fact what twin studies, adoption studies, and GWAS data show. I’ve written about all this in this space many times before.

You can think about this clearly if you just eliminate the comparison between groups that are supposedly different and look only at within-group distribution. So, for the purposes of this debate, look at women and their various metrics for intelligence and academic success, whether generally or in math/STEM/computer science. Forget about comparisons to men for a moment: within that group, on any properly validated intelligence metric, we find a normal distribution of ability. That is, there’s a mean, and there’s a distribution of about two thirds of the data points within a standard deviation from that mean, and about 95% of the data points within two standard deviations, and the distribution is just about symmetrical. Some women are better than other women on the SATs, IQ tests, quantitative reasoning tests, etc., and in predictable ways. The same exact condition applies when looking at distributions of black students, Asian students, students from Turkey, students who attend public schools, students who are left-handed, students who play Little League, etc. – real, persistent, and predictable differences of ability between individuals.

Now these individual differences don’t have much to tell us about diversity efforts like those at Google, which for the record I support, other than to say that Google is probably looking for those in the very top reaches of these distributions no matter what. But they say a hell of a lot about how we should approach education from a policy level. Policy has to reflect our empirical understandings of reality, and right now, ours doesn’t, as it is based on the false notion that all students can be brought to meet arbitrary performance standards, that there are no intrinsic limits to how well any individual student can perform, and that the purpose of schooling should train every student to be a Stanford-education Silicon Valley superstar. That’s the kind of cheery, optimistic, utterly-unachievable policy goal that comes from thinking that, because there aren’t genetic differences in intelligence between men and women or between races, there are no such differences between individual people either. That’s wrong and destructive and we can’t allow our necessary efforts to oppose bigotry to lead us in that direction.

For a lot of great thoughts on how to ethically consider genetic influences on individual intelligence, I recommend the work of the brilliant Paige Harden.


If you found value in this blog post, please consider supporting this project financially by clicking on the Patreon or Paypal link in the sidebar to the right.

The Academic “Success Sequence” – Get Lucky at Birth, Mostly

Matt Bruenig critiques the concept of the “Success Sequence” quite convincingly here. There are a lot of just-so stories in our culture about what it takes to be a success. Typically, these stories are confusing the lines of causation all over the place, failing to see that confounds and covariates are doing most of the explaining.

I sometimes get anxious emails from parents, wondering what they need to do to make sure their children are going to be OK academically. And because of networking effects and the nature of who reads this small-audience education blog, I can mostly tell them accurately that they don’t really have to do much of anything; they’ve already set up their children to succeed simply by virtue of having them. Here’s the real Academic Success Sequence:

  1. Be born to college-educated parents.1
  2. Be born to middle-class-or-above parents.
  3. Be born without a severe cognitive or developmental disability.
  4. Don’t be exposed to lead in infancy or early childhood.
  5. Don’t be born severely premature or at very low birth weight.
  6. Don’t be physically abused or neglected.

If you are one of those lucky enough to tick off these boxes, congratulations. You’ve got the vast majority of the accounted-for variance breaking in your favor. Is everything accounted for? No. We’ve got a lot of variance in cognitive and educational outcomes that never seems to be systematically explainable. I actually think that’s a good thing – perfect determinism is contrary to the fight for human meaning – but it’s important to say that this variance is not only not currently accounted for, it is likely never-to-be accounted for. This is what the behavioral geneticists call the “gloomy prospect“: the possibility that large portions of unaccounted-for variation in psychological traits like intelligence are the product of truly non-systematic events, like particular psychological traumas, getting a concussion, meeting the right person, having the right conversation at the right time….

Thus it’s the case that some people can “win” in all of the above categories and still suffer from real hardship in life, just as some can be on the wrong side in many or all of them and flourish. Still: if you’re an educated, employed parent raising a healthy child in a stable home environment, the odds are strongly in the favor of that child’s eventual academic success. Of course, none of this stuff is stuff that individuals can control, and much of it is not stuff that parents can control either – particularly given that the parents were once the children whose outcomes were similarly conditioned….

Now many people will say, well yeah, of course these things matter. But what do we do beyond that stuff? How do we set our kids up to succeed? I’m not going to say that nothing you do matters. But in terms of moving the quantitative indicators that people are, sadly, most fixated on are stubborn and hard to move. Some things appear to work – intensive one-on-one or small-group tutoring seems to me to have the most promising research literature – but we’re playing with small effect sizes here, particularly in comparison to the influence of the factors listed above. Of course you want to bend as much of the variance in a positive direction as you can. But the effects tend to be so small, and thus so subject to being offset by minor random fluctuations in uncontrolled variation, that it’s just not worth worrying about them. The best thing you can do for your kid is to be present and kind and supportive and then stop stressing out.

The great irony is that we’ve seen this growing culture of panic on the part of bourgie parents about their child rearing practices at the exact historical moment that we’ve learned conclusively that these practices just don’t mean very much.

In particular, the Baby Einstein stuff, trips to museums, violin lessons, edutainment software – my understanding is that there just is little to no rigorous research that shows that this stuff works to move the needle on SAT scores or GPA or similar, once you control for the kinds of confounds listed above. Does that mean that this stuff doesn’t matter, that you shouldn’t do them? Of course not. Children should all have the opportunity to lead intellectually enriched, challenging, and varied lives. I’m very grateful that I had that chance myself. But you need to appreciate them for their own sake and on their own terms, not as a means to goose test scores. And obsessing over getting your kid into the right preschool is pointless too, as is worrying over selective high schools. It may make you feel like the right kind of parent to fixate on this stuff; it may, more cynically, help you feel competitive with other parents. But extant evidence suggests it just doesn’t matter. What does matter is giving your child commitment, love, structure, and a moral education, because life is about so much more than where you go to college.

Of course, many people in our society are not lucky enough to have been born into the kind of advantaged position described above. Given that fact, you’d think that our system would be set up to minimize the impact of these unchosen factors. Instead we work to maximize their impact and call the resulting system “meritocracy.”


 

for lack of what is found there

Louis Menand in The New Yorker: 

The funny thing about the resistance all these writers put up to the idea that poems can change people’s lives is that every one of them had his life changed by a poem. I did, too. When I was fourteen or fifteen, I found a copy of “Immortal Poems of the English Language” in a book closet in my school. It was a mass-market paperback, and the editor, Oscar Williams, had judged several of his own poems sufficiently deathless to merit inclusion. But he was an excellent anthologist, and I wore that book out. It changed my life. It made me want to become a writer.

I had an almost identical experience, with an anthology put together by XJ Kennedy, a poet, essayist, translator, and all around man of letters. That’s my copy pictured here. In sophomore year of high school my old Latin teacher Mrs. Montgomery (gone, now, but never forgotten) had wanted to share a poem with me, and had dug around in her closet to find this old, little-loved and forgotten literature collection. It was divided into three sections: fiction, poetry, and drama. In time I would read the whole thing cover to cover, but at the time I obsessed over the poetry section. Growing up in a arts- and literature-obsessed home, I had gotten plenty of exposure to poetry, but this was the first time I really felt like I had the time and inclination to truly explore the form on my own. I got a real poetry education from that book, and learned not just Keats and Housman but Linda Pastan’s “Ethics” and Chesterton’s “The Donkey” and Amiri Baraka’s “Preface to a Twenty Volume Suicide Note.” I read it under my desk during algebra class and in the cafeteria and on the bus rides home from cross country meets, and today the cover is held on with masking tape, because I wore the damn thing out. When high school was over, I stole it.

I am, as you know, skeptical of the degree to which quantitative educational metrics like test scores can be changed by teachers and schools. But this carries with it the essential qualification: that test scores are not the measure of education’s value. Because I read and talk about quantitative research, and because I acknowledge that these tools are broadly predictive of all manner of eventual academic outcomes, I am often in agreement with those who view education in a reductive light. But my objections to that reductive thinking are as real and important as my objections to those who think that all individual students can be brought to the same levels of achievement on standardized tests. Indeed, precisely because differences in academic ability are real, we must take seriously all the things that education can do which are not expressible in a test score. I doubt that this book made the slightest difference to my SAT scores. Yet like Menand’s, my life was forever changed.

To the Muse
by XJ Kennedy

Give me leave, Muse, in plan view to array
Your shift and bodice by the light of day.
I would have brought an epic. Be not vexed
Instead to grace a niggling schoolroom text;
Let down your sanction, help me to oblige
Him who would leash fresh devots to your liege,
And at your altar, grant that in a flash
They, he, and I know incense from dead ash.

yes, campus activists have attempted to censor completely mainstream views

This past week, the Los Angeles Times was kind enough to run a revised version of an argument I had made here in the recent past – that Republican support of colleges and universities has collapsed, likely because of constant incidents on campus that create a widespread impression of anti-conservative bias, and that since our public universities are chartered and funded as non-partisan institutions, and because Republicans control enormous political power, our institutions are deeply threatened. I stand by that case.

I have gotten the usual grab bag of responses, most of them unmoored from specific principles about who should be able to say what on campus, and some of them directly contradictory with each other. As is typical, the number one rhetorical move has been to insist that student activists are only targeting the worst of the worst, Milo Yiannopoulos and Richard Spencer and the like. The idea is that people with mainstream views are entirely free to say whatever they want without issue because they don’t directly threaten marginalized people. That idea is factually incorrect, as anyone with the barest grasp on the facts should know.

  • Student activists at Amherst University demanded that students who had criticized their protests be formally punished by the university and forced to attend sensitivity training.
  • At Oberlin, students made a formal demand that specific professors and administrators be fired because the students did not like their politics.
  • The Evergreen State College imbroglio involved students attempting to have a professor fired for criticizing one of their political actions.
  • At Wesleyan, campus activists attempted to have the campus newspaper defunded for running a mainstream conservative editorial.
  • A Dean at Claremont McKenna resigned following student backlash to an email she sent in response to complaints about the treatment of students of color.
  • Students at Reed College attempted to shut down an appearance by Kimberly Peirce, the director of Boys Don’t Cry, removing posters advertising her talk and attempting to shout her down during her presentation.
  • At Yale, students called for the resignation of Erika Christiakis for an email she wrote about culturally insensitive Halloween costumes and for the resignation of her husband Nicholas Christiakis for defending her.
  • At the University of California Santa Barbara, the student government voted for mandatory trigger warnings, which would enable any student to skip class material that they decided was offensive.
  • Laura Kipnis, a feminist professor at Northwestern, was the subject of a literal federal investigation because she published an essay students didn’t like.
  • Mount Holyoke canceled the Vagina Monologues on campus under student pressure.
  • American Sniper, a perfectly mainstream American blockbuster, was temporarily pushed off campus by student activists.
  • Activists are Western Washington demanded the creation of a 15 person panel that would engage in surveillance of students, professors, and administrators in order to monitor everyone involved on campus for any expressions or actions that body deemed “racist, anti- black, transphobic, cissexist, misogynistic, ablest, homophobic, islamophobic, xenophobic, anti-semitism, and otherwise oppressive behavior.” That body would have the ability to discipline campus community members, including firing tenured faculty.
  • A yoga class for disabled students at a Canadian university was canceled after students complained that yoga is a form of cultural appropriation.

There are more. You are free to support any or all of these student actions. But you are not free to pretend there is no trend here. Exactly how many of these incidents must pile up before people are willing to admit that many campus activists pursue censorship of ideas and expressions that they don’t like?

The obsession with Milo and Richard Spencer makes this conversation impossible in left circles. Those people are discussed endlessly because leftists believe that doing so makes it easy to argue – “what, you want Milo to be free to harass POC on campus?!?” But in fact because most conservatives on campus will simply be mainstream Republicans, this side conversation will be almost entirely pointless. What really matters is the way that perfectly mainstream positions are being run out of campus on a regular basis. And of course with a list like this we can be sure that there are many, many more cases that went unnoticed and unreported in the wider world.

You would think it would be easy for progressives and leftists simply to say “I support many actions that campus protesters take, but these censorship efforts are counterproductive and wrong.” But that almost never happens. That’s because in contemporary life, politics has almost nothing to do with principle, or even with political tactics. Instead it has to do with aligning yourself with the right broad social circles. To criticize specific actions of campus activists sounds to too many leftists like being “the wrong kind of person,” so they refuse to criticize students even when their actions are minimally helpful and maximally counterproductive. That in turn ensures that there’s no opportunity for the students to reflect, learn, and evolve.

There have, of course, been many leftist professors who have been the subject of censorship too. I have written about these cases and fought for those professors over and over again. They come not from student pressure but from administrative fecklessness, which is to be expected, as the administrators that sometimes accede to student censorship demands and those who silence leftist professors are working under the same philosophy: a corporate desire to avoid controversy and to protect the campus as a neoliberal institution. That students so often petition these same administrators to silence on their behalf speaks to the failure to truly grapple with the nature of administrative power.

Awhile back I laid out my frustrations with this conversation. In particular, almost no one who defends campus activist attempts to censor has ever articulated a coherent policy about who is and is not allowed to say what.

Whatever else, defenders of activists attempting to censor opinions they don’t like have to stop claiming that these censorship efforts only target the most extreme cases. Because that is simply, factually false. Stop obsessing about the most extreme cases and grapple with the clear and growing attempts to censor mainstream views on campus. It’s an important conversation to have. Or you can keep shouting “Milo!” over and over again because that’s easy and doesn’t force you into any difficult choices or conversations. That will ensure that we have no coherent defense against bias claims while the Republican party sets out to dismantle our institutions, brick by brick.