The Most Infamous Example of Cultural Bias on the SAT®

Updated on April 20, 2023

This week, we examine the mystery surrounding the most infamous SAT® problem of all time.  

Every local newspaper, it seems, is obligated to print at least one story per year decrying the SAT® as a biased and invalid form of cruel and unusual punishment.

These articles no doubt make for effective click bait. However, if you're familiar with the test, you'll immediately realize that these articles are generally 100% research-free. Many of the current criticisms point to alleged problems with previous versions of the SAT®, and virtually all the articles rely too heavily on the questionable testimony of a few prominent anti-test advocates. 

Even now, a decade after the 2005 redesign of the SAT® (the last update prior to the 2016 redesign), it's still not uncommon to read rehashed articles about the cultural bias of the analogy section of the SAT®. That's right: the analogy section. 

Just so we're all on the same page, the analogy section of the SAT® was removed from the test in 2005. Still, it gets at least a mention in virtually every new article about the current SAT's® alleged deficiencies. Even more curiously, a single SAT® analogy problem gets more press than some presidential candidates. Here it is: 

The correct answer is (C) oarsman : regatta.

At issue is the fact that the correct answer requires the knowledge of vocabulary words that are more likely to have come up in the daily lives of affluent white students than in the lives of less affluent minority students. This test question was famously dissected in the controversial 1994 book, The Bell Curve, by Herrnstein and Murray. 

1994, the book's publication date, seems to be why most lazy reporters concluded that the oarsman : regatta problem appeared on an SAT® in the mid-nineties. For years I had believed the many authors who'd claimed this question appeared on a test in the 1990s. Recently, however, I came across this bit of information:

That Image is taken from Leslie Yalof Garfield, The Cost of Good Intentions: Why the Supreme Court's Decision Upholding Affirmative Action Admission Programs Is Detrimental to the Cause, 27 Pace L. Rev. 15 (2006) To be clear, I'm not commenting on Garfield's article or endorsing his claims. In fact, I haven't even read the paper in its entirety.

I mention this because I was surprised to see that Garfield's exhaustively footnoted article from 2006 indicates that the infamous oarsman : regatta problem appeared on "an early 1980s exam." This information prompted me to search for hard evidence that the problem actually did show up on a test at some point in the 1990s. It's entirely possible that the College Board did reuse the question. In fact, some recent SAT® tests have contained reused test items. Even so, Garfield doesn't seem to have been aware of any encore performances by the controversial question. During my research, I did find one other source that claimed the oarsman : regatta question last sailed in 1973, but it was clearly biased and I could not confirm the accuracy of the article's claims.

More to the point, it seems odd that the College Board would even consider reusing a test question that had been featured in a best selling book as evidence of standardized testing bias. Then again, the College Board has been rather tone deaf in its response to controversies in the past.  

After spending too much time on this mystery, I've yet to come across any evidence that the oarsman : regatta problem ever appeared on an SAT® test during the 1990s. That is my tentative conclusion for now. Obviously, the best way to get to the bottom of this would be to review every SAT® test from the 1990s. Unfortunately, I'm not among the eccentric collectors of 1990s SAT® tests.

So, I'm appealing for help. If anyone reading this happens to have a hermetically-sealed library of ancient SATs®, please look over those scrolls and let me know which SAT® test(s) the problem actually appeared on. I'll be very grateful for your help.

Updated 11/28/2017: If you're interested in the larger issues surrounding the difficulty of measuring general intelligence and the controversy that has plagued Charles Murray since the publication of The Bell Curve, you should immediately listen to Murray's interview from earlier this year on The Waking Up Podcast with Sam Harris. Regardless of your feelings about the issues, it's a fascinating interview. 

Also, if you're interested in factors that affect students' performance on test day, be sure to check out this article from our archives:

The Number One Predictor of Test Prep Score Improvement.

Please take a moment to subscribe. You can be sure that we will never send you anything that we don't think you'll find useful or enjoy reading.