What Do SAT-I Reading Items Measure

University Testing

In the early 1990s, University of Georgia Psychology Professor Stuart Katz directed a set of ground-breaking studies demonstrating that test-takers could answer many SAT “Reading Comprehension” (RC) questions correctly without even seeing the passages on which they supposedly were based. Katz and others argued that this was because the items were actually measuring factors such as outside knowledge and test-wisedness, not the construct of reading ability.


SAT promoters criticized Katz’ research as being outdated, since it examined items from an “old” version of what was then called the Scholastic Aptitude Test. Indeed, when a slightly revised exam, called the SAT-I, was introduced in 1994 (see Examiner, Winter 1992-93), one test-maker claim was that “new” RC items would be longer and more complex, thus a better measure of the underlying construct.


Now Katz and two University of Georgia colleagues have run similar experiments with questions from the SAT-I. Despite changes in the items’ word count and structure, students could still answer 36% correctly without seeing the reading passages. On five-option, multiple-choice problems like these, the probability of guessing the wanted answer should be just 20%.


On nearly three-quarters of the items, correct answer rates exceeded chance. Adjusting for possible errors in statistical significance from the relatively small sample, the authors still conclude, “Even by conservative estimates, at least six of every ten RC items are flawed.”


They continue, “Outside knowledge should play no role in an assessment of passage reading ability if the passages are missing. The findings of this study, however, show that outside knowledge plays a very large role on the RC SAT-I task.”


The new report has significant implications for test-coaching as well as for the exam’s validity. Major SAT-prep companies have long told their clients not to bother reading the passages if they were pressed for time. Now they know for certain that this approach most definitely raises scores.


- “Answering Reading Comprehension Items Without the Passages on the SAT-I,” Psychological Reports, in press, by Stuart Katz, Christopher Johnson and Erika Pohl.