ETS and Test Cheating

Status: 
Archived
Subject: 
University Testing

Despite much publicized efforts by the Educational Testing Service (ETS) to discourage cheating on its exams, including the persecution of apparently innocent students (see Examiner, Summer 1994, Summer 1992, and Spring 1992), the testmaker ignored internal warnings about security problems in its computer adaptive Graduate Record Exam (GRE), according to documents recently made public in a U.S. district court. Unlike pencil-and-paper tests, in which large numbers of students are administered identical exams simultaneously, computer adaptive tests are individualized through the automated selection of questions from a pool of items based on how each student answered the previous ones.

To protect its image and profits in the rush to introduce the new type of test into the marketplace, ETS also misled the New York State legislature about how the system actually worked.

These revelations are included in depositions submitted by top ETS officials in support of their company's lawsuit charging the Stanley H. Kaplan Educational Center with copyright violation for compiling a list of items on the computer adaptive GRE (see Examiner, Winter 1994-95). Kaplan executives never made the questions public, but simply announced they had them and met with ETS to demonstrate how easy it was to obtain the items.

According to court documents, a year before the November 1993 introduction of the computer adaptive GRE, two ETS test developers wrote about their fear [that examinees] will remember questions and reveal them to their friends or to a coaching school, and that a group of examinees [might] memoriz[e] subsets of the pool and combin[e] their knowledge. Another test designer noted, Further research clearly remains to be done in this area if adaptive testing is to become a secure alternative to conventional paper-and-pencil testing, and that [N]o amount of exposure control can eliminate the possibility of an organized group memorizing the items and sharing them with others.

Disregarding those strong concerns, ETS administered a single pool of GRE questions from November 1993 through September 1994 and a second pool for the next three months, until the Kaplan disclosures. During this period, in response to questions at a New York legislative hearing where FairTest and others testified about potential problems in computerized exams, ETS reassured the State Senate, The GRE program has chosen to use many pools of questions simultaneously. That statement was not true, at least for many more months.

In fact, in her deposition, ETS President Nancy Cole admits it is now clear, in retrospect, that back in November of 1994, one or two good memorizers could have given a substantial advantage to a subsequent test-taker. Dr. Cole notes two major concerns: First, that a relatively small number of people were able to come up with so many questions; second, that there was such a high degree of overlap among the tests that were taken by the Kaplan test-takers.

After Kaplan showed ETS officials portions of 200 questions test-takers had memorized, the testmaker first suspended the computer adaptive GRE and then limited its administration for several months.

Today, ETS claims to be using a sufficiently large number of item pools to ensure that a team of memorizers cannot compromise test security, but this claim has not been subject to outside validation. Given ETS history of misleading statements on the topic and the strong incentives for test-taker collaboration, the possibility of items becoming available to select few remains very real. Kaplan defends its actions as promoting the public interest, relying on ETS admission that the incident led to improved test design practices.

Indeed, the desire to guarantee all test-takers equal access to previously administered questions is one reason New York recently extended some truth-in-testing provisions to computer adaptive exams (see Examiner, Fall 1996). Concerns raised by the Kaplan-ETS dispute also help explain why the testmaker was unable to meet its goal of phasing out the less profitable GRE written test this year. As the controversy about many facets of computerized testing continues, the pencil-and-paper GRE, with a fresh set of items used at each administration, is likely to remain available for the next few years.

See also our fact sheet on computerized testing.