GMAT Cheating Scandal Illustrates Computerized Testing Vulnerability

Status: 
Archived
Subject: 
University Testing

FairTest Examiner, July 2008

More than 6,000 business school applicants who used Scoretop.com to prepare for the Graduate Management Admissions Test (GMAT) may face cancellation of their scores because the site posted actual exam questions while they still were being administered. The incident demonstrates a fundamental flaw of computer adaptive tests, which reuse the same item pools for weeks at a time.

Test-makers had been warned that this security problem was rife for exploitation more than a decade ago. But instead of redesigning their product, the test makers sued the Kaplan coaching company, which first revealed the problem.

For five years, Scoretop.com sold “VIP access” for $30 a month. Registered users were able to preview GMAT questions, including some currently in use on operational forms of the exam. Apparently the items were provided by test-takers who memorized them.

Under computerized adaptive testing, the questions administered to a particular student are selected from a pool of items based on the test-taker’s prior answers. A string of correct responses will produce a more difficult question. Several wrong answers in a row result in the presentation of an easier item. The item pool is periodically refreshed, but the same questions are reused for at least several weeks.

In late June, the Graduate Management Admissions Council (GMAC) sponsors of the GMAT announced that it had won a $2.35 million copyright infringement lawsuit against Scoretop.com. The court order allowed GMAC to seize the firm’s records, including user identification data, and post a “Warning” message on the website.

So far, only one GMAT score is known to have been cancelled. GMAC says, “Our initial focus is on those who disclosed live questions to Scoretop” but notes that “If, as part of this investigation, there is compelling evidence of a test taker knowingly violating GMAC policy, we may cancel GMAT scores.”

The testing industry was well aware of the vulnerability of computer adaptive tests to what it calls “pre-disclosure.” Before the 1993 introduction of the computer adaptive Graduate Record Exam (GRE), two researchers at the Educational Testing Service (ETS) wrote about their “fear [that examinees] will remember questions and reveal them to their friends or to a coaching school” and that “a group of examinees [might] memoriz[e] subsets of the pool and combin[e] their knowledge.

To expose the problem, staff from the Stanley Kaplan Education Center took the computerized GRE, compiled a list of items they had memorized, and presented it to ETS officials. ETS, which then administered the GMAT as well as the GRE, responded by suing Kaplan for copyright violations, even though the questions were never made public (http://www.fairtest.org/ets-and-test-cheating). After this incident, test-makers said they began using much larger item pools and changing them more frequently, but there is no proof for this claim. In 2006, ETS lost the GMAT testing contract after a series of administrative and scoring errors. The test is now run by the global conglomerate Pearson.

The Scoretop.com lawsuit demonstrates that security risks created by repeated use of the same test items in computerized adaptive testing still have not been addressed.