Kentucky Portfolio Scoring Improves

Status: 
Archived
Subject: 
K-12 Testing

The reliability of scoring Kentucky s writing portfolios has increased substantially over the past three years, according to an independent review of 6,592 portfolios from 100 schools conducted to assure comparability among schools. The independent readers agreed with the original school scores at a rate of 73.1 percent for portfolios from schools deliberately selected because of possible discrepancies in their test scores, and at a rate of 76.8 percent for schools selected at random.

 

The audit was used to modify school scores. It led the state to lower the deliberately-selected schools score by an average of 10.87 points on a 140-point scale, and lower the randomly-selected schools score by 5.09 points. After a similar audit in 1992-93, the average scoring change was a decrease of more than 35 points.

 

Kentucky's writing portfolio is part of the state s assessment and accountability system, contributing 14 percent to each school s accountability index. The index is used to evaluate schools for possible state interventions and to reward schools which show significant improvement (see Examiner, Spring 1993). Kentucky s performance assessment system has been one of the nation s most important and most studied state assessment reform efforts.

 

The improvement in Kentucky s inter-rater reliability provides additional evidence that portfolios can be used for accountability, provided that scores are not the sole basis for high-stakes decisions for individuals or schools and provided that there is sufficient professional development (see Examiner, Summer 1996). As teachers become more familiar with portfolios, their scoring accuracy increases. While teachers continued to score their own students more highly than did the auditors, the gap has narrowed substantially.

 

Meanwhile, under conservative pressure, Kentucky is adding some multiple-choice items back into its exams. Earlier, the state had included but not counted such items, then deleted them entirely (see Examiner, Fall 1994). The state s math portfolio program is also under review.