%0 Journal Article %J Applied Psychological Measurement %D 2010 %T A Method for the Comparison of Item Selection Rules in Computerized Adaptive Testing %A Barrada, Juan Ramón %A Olea, Julio %A Ponsoda, Vicente %A Abad, Francisco José %X

In a typical study comparing the relative efficiency of two item selection rules in computerized adaptive testing, the common result is that they simultaneously differ in accuracy and security, making it difficult to reach a conclusion on which is the more appropriate rule. This study proposes a strategy to conduct a global comparison of two or more selection rules. A plot showing the performance of each selection rule for several maximum exposure rates is obtained and the whole plot is compared with other rule plots. The strategy was applied in a simulation study with fixed-length CATs for the comparison of six item selection rules: the point Fisher information, Fisher information weighted by likelihood, Kullback-Leibler weighted by likelihood, maximum information stratification with blocking, progressive and proportional methods. Our results show that there is no optimal rule for any overlap value or root mean square error (RMSE). The fact that a rule, for a given level of overlap, has lower RMSE than another does not imply that this pattern holds for another overlap rate. A fair comparison of the rules requires extensive manipulation of the maximum exposure rates. The best methods were the Kullback-Leibler weighted by likelihood, the proportional method, and the maximum information stratification method with blocking.

%B Applied Psychological Measurement %V 34 %P 438-452 %U http://apm.sagepub.com/content/34/6/438.abstract %R 10.1177/0146621610370152 %0 Journal Article %J Applied Psychological Measurement %D 2009 %T Multiple Maximum Exposure Rates in Computerized Adaptive Testing %A Barrada, Juan Ramón %A Veldkamp, Bernard P. %A Olea, Julio %X

Computerized adaptive testing is subject to security problems, as the item bank content remains operative over long periods and administration time is flexible for examinees. Spreading the content of a part of the item bank could lead to an overestimation of the examinees' trait level. The most common way of reducing this risk is to impose a maximum exposure rate (rmax) that no item should exceed. Several methods have been proposed with this aim. All of these methods establish a single value of rmax throughout the test. This study presents a new method, the multiple-rmax method, that defines as many values of rmax as the number of items presented in the test. In this way, it is possible to impose a high degree of randomness in item selection at the beginning of the test, leaving the administration of items with the best psychometric properties to the moment when the trait level estimation is most accurate. The implementation of the multiple-r max method is described and is tested in simulated item banks and in an operative bank. Compared with a single maximum exposure method, the new method has a more balanced usage of the item bank and delays the possible distortion of trait estimation due to security problems, with either no or only slight decrements of measurement accuracy.

%B Applied Psychological Measurement %V 33 %P 58-73 %U http://apm.sagepub.com/content/33/1/58.abstract %R 10.1177/0146621608315329 %0 Journal Article %J Educational and Psychological Measurement %D 2003 %T Psychometric and Psychological Effects of Item Selection and Review on Computerized Testing %A Revuelta, Javier %A Ximénez, M. Carmen %A Olea, Julio %X

Psychometric properties of computerized testing, together with anxiety and comfort of examinees, are investigated in relation to item selection routine and the opportunity for response review. Two different hypotheses involving examinee anxiety were used to design test properties: perceived control and perceived performance. The study involved three types of administration of a computerized English test for Spanish speakers (adaptive, easy adaptive, and fixed) and four review conditions (no review, review at end, review by blocks of 5 items, and review item-by-item). These were applied to a sample of 557 first-year psychology undergraduate students to examine main and interaction effects of test type and review on psychometric and psychological variables. Statistically significant effects were found in test precision among the different types of test. Response review improved ability estimates and increased testing time. No psychological effects on anxiety were found. Examinees in all review conditions considered more important the possibility of review than those who were not allowed to review. These results concur with previous findings on examinees' preference for item review and raise some issues that should be addressed in the field of tests with item review.

%B Educational and Psychological Measurement %V 63 %P 791-808 %U http://epm.sagepub.com/content/63/5/791.abstract %R 10.1177/0013164403251282