%0 Journal Article %J Applied Psychological Measurement %D 2014 %T General Test Overlap Control: Improved Algorithm for CAT and CCT %A Chen, Shu-Ying %A Lei, Pui-Wa %A Chen, Jyun-Hong %A Liu, Tzu-Chen %X

This article proposed a new online test overlap control algorithm that is an improvement of Chen’s algorithm in controlling general test overlap rate for item pooling among a group of examinees. Chen’s algorithm is not very efficient in that not only item pooling between current examinee and prior examinees is controlled for but also item pooling between previous examinees, which would have been controlled for when they were current examinees. The proposed improvement increases efficiency by only considering item pooling between current and previous examinees, and its improved performance over Chen is demonstrated in a simulated computerized adaptive testing (CAT) environment. Moreover, the proposed algorithm is adapted for computerized classification testing (CCT) using the sequential probability ratio test procedure and is evaluated against some existing exposure control procedures. The proposed algorithm appears to work best in controlling general test overlap rate among the exposure control procedures examined without sacrificing much classification precision, though longer tests might be required for more stringent control of item pooling among larger groups. Given the capability of the proposed algorithm in controlling item pooling among a group of examinees of any size and its ease of implementation, it appears to be a good test overlap control method.

%B Applied Psychological Measurement %V 38 %P 229-244 %U http://apm.sagepub.com/content/38/3/229.abstract %R 10.1177/0146621613513494 %0 Journal Article %J Applied Psychological Measurement %D 2013 %T Variable-Length Computerized Adaptive Testing Based on Cognitive Diagnosis Models %A Hsu, Chia-Ling %A Wang, Wen-Chung %A Chen, Shu-Ying %X

Interest in developing computerized adaptive testing (CAT) under cognitive diagnosis models (CDMs) has increased recently. CAT algorithms that use a fixed-length termination rule frequently lead to different degrees of measurement precision for different examinees. Fixed precision, in which the examinees receive the same degree of measurement precision, is a major advantage of CAT over nonadaptive testing. In addition to the precision issue, test security is another important issue in practical CAT programs. In this study, the authors implemented two termination criteria for the fixed-precision rule and evaluated their performance under two popular CDMs using simulations. The results showed that using the two criteria with the posterior-weighted Kullback–Leibler information procedure for selecting items could achieve the prespecified measurement precision. A control procedure was developed to control item exposure and test overlap simultaneously among examinees. The simulation results indicated that in contrast to no method of controlling exposure, the control procedure developed in this study could maintain item exposure and test overlap at the prespecified level at the expense of only a few more items.

%B Applied Psychological Measurement %V 37 %P 563-582 %U http://apm.sagepub.com/content/37/7/563.abstract %R 10.1177/0146621613488642 %0 Journal Article %J Applied Psychological Measurement %D 2010 %T A Procedure for Controlling General Test Overlap in Computerized Adaptive Testing %A Chen, Shu-Ying %X

To date, exposure control procedures that are designed to control test overlap in computerized adaptive tests (CATs) are based on the assumption of item sharing between pairs of examinees. However, in practice, examinees may obtain test information from more than one previous test taker. This larger scope of information sharing needs to be considered in conducting test overlap control. The purpose of this study is to propose a test overlap control method such that the proportion of overlapping items encountered by an examinee with a group of previous examinees (described as general test overlap rate) can be controlled. Results indicated that item exposure rate and general test overlap rate could be simultaneously controlled by implementing the procedure. In addition, these two indices were controlled on the fly without any iterative simulations conducted prior to operational CATs. Thus, the proposed procedure would be an efficient method for controlling both the item exposure and general test overlap in CATs.

%B Applied Psychological Measurement %V 34 %P 393-409 %U http://apm.sagepub.com/content/34/6/393.abstract %R 10.1177/0146621610367788 %0 Journal Article %J Journal of Educational Measurement %D 2006 %T Comparing Methods of Assessing Differential Item Functioning in a Computerized Adaptive Testing Environment %A Lei, Pui-Wa %A Chen, Shu-Ying %A Yu, Lan %X

Mantel-Haenszel and SIBTEST, which have known difficulty in detecting non-unidirectional differential item functioning (DIF), have been adapted with some success for computerized adaptive testing (CAT). This study adapts logistic regression (LR) and the item-response-theory-likelihood-ratio test (IRT-LRT), capable of detecting both unidirectional and non-unidirectional DIF, to the CAT environment in which pretest items are assumed to be seeded in CATs but not used for trait estimation. The proposed adaptation methods were evaluated with simulated data under different sample size ratios and impact conditions in terms of Type I error, power, and specificity in identifying the form of DIF. The adapted LR and IRT-LRT procedures are more powerful than the CAT version of SIBTEST for non-unidirectional DIF detection. The good Type I error control provided by IRT-LRT under extremely unequal sample sizes and large impact is encouraging. Implications of these and other findings are discussed.

%B Journal of Educational Measurement %V 43 %P 245–264 %U http://dx.doi.org/10.1111/j.1745-3984.2006.00015.x %R 10.1111/j.1745-3984.2006.00015.x %0 Journal Article %J Applied Psychological Measurement %D 2005 %T Controlling Item Exposure and Test Overlap in Computerized Adaptive Testing %A Chen, Shu-Ying %A Lei, Pui-Wa %X

This article proposes an item exposure control method, which is the extension of the Sympson and Hetter procedure and can provide item exposure control at both the item and test levels. Item exposure rate and test overlap rate are two indices commonly used to track item exposure in computerized adaptive tests. By considering both indices, item exposure can be monitored at both the item and test levels. To control the item exposure rate and test overlap rate simultaneously, the modified procedure attempted to control not only the maximum value but also the variance of item exposure rates. Results indicated that the item exposure rate and test overlap rate could be controlled simultaneously by implementing the modified procedure. Item exposure control was improved and precision of trait estimation decreased when a prespecified maximum test overlap rate was stringent.

%B Applied Psychological Measurement %V 29 %P 204-217 %U http://apm.sagepub.com/content/29/3/204.abstract %R 10.1177/0146621604271495