TY - JOUR T1 - Bayesian item selection in constrained adaptive testing JF - Psicologica Y1 - 2010 A1 - Veldkamp, B. P. KW - computerized adaptive testing AB - Application of Bayesian item selection criteria in computerized adaptive testing might result in improvement of bias and MSE of the ability estimates. The question remains how to apply Bayesian item selection criteria in the context of constrained adaptive testing, where large numbers of specifications have to be taken into account in the item selection process. The Shadow Test Approach is a general purpose algorithm for administering constrained CAT. In this paper it is shown how the approach can be slightly modified to handle Bayesian item selection criteria. No differences in performance were found between the shadow test approach and the modifiedapproach. In a simulation study of the LSAT, the effects of Bayesian item selection criteria are illustrated. The results are compared to item selection based on Fisher Information. General recommendations about the use of Bayesian item selection criteria are provided. VL - 31 ER - TY - CHAP T1 - Designing Item Pools for Adaptive Testing T2 - Elements of Adaptive Testing Y1 - 2010 A1 - Veldkamp, B. P. A1 - van der Linden, W. J. JF - Elements of Adaptive Testing ER - TY - JOUR T1 - Detection of aberrant item score patterns in computerized adaptive testing: An empirical example using the CUSUM JF - Personality and Individual Differences Y1 - 2010 A1 - Egberink, I. J. L. A1 - Meijer, R. R. A1 - Veldkamp, B. P. A1 - Schakel, L. A1 - Smid, N. G. KW - CAT KW - computerized adaptive testing KW - CUSUM approach KW - person Fit AB - The scalability of individual trait scores on a computerized adaptive test (CAT) was assessed through investigating the consistency of individual item score patterns. A sample of N = 428 persons completed a personality CAT as part of a career development procedure. To detect inconsistent item score patterns, we used a cumulative sum (CUSUM) procedure. Combined information from the CUSUM, other personality measures, and interviews showed that similar estimated trait values may have a different interpretation.Implications for computer-based assessment are discussed. VL - 48 SN - 01918869 ER - TY - JOUR T1 - Comparison of methods for controlling maximum exposure rates in computerized adaptive testing JF - Psicothema Y1 - 2009 A1 - Barrada, J A1 - Abad, F. J. A1 - Veldkamp, B. P. KW - *Numerical Analysis, Computer-Assisted KW - Psychological Tests/*standards/*statistics & numerical data AB - This paper has two objectives: (a) to provide a clear description of three methods for controlling the maximum exposure rate in computerized adaptive testing —the Symson-Hetter method, the restricted method, and the item-eligibility method— showing how all three can be interpreted as methods for constructing the variable sub-bank of items from which each examinee receives the items in his or her test; (b) to indicate the theoretical and empirical limitations of each method and to compare their performance. With the three methods, we obtained basically indistinguishable results in overlap rate and RMSE (differences in the third decimal place). The restricted method is the best method for controlling exposure rate, followed by the item-eligibility method. The worst method is the Sympson-Hetter method. The restricted method presents problems of sequential overlap rate. Our advice is to use the item-eligibility method, as it saves time and satisfies the goals of restricting maximum exposure. Comparación de métodos para el control de tasa máxima en tests adaptativos informatizados. Este artículo tiene dos objetivos: (a) ofrecer una descripción clara de tres métodos para el control de la tasa máxima en tests adaptativos informatizados, el método Symson-Hetter, el método restringido y el métodode elegibilidad del ítem, mostrando cómo todos ellos pueden interpretarse como métodos para la construcción del subbanco de ítems variable, del cual cada examinado recibe los ítems de su test; (b) señalar las limitaciones teóricas y empíricas de cada método y comparar sus resultados. Se obtienen resultados básicamente indistinguibles en tasa de solapamiento y RMSE con los tres métodos (diferencias en la tercera posición decimal). El método restringido es el mejor en el control de la tasa de exposición,seguido por el método de elegibilidad del ítem. El peor es el método Sympson-Hetter. El método restringido presenta un problema de solapamiento secuencial. Nuestra recomendación sería utilizar el método de elegibilidad del ítem, puesto que ahorra tiempo y satisface los objetivos de limitar la tasa máxima de exposición. VL - 21 SN - 0214-9915 (Print)0214-9915 (Linking) N1 - Barrada, Juan RamonAbad, Francisco JoseVeldkamp, Bernard PComparative StudySpainPsicothemaPsicothema. 2009 May;21(2):313-20. ER - TY - CHAP T1 - The development of a computerized adaptive test for integrity Y1 - 2007 A1 - Egberink, I. J. L. A1 - Veldkamp, B. P. CY - D. J. Weiss (Ed.), Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing. N1 - {PDf file, 290 KB} ER - TY - RPRT T1 - A multiple objective test assembly approach for exposure control problems in computerized adaptive testing Y1 - 2007 A1 - Veldkamp, B. P. A1 - Verschoor, Angela J. A1 - Theo Eggen JF - Measurement and Research Department Reports PB - Cito CY - Arnhem, The Netherlands ER - TY - JOUR T1 - Assembling a computerized adaptive testing item pool as a set of linear tests JF - Journal of Educational and Behavioral Statistics Y1 - 2006 A1 - van der Linden, W. J. A1 - Ariel, A. A1 - Veldkamp, B. P. KW - Algorithms KW - computerized adaptive testing KW - item pool KW - linear tests KW - mathematical models KW - statistics KW - Test Construction KW - Test Items AB - Test-item writing efforts typically results in item pools with an undesirable correlational structure between the content attributes of the items and their statistical information. If such pools are used in computerized adaptive testing (CAT), the algorithm may be forced to select items with less than optimal information, that violate the content constraints, and/or have unfavorable exposure rates. Although at first sight somewhat counterintuitive, it is shown that if the CAT pool is assembled as a set of linear test forms, undesirable correlations can be broken down effectively. It is proposed to assemble such pools using a mixed integer programming model with constraints that guarantee that each test meets all content specifications and an objective function that requires them to have maximal information at a well-chosen set of ability values. An empirical example with a previous master pool from the Law School Admission Test (LSAT) yielded a CAT with nearly uniform bias and mean-squared error functions for the ability estimator and item-exposure rates that satisfied the target for all items in the pool. PB - Sage Publications: US VL - 31 SN - 1076-9986 (Print) ER - TY - CONF T1 - Multiple maximum exposure rates in computerized adaptive testing T2 - Paper presented at the SMABS-EAM Conference Y1 - 2006 A1 - Barrada, J A1 - Veldkamp, B. P. A1 - Olea, J. JF - Paper presented at the SMABS-EAM Conference CY - Budapest, Hungary ER - TY - ABST T1 - Constraining item exposure in computerized adaptive testing with shadow tests Y1 - 2005 A1 - van der Linden, W. J. A1 - Veldkamp, B. P. CY - Law School Admission Council Computerized Testing Report 02-03 ER - TY - JOUR T1 - Infeasibility in automated test assembly models: A comparison study of different methods JF - Journal of Educational Measurement Y1 - 2005 A1 - Huitzing, H. A. A1 - Veldkamp, B. P. A1 - Verschoor, A. J. KW - Algorithms KW - Item Content (Test) KW - Models KW - Test Construction AB - Several techniques exist to automatically put together a test meeting a number of specifications. In an item bank, the items are stored with their characteristics. A test is constructed by selecting a set of items that fulfills the specifications set by the test assembler. Test assembly problems are often formulated in terms of a model consisting of restrictions and an objective to be maximized or minimized. A problem arises when it is impossible to construct a test from the item pool that meets all specifications, that is, when the model is not feasible. Several methods exist to handle these infeasibility problems. In this article, test assembly models resulting from two practical testing programs were reconstructed to be infeasible. These models were analyzed using methods that forced a solution (Goal Programming, Multiple-Goal Programming, Greedy Heuristic), that analyzed the causes (Relaxed and Ordered Deletion Algorithm (RODA), Integer Randomized Deletion Algorithm (IRDA), Set Covering (SC), and Item Sampling), or that analyzed the causes and used this information to force a solution (Irreducible Infeasible Set-Solver). Specialized methods such as the IRDA and the Irreducible Infeasible Set-Solver performed best. Recommendations about the use of different methods are given. (PsycINFO Database Record (c) 2005 APA ) (journal abstract) VL - 42 ER - TY - JOUR T1 - Constraining item exposure in computerized adaptive testing with shadow tests JF - Journal of Educational and Behavioral Statistics Y1 - 2004 A1 - van der Linden, W. J. A1 - Veldkamp, B. P. KW - computerized adaptive testing KW - item exposure control KW - item ineligibility constraints KW - Probability KW - shadow tests AB - Item-exposure control in computerized adaptive testing is implemented by imposing item-ineligibility constraints on the assembly process of the shadow tests. The method resembles Sympson and Hetter’s (1985) method of item-exposure control in that the decisions to impose the constraints are probabilistic. The method does not, however, require time-consuming simulation studies to set values for control parameters before the operational use of the test. Instead, it can set the probabilities of item ineligibility adaptively during the test using the actual item-exposure rates. An empirical study using an item pool from the Law School Admission Test showed that application of the method yielded perfect control of the item-exposure rates and had negligible impact on the bias and mean-squared error functions of the ability estimator. PB - American Educational Research Assn: US VL - 29 SN - 1076-9986 (Print) ER - TY - JOUR T1 - Constructing rotating item pools for constrained adaptive testing JF - Journal of Educational Measurement Y1 - 2004 A1 - Ariel, A. A1 - Veldkamp, B. P. A1 - van der Linden, W. J. KW - computerized adaptive tests KW - constrained adaptive testing KW - item exposure KW - rotating item pools AB - Preventing items in adaptive testing from being over- or underexposed is one of the main problems in computerized adaptive testing. Though the problem of overexposed items can be solved using a probabilistic item-exposure control method, such methods are unable to deal with the problem of underexposed items. Using a system of rotating item pools, on the other hand, is a method that potentially solves both problems. In this method, a master pool is divided into (possibly overlapping) smaller item pools, which are required to have similar distributions of content and statistical attributes. These pools are rotated among the testing sites to realize desirable exposure rates for the items. A test assembly model, motivated by Gulliksen's matched random subtests method, was explored to help solve the problem of dividing a master pool into a set of smaller pools. Different methods to solve the model are proposed. An item pool from the Law School Admission Test was used to evaluate the performances of computerized adaptive tests from systems of rotating item pools constructed using these methods. (PsycINFO Database Record (c) 2007 APA, all rights reserved) PB - Blackwell Publishing: United Kingdom VL - 41 SN - 0022-0655 (Print) ER - TY - CONF T1 - Constraining item exposure in computerized adaptive testing with shadow tests T2 - Paper presented at the Annual meeting of the National Council on Measurement in Education Y1 - 2003 A1 - van der Linden, W. J. A1 - Veldkamp, B. P. JF - Paper presented at the Annual meeting of the National Council on Measurement in Education CY - Chicago IL N1 - #vdLI03-02 ER - TY - CONF T1 - Controlling item exposure and item eligibility in computerized adaptive testing Y1 - 2003 A1 - van der Linden, W. J. A1 - Veldkamp, B. P. ER - TY - CONF T1 - Implementing an alternative to Sympson-Hetter item-exposure control in constrained adaptive testing T2 - Paper presented at the Annual meeting of the National Council on Measurement in Education Y1 - 2003 A1 - Veldkamp, B. P. A1 - van der Linden, W. J. JF - Paper presented at the Annual meeting of the National Council on Measurement in Education CY - Chicago IL ER - TY - CHAP T1 - Item selection in polytomous CAT T2 - New developments in psychometrics Y1 - 2003 A1 - Veldkamp, B. P. ED - A. Okada ED - K. Shigenasu ED - Y. Kano ED - J. Meulman KW - computerized adaptive testing JF - New developments in psychometrics PB - Psychometric Society, Springer CY - Tokyo, Japan ER - TY - CHAP T1 - Item selection in polytomous CAT Y1 - 2003 A1 - Veldkamp, B. P. CY - H. Yanai, A. Okada, K. Shigemasu, Y Kano, and J. J. Meulman (eds.), New developments in psychometrics (pp. 207-214). Tokyo, Japan: Springer-Verlag. N1 - #VE03207 {PDF file, 79 KB} ER - TY - ABST T1 - Constraining item exposure in computerized adaptive testing with shadow tests (Research Report No. 02-06) Y1 - 2002 A1 - van der Linden, W. J. A1 - Veldkamp, B. P. CY - University of Twente, The Netherlands ER - TY - RPRT T1 - Mathematical-programming approaches to test item pool design Y1 - 2002 A1 - Veldkamp, B. P. A1 - van der Linden, W. J. A1 - Ariel, A. KW - Adaptive Testing KW - Computer Assisted KW - Computer Programming KW - Educational Measurement KW - Item Response Theory KW - Mathematics KW - Psychometrics KW - Statistical Rotation computerized adaptive testing KW - Test Items KW - Testing AB - (From the chapter) This paper presents an approach to item pool design that has the potential to improve on the quality of current item pools in educational and psychological testing and hence to increase both measurement precision and validity. The approach consists of the application of mathematical programming techniques to calculate optimal blueprints for item pools. These blueprints can be used to guide the item-writing process. Three different types of design problems are discussed, namely for item pools for linear tests, item pools computerized adaptive testing (CAT), and systems of rotating item pools for CAT. The paper concludes with an empirical example of the problem of designing a system of rotating item pools for CAT. PB - University of Twente, Faculty of Educational Science and Technology CY - Twente, The Netherlands SN - 02-09 N1 - Using Smart Source ParsingAdvances in psychology research, Vol. ( Hauppauge, NY : Nova Science Publishers, Inc, [URL:http://www.Novapublishers.com]. vi, 228 pp ER - TY - JOUR T1 - Multidimensional adaptive testing with constraints on test content JF - Psychometrika Y1 - 2002 A1 - Veldkamp, B. P. A1 - van der Linden, W. J. AB - The case of adaptive testing under a multidimensional response model with large numbers of constraints on the content of the test is addressed. The items in the test are selected using a shadow test approach. The 0–1 linear programming model that assembles the shadow tests maximizes posterior expected Kullback-Leibler information in the test. The procedure is illustrated for five different cases of multidimensionality. These cases differ in (a) the numbers of ability dimensions that are intentional or should be considered as ldquonuisance dimensionsrdquo and (b) whether the test should or should not display a simple structure with respect to the intentional ability dimensions. VL - 67 N1 - Psychometric Society, US ER - TY - ABST T1 - Implementing constrained CAT with shadow tests for large item pools Y1 - 2001 A1 - Veldkamp, B. P. CY - Submitted for publication ER - TY - CHAP T1 - Designing item pools for computerized adaptive testing T2 - Computerized adaptive testing: Theory and practice Y1 - 2000 A1 - Veldkamp, B. P. A1 - van der Linden, W. J. JF - Computerized adaptive testing: Theory and practice PB - Kluwer Academic Publishers CY - Dendrecht, The Netherlands ER - TY - JOUR T1 - An integer programming approach to item bank design JF - Applied Psychological Measurement Y1 - 2000 A1 - van der Linden, W. J. A1 - Veldkamp, B. P. A1 - Reese, L. M. KW - Aptitude Measures KW - Item Analysis (Test) KW - Item Response Theory KW - Test Construction KW - Test Items AB - An integer programming approach to item bank design is presented that can be used to calculate an optimal blueprint for an item bank, in order to support an existing testing program. The results are optimal in that they minimize the effort involved in producing the items as revealed by current item writing patterns. Also presented is an adaptation of the models, which can be used as a set of monitoring tools in item bank management. The approach is demonstrated empirically for an item bank that was designed for the Law School Admission Test. VL - 24 ER - TY - ABST T1 - Modifications of the branch-and-bound algorithm for application in constrained adaptive testing (Research Report 00-05) Y1 - 2000 A1 - Veldkamp, B. P. CY - Enschede, The Netherlands: University of Twente, Faculty of Educational Science and Technology, Department of Measurement and Data Analysis ER - TY - ABST T1 - Multidimensional adaptive testing with constraints on test content (Research Report 00-11) Y1 - 2000 A1 - Veldkamp, B. P. A1 - van der Linden, W. J. CY - Enschede, The Netherlands: University of Twente, Faculty of Educational Science and Technology, Department of Measurement and Data Analysis ER - TY - ABST T1 - Designing item pools for computerized adaptive testing (Research Report 99-03 ) Y1 - 1999 A1 - Veldkamp, B. P. A1 - van der Linden, W. J. CY - Enschede, The Netherlands: University of Twente, Faculty of Educational Science and Technology, Department of Measurement and Data Analysis ER -