Adaptive short forms for outpatient rehabilitation outcome assessment - Jette, A. M.; Haley, S. M.; Ni, P.; Moed, R. (American Journal of Physical Medicine and Rehabilitation, 2008)
OBJECTIVE: To develop outpatient Adaptive Short Forms for the Activity Measure for Post-Acute Care item bank for use in outpatient therapy settings. DESIGN: A convenience sample of 11,809 adults with spine, lower limb, upper limb, and miscellaneous orthopedic impairments who received outpatient rehabilitation in 1 of 127 outpatient rehabilitation clinics in the United States. We identified optimal items for use in developing outpatient Adaptive Short Forms based on the Basic Mobility and Daily Activities domains of the Activity Measure for Post-Acute Care item bank. Patient scores were derived from the Activity Measure for Post-Acute Care computerized adaptive testing program. Items were selected for inclusion on the Adaptive Short Forms based on functional content, range of item coverage, measurement precision, item exposure rate, and data collection burden. RESULTS: Two outpatient Adaptive Short Forms were developed: (1) an 18-item Basic Mobility Adaptive Short Form and (2) a 15-item Daily Activities Adaptive Short Form, derived from the same item bank used to develop the Activity Measure for Post-Acute Care computerized adaptive testing program. Both Adaptive Short Forms achieved acceptable psychometric properties. CONCLUSIONS: In outpatient postacute care settings where computerized adaptive testing outcome applications are currently not feasible, item response theory-derived Adaptive Short Forms provide the efficient capability to monitor patients' functional outcomes. The development of Adaptive Short Form functional outcome instruments linked by a common, calibrated item bank has the potential to create a bridge to outcome monitoring across postacute care settings and can facilitate the eventual transformation from Adaptive Short Forms to computerized adaptive testing applications easier and more acceptable to the rehabilitation community.
Jette, Alan MHaley, Stephen MNi, PengshengMoed, RichardK02 HD45354-01/HD/NICHD NIH HHS/United StatesR01 HD43568/HD/NICHD NIH HHS/United StatesResearch Support, N.I.H., ExtramuralResearch Support, U.S. Gov't, Non-P.H.S.Research Support, U.S. Gov't, P.H.S.United StatesAmerican journal of physical medicine & rehabilitation / Association of Academic PhysiatristsAm J Phys Med Rehabil. 2008 Oct;87(10):842-52.
Keywords: *Activities of Daily Living; *Ambulatory Care Facilities; *Mobility Limitation; *Treatment Outcome; Disabled Persons/psychology/*rehabilitation; Female; Humans; Male; Middle Aged; Questionnaires; Rehabilitation Centers

Are we ready for computerized adaptive testing? - Unick, G. J.; Shumway, M.; Hargreaves, W. (Psychiatric Services, 2008)
Unick, George JShumway, MarthaHargreaves, WilliamCommentUnited StatesPsychiatric services (Washington, D.C.)Psychiatr Serv. 2008 Apr;59(4):369.
Keywords: *Attitude of Health Personnel; *Diagnosis, Computer-Assisted/instrumentation; Humans; Mental Disorders/*diagnosis; Software

Assessing self-care and social function using a computer adaptive testing version of the pediatric evaluation of disability inventory - Coster, W. J.; Haley, S. M.; Ni, P.; Dumas, H. M.; Fragala-Pinkham, M. A. (Archives of Physical Medicine and Rehabilitation, 2008)
OBJECTIVE: To examine score agreement, validity, precision, and response burden of a prototype computer adaptive testing (CAT) version of the self-care and social function scales of the Pediatric Evaluation of Disability Inventory compared with the full-length version of these scales. DESIGN: Computer simulation analysis of cross-sectional and longitudinal retrospective data; cross-sectional prospective study. SETTING: Pediatric rehabilitation hospital, including inpatient acute rehabilitation, day school program, outpatient clinics; community-based day care, preschool, and children's homes. PARTICIPANTS: Children with disabilities (n=469) and 412 children with no disabilities (analytic sample); 38 children with disabilities and 35 children without disabilities (cross-validation sample). INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Summary scores from prototype CAT applications of each scale using 15-, 10-, and 5-item stopping rules; scores from the full-length self-care and social function scales; time (in seconds) to complete assessments and respondent ratings of burden. RESULTS: Scores from both computer simulations and field administration of the prototype CATs were highly consistent with scores from full-length administration (r range, .94-.99). Using computer simulation of retrospective data, discriminant validity, and sensitivity to change of the CATs closely approximated that of the full-length scales, especially when the 15- and 10-item stopping rules were applied. In the cross-validation study the time to administer both CATs was 4 minutes, compared with over 16 minutes to complete the full-length scales. CONCLUSIONS: Self-care and social function score estimates from CAT administration are highly comparable with those obtained from full-length scale administration, with small losses in validity and precision and substantial decreases in administration time.
Coster, Wendy JHaley, Stephen MNi, PengshengDumas, Helene MFragala-Pinkham, Maria AK02 HD45354-01A1/HD/NICHD NIH HHS/United StatesR41 HD052318-01A1/HD/NICHD NIH HHS/United StatesR43 HD42388-01/HD/NICHD NIH HHS/United StatesComparative StudyResearch Support, N.I.H., ExtramuralUnited StatesArchives of physical medicine and rehabilitationArch Phys Med Rehabil. 2008 Apr;89(4):622-9.
Keywords: *Disability Evaluation; *Social Adjustment; Activities of Daily Living; Adolescent; Age Factors; Child; Child, Preschool; Computer Simulation; Cross-Over Studies; Disabled Children/*rehabilitation; Female; Follow-Up Studies; Humans; Infant; Male; Outcome Assessment (Health Care); Reference Values; Reproducibility of Results; Retrospective Studies; Risk Factors; Self Care/*standards/trends; Sex Factors; Sickness Impact Profile

An Automated Decision System for Computer Adaptive Testing Using Genetic Algorithms - Phankokkruad, M.; Woraratpanya, K. (Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing, 2008. SNPD'08. Ninth ACIS International Conference on, 2008)
This paper proposes an approach to solve the triangle decision tree problem for computer adaptive testing (CAT) using genetic algorithms (GAs). In this approach, item response theory (IRT) parameters composed of discrimination, difficulty, and guess are firstly obtained and stored in an item bank. Then a fitness function, which is based on IRT parameters, of GAs for obtaining an optimal solution is set up. Finally, the GAs is applied to the parameters of the item bank so that an optimal decision tree is generated. Based on a six-level triangle-decision tree for examination items, the experimental results show that the optimal decision tree can be generated correctly when compared with the standard patterns.

Binary items and beyond: a simulation of computer adaptive testing using the Rasch partial credit model - Lange, R. (Journal of Applied Measurement, 2008)
Past research on Computer Adaptive Testing (CAT) has focused almost exclusively on the use of binary items and minimizing the number of items to be administrated. To address this situation, extensive computer simulations were performed using partial credit items with two, three, four, and five response categories. Other variables manipulated include the number of available items, the number of respondents used to calibrate the items, and various manipulations of respondents' true locations. Three item selection strategies were used, and the theoretically optimal Maximum Information method was compared to random item selection and Bayesian Maximum Falsification approaches. The Rasch partial credit model proved to be quite robust to various imperfections, and systematic distortions did occur mainly in the absence of sufficient numbers of items located near the trait or performance levels of interest. The findings further indicate that having small numbers of items is more problematic in practice than having small numbers of respondents to calibrate these items. Most importantly, increasing the number of response categories consistently improved CAT's efficiency as well as the general quality of the results. In fact, increasing the number of response categories proved to have a greater positive impact than did the choice of item selection method, as the Maximum Information approach performed only slightly better than the Maximum Falsification approach. Accordingly, issues related to the efficiency of item selection methods are far less important than is commonly suggested in the literature. However, being based on computer simulations only, the preceding presumes that actual respondents behave according to the Rasch model. CAT research could thus benefit from empirical studies aimed at determining whether, and if so, how, selection strategies impact performance.
Lange, RenseUnited StatesJournal of applied measurementJ Appl Meas. 2008;9(1):81-104.
Keywords: *Data Interpretation, Statistical; *User-Computer Interface; Educational Measurement/*statistics & numerical data; Humans; Illinois; Models, Statistical

CAT-MD: Computerized adaptive testing on mobile devices - Triantafillou, E.; Georgiadou, E.; Economides, A. A. (International Journal of Web-Based Learning and Teaching Technologies, 2008)

Combining computer adaptive testing technology with cognitively diagnostic assessment - McGlohen, M.; Chang, Hua-Hua (Behavioral Research Methods, 2008)
A major advantage of computerized adaptive testing (CAT) is that it allows the test to home in on an examinee's ability level in an interactive manner. The aim of the new area of cognitive diagnosis is to provide information about specific content areas in which an examinee needs help. The goal of this study was to combine the benefit of specific feedback from cognitively diagnostic assessment with the advantages of CAT. In this study, three approaches to combining these were investigated: (1) item selection based on the traditional ability level estimate (theta), (2) item selection based on the attribute mastery feedback provided by cognitively diagnostic assessment (alpha), and (3) item selection based on both the traditional ability level estimate (theta) and the attribute mastery feedback provided by cognitively diagnostic assessment (alpha). The results from these three approaches were compared for theta estimation accuracy, attribute mastery estimation accuracy, and item exposure control. The theta- and alpha-based condition outperformed the alpha-based condition regarding theta estimation, attribute mastery pattern estimation, and item exposure control. Both the theta-based condition and the theta- and alpha-based condition performed similarly with regard to theta estimation, attribute mastery estimation, and item exposure control, but the theta- and alpha-based condition has an additional advantage in that it uses the shadow test method, which allows the administrator to incorporate additional constraints in the item selection process, such as content balancing, item type constraints, and so forth, and also to select items on the basis of both the current theta and alpha estimates, which can be built on top of existing 3PL testing programs.
McGlohen, MeghanChang, Hua-HuaUnited StatesBehavior research methodsBehav Res Methods. 2008 Aug;40(3):808-21.
Keywords: *Cognition; *Computers; *Models, Statistical; *User-Computer Interface; Diagnosis, Computer-Assisted/*instrumentation; Humans

Comparability of Computer-Based and Paper-and-Pencil Testing in K–12 Reading Assessments - Shudong Wang,; Hong Jiao,; Young, Michael J.; Brooks, Thomas; Olson, John (Educational and Psychological Measurement, 2008)
In recent years, computer-based testing (CBT) has grown in popularity, is increasingly being implemented across the United States, and will likely become the primary mode for delivering tests in the future. Although CBT offers many advantages over traditional paper-and-pencil testing, assessment experts, researchers, practitioners, and users have expressed concern about the comparability of scores between the two test administration modes. To help provide an answer to this issue, a meta-analysis was conducted to synthesize the administration mode effects of CBTs and paper-and-pencil tests on K—12 student reading assessments. Findings indicate that the administration mode had no statistically significant effect on K—12 student reading achievement scores. Four moderator variables—study design, sample size, computer delivery algorithm, and computer practice—made statistically significant contributions to predicting effect size. Three moderator variables—grade level, type of test, and computer delivery method—did not affect the differences in reading scores between test modes.

Computer Adaptive-Attribute Testing A New Approach to Cognitive Diagnostic Assessment - Gierl, M. J.; Zhou, J. (Zeitschrift für Psychologie / Journal of Psychology, 2008)
The influence of interdisciplinary forces stemming from developments in cognitive science,mathematical statistics, educational psychology, and computing science are beginning to appear in educational and psychological assessment. Computer adaptive-attribute testing (CA-AT) is one example. The concepts and procedures in CA-AT can be found at the intersection between computer adaptive testing and cognitive diagnostic assessment. CA-AT allows us to fuse the administrative benefits of computer adaptive testing with the psychological benefits of cognitive diagnostic assessment to produce an innovative psychologically-based adaptive testing approach. We describe the concepts behind CA-AT as well as illustrate how it can be used to promote formative, computer-based, classroom assessment.
Keywords: cognition and assessment; cognitive diagnostic assessment; computer adaptive testing

Computer-Based and Paper-and-Pencil Administration Mode Effects on a Statewide End-of-Course English Test - Kim, Do-Hong; Huynh, Huynh (Educational and Psychological Measurement, 2008)
The current study compared student performance between paper-and-pencil testing (PPT) and computer-based testing (CBT) on a large-scale statewide end-of-course English examination. Analyses were conducted at both the item and test levels. The overall results suggest that scores obtained from PPT and CBT were comparable. However, at the content domain level, a rather large difference in the reading comprehension section suggests that reading comprehension test may be more affected by the test administration mode. Results from the confirmatory factor analysis suggest that the administration mode did not alter the construct of the test.

Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes - Haley, S. M.; Gandek, B.; Siebens, H.; Black-Schaffer, R. M.; Sinclair, S. J.; Tao, W.; Coster, W. J.; Ni, P.; Jette, A. M. (Archives of Physical Medicine and Rehabilitation, 2008)
OBJECTIVES: To measure participation outcomes with a computerized adaptive test (CAT) and compare CAT and traditional fixed-length surveys in terms of score agreement, respondent burden, discriminant validity, and responsiveness. DESIGN: Longitudinal, prospective cohort study of patients interviewed approximately 2 weeks after discharge from inpatient rehabilitation and 3 months later. SETTING: Follow-up interviews conducted in patient's home setting. PARTICIPANTS: Adults (N=94) with diagnoses of neurologic, orthopedic, or medically complex conditions. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Participation domains of mobility, domestic life, and community, social, & civic life, measured using a CAT version of the Participation Measure for Postacute Care (PM-PAC-CAT) and a 53-item fixed-length survey (PM-PAC-53). RESULTS: The PM-PAC-CAT showed substantial agreement with PM-PAC-53 scores (intraclass correlation coefficient, model 3,1, .71-.81). On average, the PM-PAC-CAT was completed in 42% of the time and with only 48% of the items as compared with the PM-PAC-53. Both formats discriminated across functional severity groups. The PM-PAC-CAT had modest reductions in sensitivity and responsiveness to patient-reported change over a 3-month interval as compared with the PM-PAC-53. CONCLUSIONS: Although continued evaluation is warranted, accurate estimates of participation status and responsiveness to change for group-level analyses can be obtained from CAT administrations, with a sizeable reduction in respondent burden.
Haley, Stephen MGandek, BarbaraSiebens, HilaryBlack-Schaffer, Randie MSinclair, Samuel JTao, WeiCoster, Wendy JNi, PengshengJette, Alan MK02 HD045354-01A1/HD/NICHD NIH HHS/United StatesK02 HD45354-01/HD/NICHD NIH HHS/United StatesR01 HD043568/HD/NICHD NIH HHS/United StatesR01 HD043568-01/HD/NICHD NIH HHS/United StatesResearch Support, N.I.H., ExtramuralUnited StatesArchives of physical medicine and rehabilitationArch Phys Med Rehabil. 2008 Feb;89(2):275-83.
Keywords: *Activities of Daily Living; *Adaptation, Physiological; *Computer Systems; *Questionnaires; Adult; Aged; Aged, 80 and over; Chi-Square Distribution; Factor Analysis, Statistical; Female; Humans; Longitudinal Studies; Male; Middle Aged; Outcome Assessment (Health Care)/*methods; Patient Discharge; Prospective Studies; Rehabilitation/*standards; Subacute Care/*standards

Computerized adaptive testing for patients with knee inpairments produced valid and responsive measures of function - Hart, D. L.; Wang, Y-C.; Stratford, P. W.; Mioduski, J. E. (Journal of Clinical Epidemiology, 2008)

Computerized adaptive testing in back pain: Validation of the CAT-5D-QOL - Kopec, J. A.; Badii, M.; McKenna, M.; Lima, V. D.; Sayre, E. C.; Dvorak, M. (Spine, 2008)
STUDY DESIGN: We have conducted an outcome instrument validation study. OBJECTIVE: Our objective was to develop a computerized adaptive test (CAT) to measure 5 domains of health-related quality of life (HRQL) and assess its feasibility, reliability, validity, and efficiency. SUMMARY OF BACKGROUND DATA: Kopec and colleagues have recently developed item response theory based item banks for 5 domains of HRQL relevant to back pain and suitable for CAT applications. The domains are Daily Activities (DAILY), Walking (WALK), Handling Objects (HAND), Pain or Discomfort (PAIN), and Feelings (FEEL). METHODS: An adaptive algorithm was implemented in a web-based questionnaire administration system. The questionnaire included CAT-5D-QOL (5 scales), Modified Oswestry Disability Index (MODI), Roland-Morris Disability Questionnaire (RMDQ), SF-36 Health Survey, and standard clinical and demographic information. Participants were outpatients treated for mechanical back pain at a referral center in Vancouver, Canada. RESULTS: A total of 215 patients completed the questionnaire and 84 completed a retest. On average, patients answered 5.2 items per CAT-5D-QOL scale. Reliability ranged from 0.83 (FEEL) to 0.92 (PAIN) and was 0.92 for the MODI, RMDQ, and Physical Component Summary (PCS-36). The ceiling effect was 0.5% for PAIN compared with 2% for MODI and 5% for RMQ. The CAT-5D-QOL scales correlated as anticipated with other measures of HRQL and discriminated well according to the level of satisfaction with current symptoms, duration of the last episode, sciatica, and disability compensation. The average relative discrimination index was 0.87 for PAIN, 0.67 for DAILY and 0.62 for WALK, compared with 0.89 for MODI, 0.80 for RMDQ, and 0.59 for PCS-36. CONCLUSION: The CAT-5D-QOL is feasible, reliable, valid, and efficient in patients with back pain. This methodology can be recommended for use in back pain research and should improve outcome assessment, facilitate comparisons across studies, and reduce patient burden.
Kopec, Jacek ABadii, MaziarMcKenna, MarioLima, Viviane DSayre, Eric CDvorak, MarcelResearch Support, Non-U.S. Gov'tValidation StudiesUnited StatesSpineSpine (Phila Pa 1976). 2008 May 20;33(12):1384-90.
Keywords: *Disability Evaluation; *Health Status Indicators; *Quality of Life; Adult; Aged; Algorithms; Back Pain/*diagnosis/psychology; British Columbia; Diagnosis, Computer-Assisted/*standards; Feasibility Studies; Female; Humans; Internet; Male; Middle Aged; Predictive Value of Tests; Questionnaires/*standards; Reproducibility of Results

Computerized Adaptive Testing of Personality Traits - Hol, A. M.; Vorst, H. C. M.; Mellenbergh, G. J. (Zeitschrift für Psychologie / Journal of Psychology, 2008)
A computerized adaptive testing (CAT) procedure was simulated with ordinal polytomous personality data collected using a conventional paper-and-pencil testing format. An adapted Dutch version of the dominance scale of Gough and Heilbrun’s Adjective Check List (ACL) was used. This version contained Likert response scales with five categories. Item parameters were estimated using Samejima’s graded response model from the responses of 1,925 subjects. The CAT procedure was simulated using the responses of 1,517 other subjects. The value of the required standard error in the stopping rule of the CAT was manipulated. The relationship between CAT latent trait estimates and estimates based on all dominance items was studied. Additionally, the pattern of relationships between the CAT latent trait estimates and the other ACL scales was compared to that between latent trait estimates based on the entire item pool and the other ACL scales. The CAT procedure resulted in latent trait estimates qualitatively equivalent to latent trait estimates based on all items, while a substantial reduction of the number of used items could be realized (at the stopping rule of 0.4 about 33% of the 36 items was used).
Keywords: Adaptive Testing; cmoputer-assisted testing; Item Response Theory; Likert scales; Personality Measures

Controlling item exposure and test overlap on the fly in computerized adaptive testing - Chen, S-Y.; Lei, P. W.; Liao, W. H. (British Journal of Mathematical and Statistical Psychology, 2008)
This paper proposes an on-line version of the Sympson and Hetter procedure with test overlap control (SHT) that can provide item exposure control at both the item and test levels on the fly without iterative simulations. The on-line procedure is similar to the SHT procedure in that exposure parameters are used for simultaneous control of item exposure rates and test overlap rate. The exposure parameters for the on-line procedure, however, are updated sequentially on the fly, rather than through iterative simulations conducted prior to operational computerized adaptive tests (CATs). Unlike the SHT procedure, the on-line version can control item exposure rate and test overlap rate without time-consuming iterative simulations even when item pools or examinee populations have been changed. Moreover, the on-line procedure was found to perform better than the SHT procedure in controlling item exposure and test overlap for examinees who take tests earlier. Compared with two other on-line alternatives, this proposed on-line method provided the best all-around test security control. Thus, it would be an efficient procedure for controlling item exposure and test overlap in CATs.
Chen, Shu-YingLei, Pui-WaLiao, Wen-HanResearch Support, Non-U.S. Gov'tEnglandThe British journal of mathematical and statistical psychologyBr J Math Stat Psychol. 2008 Nov;61(Pt 2):471-92. Epub 2007 Jul 23.
Keywords: *Decision Making, Computer-Assisted; *Models, Psychological; Humans

Developing a progressive approach to using the GAIN in order to reduce the duration and cost of assessment with the GAIN short screener, Quick and computer adaptive testing - Dennis, M. L.; Funk, R.; Titus, J.; Riley, B. B.; Hosman, S.; Kinne, S. (Joint Meeting on Adolescent Treatment Effectiveness, 2008)
ProCite field[6]: Paper presented at the

The D-optimality item selection criterion in the early stage of CAT: A study with the graded response model - Passos, V. L.; Berger, M. P. F.; Tan, F. E. S. (Journal of Educational and Behavioral Statistics, 2008)
During the early stage of computerized adaptive testing (CAT), item selection criteria based on Fisher’s information often produce less stable latent trait estimates than the Kullback-Leibler global information criterion. Robustness against early stage instability has been reported for the D-optimality criterion in a polytomous CAT with the Nominal Response Model and is shown herein to be reproducible for the Graded Response Model. For comparative purposes, the A-optimality and the global information criteria are also applied. Their item selection is investigated as a function of test progression and item bank composition. The results indicate how the selection of specific item parameters underlies the criteria performances evaluated via accuracy and precision of estimation. In addition, the criteria item exposure rates are compared, without the use of any exposure controlling measure. On the account of stability, precision, accuracy, numerical simplicity, and less evidently, item exposure rate, the D-optimality criterion can be recommended for CAT.
Keywords: computerized adaptive testing; D optimality; item selection

Effect of early misfit in computerized adaptive testing on the recovery of theta - Guyer, R. D. (2008)
{PDF file, 1,004 KB}

Efficiency and sensitivity of multidimensional computerized adaptive testing of pediatric physical functioning - Allen, D. D.; Ni, P.; Haley, S. M. (Disability & Rehabilitation, 2008)
PURPOSE: Computerized adaptive tests (CATs) have efficiency advantages over fixed-length tests of physical functioning but may lose sensitivity when administering extremely low numbers of items. Multidimensional CATs may efficiently improve sensitivity by capitalizing on correlations between functional domains. Using a series of empirical simulations, we assessed the efficiency and sensitivity of multidimensional CATs compared to a longer fixed-length test. METHOD: Parent responses to the Pediatric Evaluation of Disability Inventory before and after intervention for 239 children at a pediatric rehabilitation hospital provided the data for this retrospective study. Reliability, effect size, and standardized response mean were compared between full-length self-care and mobility subscales and simulated multidimensional CATs with stopping rules at 40, 30, 20, and 10 items. RESULTS: Reliability was lowest in the 10-item CAT condition for the self-care (r = 0.85) and mobility (r = 0.79) subscales; all other conditions had high reliabilities (r > 0.94). All multidimensional CAT conditions had equivalent levels of sensitivity compared to the full set condition for both domains. CONCLUSIONS: Multidimensional CATs efficiently retain the sensitivity of longer fixed-length measures even with 5 items per dimension (10-item CAT condition). Measuring physical functioning with multidimensional CATs could enhance sensitivity following intervention while minimizing response burden.
Allen, Diane DNi, PengshengHaley, Stephen MK02 HD45354-01/HD/NICHD NIH HHS/United StatesNIDDR H133P0001/DD/NCBDD CDC HHS/United StatesResearch Support, N.I.H., ExtramuralEnglandDisability and rehabilitationDisabil Rehabil. 2008;30(6):479-84.
Keywords: *Disability Evaluation; Child; Computers; Disabled Children/*classification/rehabilitation; Efficiency; Humans; Outcome Assessment (Health Care); Psychometrics; Reproducibility of Results; Retrospective Studies; Self Care; Sensitivity and Specificity

Functioning and validity of a computerized adaptive test to measure anxiety (A CAT) - Becker, J.; Fliege, H.; Kocalevent, R. D.; Bjorner, J. B.; Rose, M.; Walter, O. B.; Klapp, B. F. (Depression and Anxiety, 2008)
Background: The aim of this study was to evaluate the Computerized Adaptive Test to measure anxiety (A-CAT), a patient-reported outcome questionnaire that uses computerized adaptive testing to measure anxiety. Methods: The A-CAT builds on an item bank of 50 items that has been built using conventional item analyses and item response theory analyses. The A-CAT was administered on Personal Digital Assistants to n=357 patients diagnosed and treated at the department of Psychosomatic Medicine and Psychotherapy, Charité Berlin, Germany. For validation purposes, two subgroups of patients (n=110 and 125) answered the A-CAT along with established anxiety and depression questionnaires. Results: The A-CAT was fast to complete (on average in 2 min, 38 s) and a precise item response theory based CAT score (reliability>.9) could be estimated after 4–41 items. On average, the CAT displayed 6 items (SD=4.2). Convergent validity of the A-CAT was supported by correlations to existing tools (Hospital Anxiety and Depression Scale-A, Beck Anxiety Inventory, Berliner Stimmungs-Fragebogen A/D, and State Trait Anxiety Inventory: r=.56–.66); discriminant validity between diagnostic groups was higher for the A-CAT than for other anxiety measures. Conclusions: The German A-CAT is an efficient, reliable, and valid tool for assessing anxiety in patients suffering from anxiety disorders and other conditions with significant potential for initial assessment and long-term treatment monitoring. Future research directions are to explore content balancing of the item selection algorithm of the CAT, to norm the tool to a healthy sample, and to develop practical cutoff scores. Depression and Anxiety, 2008. © 2008 Wiley-Liss, Inc.

ICAT: An adaptive testing procedure for the identification of idiosyncratic knowledge patterns - Kingsbury, G. G.; Houser, R.L. (Zeitschrift für Psychologie / Journal of Psychology, 2008)

ICAT: An adaptive testing procedure for the identification of idiosyncratic knowledge patterns - Kingsbury, G. G.; Houser, R.L. (Zeitschrift für Psychologie, 2008)
Traditional adaptive tests provide an efficient method for estimating student achievements levels, by adjusting the characteristicsof the test questions to match the performance of each student. These traditional adaptive tests are not designed to identify diosyncraticknowledge patterns. As students move through their education, they learn content in any number of different ways related to their learning style and cognitive development. This may result in a student having different achievement levels from one content area to another within a domain of content. This study investigates whether such idiosyncratic knowledge patterns exist. It discusses the differences between idiosyncratic knowledge patterns and multidimensionality. Finally, it proposes an adaptive testing procedure that can be used to identify a student’s areas of strength and weakness more efficiently than current adaptive testing approaches. The findings of the study indicate that a fairly large number of students may have test results that are influenced by their idiosyncratic knowledge patterns. The findings suggest that these patterns persist across time for a large number of students, and that the differences in student performance between content areas within a subject domain are large enough to allow them to be useful in instruction. Given the existence of idiosyncratic patterns of knowledge, the proposed testing procedure may enable us to provide more useful information to teachers. It should also allow us to differentiate between idiosyncratic patterns or knowledge, and important mutidimensionality in the testing data.
Keywords: computerized adaptive testing

Impact of altering randomization intervals on precision of measurement and item exposure - Muckle, T. J.; Bergstrom, B. A.; Becker, K.; Stahl, J. A. (Journal of Applied Measurement, 2008)
This paper reports on the use of simulation when a randomization procedure is used to control item exposure in a computerized adaptive test for certification. We present a method to determine the optimum width of the interval from which items are selected and we report on the impact of relaxing the interval width on measurement precision and item exposure. Results indicate that, if the item bank is well targeted, it may be possible to widen the randomization interval and thus reduce item exposure, without seriously impacting the error of measure for test takers whose ability estimate is near the pass point.

Implementing Sympson-Hetter Item-Exposure Control in a Shadow-Test Approach to Constrained Adaptive Testing - Veldkamp, Bernard P.; van der Linden, Wim J. (International Journal of Testing, 2008)

Incorporating randomness in the Fisher information for improving item-exposure control in CATs - Barrada, J; Olea, J.; Ponsoda, V.; Abad, F. J. (British Journal of Mathematical and Statistical Psychology, 2008)

An initial application of computerized adaptive testing (CAT) for measuring disability in patients with low back pain - Elhan, A. H.; Oztuna, D.; Kutlay, S.; Kucukdeveci, A. A.; Tennant, A. (BMC Musculoskelet Disorders, 2008)
ABSTRACT: BACKGROUND: Recent approaches to outcome measurement involving Computerized Adaptive Testing (CAT) offer an approach for measuring disability in low back pain (LBP) in a way that can reduce the burden upon patient and professional. The aim of this study was to explore the potential of CAT in LBP for measuring disability as defined in the International Classification of Functioning, Disability and Health (ICF) which includes impairments, activity limitation, and participation restriction. METHODS: 266 patients with low back pain answered questions from a range of widely used questionnaires. An exploratory factor analysis (EFA) was used to identify disability dimensions which were then subjected to Rasch analysis. Reliability was tested by internal consistency and person separation index (PSI). Discriminant validity of disability levels were evaluated by Spearman correlation coefficient (r), intraclass correlation coefficient [ICC(2,1)] and the Bland-Altman approach. A CAT was developed for each dimension, and the results checked against simulated and real applications from a further 133 patients. RESULTS: Factor analytic techniques identified two dimensions named "body functions" and "activity-participation". After deletion of some items for failure to fit the Rasch model, the remaining items were mostly free of Differential Item Functioning (DIF) for age and gender. Reliability exceeded 0.90 for both dimensions. The disability levels generated using all items and those obtained from the real CAT application were highly correlated (i.e. >0.97 for both dimensions). On average, 19 and 14 items were needed to estimate the precise disability levels using the initial CAT for the first and second dimension. However, a marginal increase in the standard error of the estimate across successive iterations substantially reduced the number of items required to make an estimate. CONCLUSIONS: Using a combination approach of EFA and Rasch analysis this study has shown that it is possible to calibrate items onto a single metric in a way that can be used to provide the basis of a CAT application. Thus there is an opportunity to obtain a wide variety of information to evaluate the biopsychosocial model in its more complex forms, without necessarily increasing the burden of information collection for patients.
Journal articleBMC musculoskeletal disordersBMC Musculoskelet Disord. 2008 Dec 18;9(1):166.

Investigating item exposure control on the fly in computerized adaptive testing - Wu, M.-L.; Chen, S-Y. (Psychological Testing, 2008)

Item exposure control in a-stratified computerized adaptive testing - Jhu, Y.-J.,; Chen, S-Y. (Psychological Testing, 2008)

Letting the CAT out of the bag: Comparing computer adaptive tests and an 11-item short form of the Roland-Morris Disability Questionnaire - Cook, K. F.; Choi, S. W.; Crane, P. K.; Deyo, R. A.; Johnson, K. L.; Amtmann, D. (Spine, 2008)
STUDY DESIGN: A post hoc simulation of a computer adaptive administration of the items of a modified version of the Roland-Morris Disability Questionnaire. OBJECTIVE: To evaluate the effectiveness of adaptive administration of back pain-related disability items compared with a fixed 11-item short form. SUMMARY OF BACKGROUND DATA: Short form versions of the Roland-Morris Disability Questionnaire have been developed. An alternative to paper-and-pencil short forms is to administer items adaptively so that items are presented based on a person's responses to previous items. Theoretically, this allows precise estimation of back pain disability with administration of only a few items. MATERIALS AND METHODS: Data were gathered from 2 previously conducted studies of persons with back pain. An item response theory model was used to calibrate scores based on all items, items of a paper-and-pencil short form, and several computer adaptive tests (CATs). RESULTS: Correlations between each CAT condition and scores based on a 23-item version of the Roland-Morris Disability Questionnaire ranged from 0.93 to 0.98. Compared with an 11-item short form, an 11-item CAT produced scores that were significantly more highly correlated with scores based on the 23-item scale. CATs with even fewer items also produced scores that were highly correlated with scores based on all items. For example, scores from a 5-item CAT had a correlation of 0.93 with full scale scores. Seven- and 9-item CATs correlated at 0.95 and 0.97, respectively. A CAT with a standard-error-based stopping rule produced scores that correlated at 0.95 with full scale scores. CONCLUSION: A CAT-based back pain-related disability measure may be a valuable tool for use in clinical and research contexts. Use of CAT for other common measures in back pain research, such as other functional scales or measures of psychological distress, may offer similar advantages.
Cook, Karon FChoi, Seung WCrane, Paul KDeyo, Richard AJohnson, Kurt LAmtmann, Dagmar5 P60-AR48093/AR/United States NIAMS5U01AR052171-03/AR/United States NIAMSComparative StudyResearch Support, N.I.H., ExtramuralUnited StatesSpineSpine. 2008 May 20;33(12):1378-83.
Keywords: *Disability Evaluation; *Health Status Indicators; Adult; Aged; Aged, 80 and over; Back Pain/*diagnosis/psychology; Calibration; Computer Simulation; Diagnosis, Computer-Assisted/*standards; Humans; Middle Aged; Models, Psychological; Predictive Value of Tests; Questionnaires/*standards; Reproducibility of Results

Local Dependence in an Operational CAT: Diagnosis and Implications - Pommerich, Mary; Segall, Daniel O. (Journal of Educational Measurement, 2008)
The accuracy of CAT scores can be negatively affected by local dependence if the CAT utilizes parameters that are misspecified due to the presence of local dependence and/or fails to control for local dependence in responses during the administration stage. This article evaluates the existence and effect of local dependence in a test of Mathematics Knowledge. Diagnostic tools were first used to evaluate the existence of local dependence in items that were calibrated under a 3PL model. A simulation study was then used to evaluate the effect of local dependence on the precision of examinee CAT scores when the 3PL model was used for selection and scoring. The diagnostic evaluation showed strong evidence for local dependence. The simulation suggested that local dependence in parameters had a minimal effect on CAT score precision, while local dependence in responses had a substantial effect on score precision, depending on the degree of local dependence present.

Measuring physical functioning in children with spinal impairments with computerized adaptive testing - Mulcahey, M. J.; Haley, S. M.; Duffy, T.; Pengsheng, N.; Betz, R. R. (Journal of Pediatric Orthopedics, 2008)
BACKGROUND: The purpose of this study was to assess the utility of measuring current physical functioning status of children with scoliosis and kyphosis by applying computerized adaptive testing (CAT) methods. Computerized adaptive testing uses a computer interface to administer the most optimal items based on previous responses, reducing the number of items needed to obtain a scoring estimate. METHODS: This was a prospective study of 77 subjects (0.6-19.8 years) who were seen by a spine surgeon during a routine clinic visit for progress spine deformity. Using a multidimensional version of the Pediatric Evaluation of Disability Inventory CAT program (PEDI-MCAT), we evaluated content range, accuracy and efficiency, known-group validity, concurrent validity with the Pediatric Outcomes Data Collection Instrument, and test-retest reliability in a subsample (n = 16) within a 2-week interval. RESULTS: We found the PEDI-MCAT to have sufficient item coverage in both self-care and mobility content for this sample, although most patients tended to score at the higher ends of both scales. Both the accuracy of PEDI-MCAT scores as compared with a fixed format of the PEDI (r = 0.98 for both mobility and self-care) and test-retest reliability were very high [self-care: intraclass correlation (3,1) = 0.98, mobility: intraclass correlation (3,1) = 0.99]. The PEDI-MCAT took an average of 2.9 minutes for the parents to complete. The PEDI-MCAT detected expected differences between patient groups, and scores on the PEDI-MCAT correlated in expected directions with scores from the Pediatric Outcomes Data Collection Instrument domains. CONCLUSIONS: Use of the PEDI-MCAT to assess the physical functioning status, as perceived by parents of children with complex spinal impairments, seems to be feasible and achieves accurate and efficient estimates of self-care and mobility function. Additional item development will be needed at the higher functioning end of the scale to avoid ceiling effects for older children. LEVEL OF EVIDENCE: This is a level II prospective study designed to establish the utility of computer adaptive testing as an evaluation method in a busy pediatric spine practice.
Mulcahey, M JHaley, Stephen MDuffy, TheresaPengsheng, NiBetz, Randal RK02 HD045354-01A1/HD/NICHD NIH HHS/United StatesUnited StatesJournal of pediatric orthopedicsJ Pediatr Orthop. 2008 Apr-May;28(3):330-5.
Keywords: *Disability Evaluation; Adolescent; Child; Child, Preschool; Computer Simulation; Cross-Sectional Studies; Disabled Children/*rehabilitation; Female; Humans; Infant; Kyphosis/*diagnosis/rehabilitation; Male; Prospective Studies; Reproducibility of Results; Scoliosis/*diagnosis/rehabilitation

Modern sequential analysis and its application to computerized adaptive testing - Bartroff, J.; Finkelman, M.; Lai, T. L. (Psychometrika, 2008)
After a brief review of recent advances in sequential analysis involving sequential generalized likelihood ratio tests, we discuss their use in psychometric testing and extend the asymptotic optimality theory of these sequential tests to the case of sequentially generated experiments, of particular interest in computerized adaptive testing.We then show how these methods can be used to design adaptive mastery tests, which are asymptotically optimal and are also shown to provide substantial improvements over currently used sequential and fixed length tests.

A Monte Carlo Approach for Adaptive Testing With Content Constraints - Belov, Dmitry I.; Armstrong, Ronald D.; Weissman, Alexander (Applied Psychological Measurement, 2008)
This article presents a new algorithm for computerized adaptive testing (CAT) when content constraints are present. The algorithm is based on shadow CAT methodology to meet content constraints but applies Monte Carlo methods and provides the following advantages over shadow CAT: (a) lower maximum item exposure rates, (b) higher utilization of the item pool, and (c) more robust ability estimates. Computer simulations with Law School Admission Test items demonstrated that the new algorithm (a) produces similar ability estimates as shadow CAT but with half the maximum item exposure rate and 100% pool utilization and (b) produces more robust estimates when a high- (or low-) ability examinee performs poorly (or well) at the beginning of the test.

A monte carlo approach for adaptive testing with content constraints - Belov, D. I.; Armstrong, R. D.; Weissman, A. (Applied Psychological Measurement, 2008)

A Monte Carlo approach to the design, assembly, and evaluation of multistage adaptive tests - Belov, D.I., Armstrong, R.D. (Applied Psychological Measurement, 2008)

A Monte Carlo Approach to the Design, Assembly, and Evaluation of Multistage Adaptive Tests - Belov, Dmitry I.; Armstrong, Ronald D. (Applied Psychological Measurement, 2008)
This article presents an application of Monte Carlo methods for developing and assembling multistage adaptive tests (MSTs). A major advantage of the Monte Carlo assembly over other approaches (e.g., integer programming or enumerative heuristics) is that it provides a uniform sampling from all MSTs (or MST paths) available from a given item pool. The uniform sampling allows a statistically valid analysis for MST design and evaluation. Given an item pool, MST model, and content constraints for test assembly, three problems are addressed in this study. They are (a) the construction of item response theory (IRT) targets for each MST path, (b) the assembly of an MST such that each path satisfies content constraints and IRT constraints, and (c) an analysis of the pool and constraints to increase the number of nonoverlapping MSTs that can be assembled from the pool. The primary intent is to produce reliable measurements and enhance pool utilization.

The NAPLEX: evolution, purpose, scope, and educational implications - Newton, D. W.; Boyle, M.; Catizone, C. A. (American Journal of Pharmaceutical Education, 2008)
Since 2004, passing the North American Pharmacist Licensure Examination (NAPLEX) has been a requirement for earning initial pharmacy licensure in all 50 United States. The creation and evolution from 1952-2005 of the particular pharmacy competency testing areas and quantities of questions are described for the former paper-and-pencil National Association of Boards of Pharmacy Licensure Examination (NABPLEX) and the current candidate-specific computer adaptive NAPLEX pharmacy licensure examinations. A 40% increase in the weighting of NAPLEX Blueprint Area 2 in May 2005, compared to that in the preceding 1997-2005 Blueprint, has implications for candidates' NAPLEX performance and associated curricular content and instruction. New pharmacy graduates' scores on the NAPLEX are neither intended nor validated to serve as a criterion for assessing or judging the quality or effectiveness of pharmacy curricula and instruction. The newest cycle of NAPLEX Blueprint revision, a continual process to ensure representation of nationwide contemporary practice, began in early 2008. It may take up to 2 years, including surveying several thousand national pharmacists, to complete.
Newton, David WBoyle, MariaCatizone, Carmen AHistorical ArticleUnited StatesAmerican journal of pharmaceutical educationAm J Pharm Educ. 2008 Apr 15;72(2):33.
Keywords: *Educational Measurement; Education, Pharmacy/*standards; History, 20th Century; History, 21st Century; Humans; Licensure, Pharmacy/history/*legislation & jurisprudence; North America; Pharmacists/*legislation & jurisprudence; Software

Predicting item exposure parameters in computerized adaptive testing - Chen, S-Y.; Doong, S. H. (British Journal of Mathematical and Statistical Psychology, 2008)
The purpose of this study is to find a formula that describes the relationship between item exposure parameters and item parameters in computerized adaptive tests by using genetic programming (GP) - a biologically inspired artificial intelligence technique. Based on the formula, item exposure parameters for new parallel item pools can be predicted without conducting additional iterative simulations. Results show that an interesting formula between item exposure parameters and item parameters in a pool can be found by using GP. The item exposure parameters predicted based on the found formula were close to those observed from the Sympson and Hetter (1985) procedure and performed well in controlling item exposure rates. Similar results were observed for the Stocking and Lewis (1998) multinomial model for item selection and the Sympson and Hetter procedure with content balancing. The proposed GP approach has provided a knowledge-based solution for finding item exposure parameters.
Chen, Shu-YingDoong, Shing-HwangResearch Support, Non-U.S. Gov'tEnglandThe British journal of mathematical and statistical psychologyBr J Math Stat Psychol. 2008 May;61(Pt 1):75-91.
Keywords: *Algorithms; *Artificial Intelligence; Aptitude Tests/*statistics & numerical data; Diagnosis, Computer-Assisted/*statistics & numerical data; Humans; Models, Statistical; Psychometrics/statistics & numerical data; Reproducibility of Results; Software

Preparing the implementation of computerized adaptive testing for high-stakes examinations - Huh, S. (Journal of Educational Evaluation for Health Professions, 2008)
Huh, SunEditorialKorea (South)Journal of educational evaluation for health professionsJ Educ Eval Health Prof. 2008;5:1. Epub 2008 Dec 22.

Rotating item banks versus restriction of maximum exposure rates in computerized adaptive testing - Barrada, J; Olea, J.; Abad, F. J. (Spanish Journal of Psychology, 2008)
If examinees were to know, beforehand, part of the content of a computerized adaptive test, their estimated trait levels would then have a marked positive bias. One of the strategies to avoid this consists of dividing a large item bank into several sub-banks and rotating the sub-bank employed (Ariel, Veldkamp & van der Linden, 2004). This strategy permits substantial improvements in exposure control at little cost to measurement accuracy, However, we do not know whether this option provides better results than using the master bank with greater restriction in the maximum exposure rates (Sympson & Hetter, 1985). In order to investigate this issue, we worked with several simulated banks of 2100 items, comparing them, for RMSE and overlap rate, with the same banks divided in two, three... up to seven sub-banks. By means of extensive manipulation of the maximum exposure rate in each bank, we found that the option of rotating banks slightly outperformed the option of restricting maximum exposure rate of the master bank by means of the Sympson-Hetter method.
Barrada, Juan RamonOlea, JulioAbad, Francisco JoseResearch Support, Non-U.S. Gov'tSpainThe Spanish journal of psychologySpan J Psychol. 2008 Nov;11(2):618-25.
Keywords: *Character; *Databases; *Software Design; Aptitude Tests/*statistics & numerical data; Bias (Epidemiology); Computing Methodologies; Diagnosis, Computer-Assisted/*statistics & numerical data; Educational Measurement/*statistics & numerical data; Humans; Mathematical Computing; Psychometrics/statistics & numerical data

Severity of Organized Item Theft in Computerized Adaptive Testing: A Simulation Study - Qing Yi,; Jinming Zhang,; Chang, Hua-Hua (Applied Psychological Measurement, 2008)
Criteria had been proposed for assessing the severity of possible test security violations for computerized tests with high-stakes outcomes. However, these criteria resulted from theoretical derivations that assumed uniformly randomized item selection. This study investigated potential damage caused by organized item theft in computerized adaptive testing (CAT) for two realistic item selection methods, maximum item information and a-stratified with content blocking, using the randomized method as a baseline for comparison. Damage caused by organized item theft was evaluated by the number of compromised items each examinee could encounter and the impact of the compromised items on examinees' ability estimates. Severity of test security violation was assessed under self-organized and organized item theft simulation scenarios. Results indicated that though item theft could cause severe damage to CAT with either item selection method, the maximum item information method was more vulnerable to the organized item theft simulation than was the a-stratified method.

Some new developments in adaptive testing technology - van der Linden, W. J. (Zeitschrift für Psychologie, 2008)
In an ironic twist of history, modern psychological testing has returned to an adaptive format quite common when testing was not yet standardized. Important stimuli to the renewed interest in adaptive testing have been the development of item-response theory in psychometrics, which models the responses on test items using separate parameters for the items and test takers, and the use of computers in test administration, which enables us to estimate the parameter for a test taker and select the items in real time. This article reviews a selection from the latest developments in the technology of adaptive testing, such as constrained adaptive item selection, adaptive testing using rule-based item generation, multidimensional adaptive testing, adaptive use of test batteries, and the use of response times in adaptive testing.
Keywords: computerized adaptive testing

Strategies for controlling item exposure in computerized adaptive testing with the partial credit model - Davis, L. L.; Dodd, B. G. (Journal of Applied Measurement, 2008)
Exposure control research with polytomous item pools has determined that randomization procedures can be very effective for controlling test security in computerized adaptive testing (CAT). The current study investigated the performance of four procedures for controlling item exposure in a CAT under the partial credit model. In addition to a no exposure control baseline condition, the Kingsbury-Zara, modified-within-.10-logits, Sympson-Hetter, and conditional Sympson-Hetter procedures were implemented to control exposure rates. The Kingsbury-Zara and the modified-within-.10-logits procedures were implemented with 3 and 6 item candidate conditions. The results show that the Kingsbury-Zara and modified-within-.10-logits procedures with 6 item candidates performed as well as the conditional Sympson-Hetter in terms of exposure rates, overlap rates, and pool utilization. These two procedures are strongly recommended for use with partial credit CATs due to their simplicity and strength of their results.
Davis, Laurie LaughlinDodd, Barbara GUnited StatesJournal of applied measurementJ Appl Meas. 2008;9(1):1-17.
Keywords: *Algorithms; *Computers; *Educational Measurement/statistics & numerical data; Humans; Questionnaires/*standards; United States

A Strategy for Controlling Item Exposure in Multidimensional Computerized Adaptive Testing - Lee, Yi-Hsuan; Ip, Edward H.; Fuh, Cheng-Der (Educational and Psychological Measurement, 2008)
Although computerized adaptive tests have enjoyed tremendous growth, solutions for important problems remain unavailable. One problem is the control of item exposure rate. Because adaptive algorithms are designed to select optimal items, they choose items with high discriminating power. Thus, these items are selected more often than others, leading to both overexposure and underutilization of some parts of the item pool. Overused items are often compromised, creating a security problem that could threaten the validity of a test. Building on a previously proposed stratification scheme to control the exposure rate for one-dimensional tests, the authors extend their method to multidimensional tests. A strategy is proposed based on stratification in accordance with a functional of the vector of the discrimination parameter, which can be implemented with minimal computational overhead. Both theoretical and empirical validation studies are provided. Empirical results indicate significant improvement over the commonly used method of controlling exposure rate that requires only a reasonable sacrifice in efficiency.

To Weight Or Not To Weight? Balancing Influence Of Initial Items In Adaptive Testing - Chang, H.-H.; Ying, Z. (Psychometrica, 2008)
It has been widely reported that in computerized adaptive testing some examinees may get much lower scores than they would normally if an alternative paper-and-pencil version were given. The main purpose of this investigation is to quantitatively reveal the cause for the underestimation phenomenon. The logistic models, including the 1PL, 2PL, and 3PL models, are used to demonstrate our assertions. Our analytical derivation shows that, under the maximum information item selection strategy, if an examinee failed a few items at the beginning of the test, easy but more discriminating items are likely to be administered. Such items are ineffective to move the estimate close to the true theta, unless the test is sufficiently long or a variable-length test is used. Our results also indicate that a certain weighting mechanism is necessary to make the algorithm rely less on the items administered at the beginning of the test.

Transitioning from fixed-length questionnaires to computer-adaptive versions - Walter, O. B.; Holling, H. (Zeitschrift für Psychologie \ Journal of Psychology, 2008)

Using computerized adaptive testing to reduce the burden of mental health assessment - Gibbons, R. D.; Weiss, D. J.; Kupfer, D. J.; Frank, E.; Fagiolini, A.; Grochocinski, V. J.; Bhaumik, D. K.; Stover, A.; Bock, R. D.; Immekus, J. C. (Psychiatric Services, 2008)
OBJECTIVE: This study investigated the combination of item response theory and computerized adaptive testing (CAT) for psychiatric measurement as a means of reducing the burden of research and clinical assessments. METHODS: Data were from 800 participants in outpatient treatment for a mood or anxiety disorder; they completed 616 items of the 626-item Mood and Anxiety Spectrum Scales (MASS) at two times. The first administration was used to design and evaluate a CAT version of the MASS by using post hoc simulation. The second confirmed the functioning of CAT in live testing. RESULTS: Tests of competing models based on item response theory supported the scale's bifactor structure, consisting of a primary dimension and four group factors (mood, panic-agoraphobia, obsessive-compulsive, and social phobia). Both simulated and live CAT showed a 95% average reduction (585 items) in items administered (24 and 30 items, respectively) compared with administration of the full MASS. The correlation between scores on the full MASS and the CAT version was .93. For the mood disorder subscale, differences in scores between two groups of depressed patients--one with bipolar disorder and one without--on the full scale and on the CAT showed effect sizes of .63 (p<.003) and 1.19 (p<.001) standard deviation units, respectively, indicating better discriminant validity for CAT. CONCLUSIONS: Instead of using small fixed-length tests, clinicians can create item banks with a large item pool, and a small set of the items most relevant for a given individual can be administered with no loss of information, yielding a dramatic reduction in administration time and patient and clinician burden.
Gibbons, Robert DWeiss, David JKupfer, David JFrank, EllenFagiolini, AndreaGrochocinski, Victoria JBhaumik, Dulal KStover, AngelaBock, R DarrellImmekus, Jason CR01-MH-30915/MH/United States NIMHR01-MH-66302/MH/United States NIMHResearch Support, N.I.H., ExtramuralUnited StatesPsychiatric services (Washington, D.C.)Psychiatr Serv. 2008 Apr;59(4):361-8.
Keywords: *Diagnosis, Computer-Assisted; *Questionnaires; Adolescent; Adult; Aged; Agoraphobia/diagnosis; Anxiety Disorders/diagnosis; Bipolar Disorder/diagnosis; Female; Humans; Male; Mental Disorders/*diagnosis; Middle Aged; Mood Disorders/diagnosis; Obsessive-Compulsive Disorder/diagnosis; Panic Disorder/diagnosis; Phobic Disorders/diagnosis; Reproducibility of Results; Time Factors

Using item banks to construct measures of patient reported outcomes in clinical trials: investigator perceptions - Flynn, K. E.; Dombeck, C. B.; DeWitt, E. M.; Schulman, K. A.; Weinfurt, K. P. (Clinical Trials, 2008)
BACKGROUND: Item response theory (IRT) promises more sensitive and efficient measurement of patient-reported outcomes (PROs) than traditional approaches; however, the selection and use of PRO measures from IRT-based item banks differ from current methods of using PRO measures. PURPOSE: To anticipate barriers to the adoption of IRT item banks into clinical trials. METHODS: We conducted semistructured telephone or in-person interviews with 42 clinical researchers who published results from clinical trials in the Journal of the American Medical Association, the New England Journal of Medicine, or other leading clinical journals from July 2005 through May 2006. Interviews included a brief tutorial on IRT item banks. RESULTS: After the tutorial, 39 of 42 participants understood the novel products available from an IRT item bank, namely customized short forms and computerized adaptive testing. Most participants (38/42) thought that item banks could be useful in their clinical trials, but they mentioned several potential barriers to adoption, including economic and logistical constraints, concerns about whether item banks are better than current PRO measures, concerns about how to convince study personnel or statisticians to use item banks, concerns about FDA or sponsor acceptance, and the lack of availability of item banks validated in specific disease populations. LIMITATIONS: Selection bias might have led to more positive responses to the concept of item banks in clinical trials. CONCLUSIONS: Clinical investigators are open to a new method of PRO measurement offered in IRT item banks, but bank developers must address investigator and stakeholder concerns before widespread adoption can be expected.
Flynn, Kathryn EDombeck, Carrie BDeWitt, Esi MorganSchulman, Kevin AWeinfurt, Kevin P5U01AR052186/AR/NIAMS NIH HHS/United StatesResearch Support, N.I.H., ExtramuralEnglandClinical trials (London, England)Clin Trials. 2008;5(6):575-86.

Using response times for item selection in adaptive testing - van der Linden, W. J. (Journal of Educational and Behavioral Statistics, 2008)

On using stochastic curtailment to shorten the SPRT in sequential mastery testing - Finkelman, M. D. (Journal of Educational and Behavioral Statistics, 2008)

Utilizing Rasch measurement models to develop a computer adaptive self-report of walking, climbing, and running - Velozo, C. A.; Wang, Y.; Lehman, L. A.; Wang, J. H. (Disability and Rehabilitation, 2008)
Purpose.The purpose of this paper is to show how the Rasch model can be used to develop a computer adaptive self-report of walking, climbing, and running.Method.Our instrument development work on the walking/climbing/running construct of the ICF Activity Measure was used to show how to develop a computer adaptive test (CAT). Fit of the items to the Rasch model and validation of the item difficulty hierarchy was accomplished using Winsteps software. Standard error was used as a stopping rule for the CAT. Finally, person abilities were connected to items difficulties using Rasch analysis ‘maps’.Results.All but the walking one mile item fit the Rasch measurement model. A CAT was developed which selectively presented items based on the last calibrated person ability measure and was designed to stop when standard error decreased to a pre-set criterion. Finally, person ability measures were connected to the ability to perform specific walking/climbing/running activities using Rasch maps.Conclusions.Rasch measurement models can be useful in developing CAT measures for rehabilitation and disability. In addition to CATs reducing respondent burden, the connection of person measures to item difficulties may be important for the clinical interpretation of measures.Read More: http://informahealthcare.com/doi/abs/10.1080/09638280701617317

The Wald–Wolfowitz Theorem Is Violated in Sequential Mastery Testing - Finkelman, M. (Sequential Analysis, 2008)

Adaptive estimators of trait level in adaptive testing: Some proposals - Raîche, G.; Blais, J. G.; Magis, D. (2007)
{PDF file, 125 KB}

Adaptive testing with the multi-unidimensional pairwise preference model - Stark, S.; Chernyshenko, O. S. (2007)
{PDF file, 145 KB}

Applying item response theory and computer adaptive testing: The challenges for health outcomes assessment - Fayers, P. M. (Quality of Life Research, 2007)
OBJECTIVES: We review the papers presented at the NCI/DIA conference, to identify areas of controversy and uncertainty, and to highlight those aspects of item response theory (IRT) and computer adaptive testing (CAT) that require theoretical or empirical research in order to justify their application to patient reported outcomes (PROs). BACKGROUND: IRT and CAT offer exciting potential for the development of a new generation of PRO instruments. However, most of the research into these techniques has been in non-healthcare settings, notably in education. Educational tests are very different from PRO instruments, and consequently problematic issues arise when adapting IRT and CAT to healthcare research. RESULTS: Clinical scales differ appreciably from educational tests, and symptoms have characteristics distinctly different from examination questions. This affects the transferring of IRT technology. Particular areas of concern when applying IRT to PROs include inadequate software, difficulties in selecting models and communicating results, insufficient testing of local independence and other assumptions, and a need of guidelines for estimating sample size requirements. Similar concerns apply to differential item functioning (DIF), which is an important application of IRT. Multidimensional IRT is likely to be advantageous only for closely related PRO dimensions. CONCLUSIONS: Although IRT and CAT provide appreciable potential benefits, there is a need for circumspection. Not all PRO scales are necessarily appropriate targets for this methodology. Traditional psychometric methods, and especially qualitative methods, continue to have an important role alongside IRT. Research should be funded to address the specific concerns that have been identified.
Fayers, Peter MNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2007;16 Suppl 1:187-94. Epub 2007 Apr 7.

Automated Simultaneous Assembly of Multistage Testlets for a High-Stakes Licensing Examination - Breithaupt, Krista; Hare, Donovan R. (Educational and Psychological Measurement, 2007)
Many challenges exist for high-stakes testing programs offering continuous computerized administration. The automated assembly of test questions to exactly meet content and other requirements, provide uniformity, and control item exposure can be modeled and solved by mixed-integer programming (MIP) methods. A case study of the computerized licensing examination of the American Institute of Certified Public Accountants is offered as one application of MIP techniques for test assembly. The solution illustrates assembly for a computer-adaptive multistage testing design. However, the general form of the constraint-based solution can be modified to generate optimal test designs for paper-based or computerized administrations, regardless of the specific psychometric model. An extension of this methodology allows for long-term planning for the production and use of test content on the basis of an exact psychometric test designs and administration schedules.

A â€œRearrangement Procedureâ€ For Scoring Adaptive Tests with Review Options - Papanastasiou, Elena C.; Reckase, Mark D. (International Journal of Testing, 2007)

Bundle models for computerized adaptive testing in e-learning assessment - Scalise, K.; Wilson, M. (2007)
{PDF file, 426 KB}

CAT Security: A practitioner’s perspective - Guo, F. (2007)
{PDF file, 104 KB}

Choices in CAT models in the context of educational testing - Theo Eggen (2007)
{PDF file, 123 KB}

Choices in CAT models in the context of educattional testing - Theo Eggen (GMAC Conference on Computerized Adaptive Testing, 2007)

Comparison of computerized adaptive testing and classical methods for measuring individual change - Kim-Kang, G.; Weiss, D. J. (2007)
{PDF file, 347 KB}

The comparison of maximum likelihood estimation and expected a posteriori in CAT using the graded response model - Chen, S-K. (Journal of Elementary Education, 2007)

A comparison of two methods of polytomous computerized classification testing for multiple cutscores - Thompson, N. A. (2007)
{PDF file, 363 KB}

Computerized adaptive personality testing: A review and illustration with the MMPI-2 Computerized Adaptive Version - Forbey, J. D.; Ben-Porath, Y. S. (Psychological Assessment, 2007)
Computerized adaptive testing in personality assessment can improve efficiency by significantly reducing the number of items administered to answer an assessment question. Two approaches have been explored for adaptive testing in computerized personality assessment: item response theory and the countdown method. In this article, the authors review the literature on each and report the results of an investigation designed to explore the utility, in terms of item and time savings, and validity, in terms of correlations with external criterion measures, of an expanded countdown method-based research version of the Minnesota Multiphasic Personality Inventory-2 (MMPI-2), the MMPI-2 Computerized Adaptive Version (MMPI-2-CA). Participants were 433 undergraduate college students (170 men and 263 women). Results indicated considerable item savings and corresponding time savings for the adaptive testing modalities compared with a conventional computerized MMPI-2 administration. Furthermore, computerized adaptive administration yielded comparable results to computerized conventional administration of the MMPI-2 in terms of both test scores and their validity. Future directions for computerized adaptive personality testing are discussed.
Forbey, Johnathan DBen-Porath, Yossef SResearch Support, Non-U.S. Gov'tUnited StatesPsychological assessmentPsychol Assess. 2007 Mar;19(1):14-24.
Keywords: Adolescent; Adult; Diagnosis, Computer-Assisted/*statistics & numerical data; Female; Humans; Male; MMPI/*statistics & numerical data; Personality Assessment/*statistics & numerical data; Psychometrics/statistics & numerical data; Reference Values; Reproducibility of Results

Computerized adaptive testing for measuring development of young children - Jacobusse, G.; Buuren, S. (Statistics in Medicine, 2007)
Developmental indicators that are used for routine measurement in The Netherlands are usually chosen to optimally identify delayed children. Measurements on the majority of children without problems are therefore quite imprecise. This study explores the use of computerized adaptive testing (CAT) to monitor the development of young children. CAT is expected to improve the measurement precision of the instrument. We do two simulation studies - one with real data and one with simulated data - to evaluate the usefulness of CAT. It is shown that CAT selects developmental indicators that maximally match the individual child, so that all children can be measured to the same precision.
Jacobusse, GertBuuren, Stef vanEnglandStatistics in medicineStat Med. 2007 Jun 15;26(13):2629-38.
Keywords: *Child Development; *Models, Statistical; Child, Preschool; Diagnosis, Computer-Assisted/*statistics & numerical data; Humans; Netherlands

Computerized adaptive testing for polytomous motivation items: Administration mode effects and a comparison with short forms - Hol, A. M.; Vorst, H. C. M.; Mellenbergh, G. J. (Applied Psychological Measurement, 2007)
In a randomized experiment (n=515), a computerized and a computerized adaptive test (CAT) are compared. The item pool consists of 24 polytomous motivation items. Although items are carefully selected, calibration data show that Samejima's graded response model did not fit the data optimally. A simulation study is done to assess possible consequences of model misfit. CAT efficiency was studied by a systematic comparison of the CAT with two types of conventional fixed length short forms, which are created to be good CAT competitors. Results showed no essential administration mode effects. Efficiency analyses show that CAT outperformed the short forms in almost all aspects when results are aggregated along the latent trait scale. The real and the simulated data results are very similar, which indicate that the real data results are not affected by model misfit. (PsycINFO Database Record (c) 2007 APA ) (journal abstract)
10.1177/0146621606297314Journal; Peer Reviewed Journal; Journal Article
Keywords: 2220 Tests & Testing; Adaptive Testing; Attitude Measurement; computer adaptive testing; Computer Assisted Testing; items; Motivation; polytomous motivation; Statistical Validity; Test Administration; Test Forms; Test Items

Computerized Adaptive Testing for Polytomous Motivation Items: Administration Mode Effects and a Comparison With Short Forms - Hol, A. Michiel; Vorst, Harrie C. M.; Mellenbergh, Gideon J. (Applied Psychological Measurement, 2007)
In a randomized experiment (n = 515), a computerized and a computerized adaptive test (CAT) are compared. The item pool consists of 24 polytomous motivation items. Although items are carefully selected, calibration data show that Samejima's graded response model did not fit the data optimally. A simulation study is done to assess possible consequences of model misfit. CAT efficiency was studied by a systematic comparison of the CAT with two types of conventional fixed length short forms, which are created to be good CAT competitors. Results showed no essential administration mode effects. Efficiency analyses show that CAT outperformed the short forms in almost all aspects when results are aggregated along the latent trait scale. The real and the simulated data results are very similar, which indicate that the real data results are not affected by model misfit.

Computerized adaptive testing with the bifactor model - Weiss, D. J.; Gibbons, R. D. (2007)
{PDF file, 159 KB}

Computerized attribute-adaptive testing: A new computerized adaptive testing approach incorporating cognitive psychology - Zhou, J.; Gierl, M. J.; Cui, Y. (2007)
{PDF file, 296 KB}

Computerized classification testing with composite hypotheses - Thompson, N. A.; Ro, S. (2007)
{PDF file, 96 KB}

Computerized classification testing with composite hypotheses - Thompson, N. A.; Ro, S. (GMAC Conference on Computerized Adaptive Testing, 2007)
Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing. Retrieved [date] from www. psych. umn. edu/psylabs/CATCentral
Keywords: computerized adaptive testing

Computerizing Organizational Attitude Surveys - Mueller, Karsten; Liebig, Christian; Hattrup, Keith (Educational and Psychological Measurement, 2007)
Two quasi-experimental field studies were conducted to evaluate the psychometric equivalence of computerized and paper-and-pencil job satisfaction measures. The present research extends previous work in the area by providing better control of common threats to validity in quasi-experimental research on test mode effects and by evaluating a more comprehensive measurement model for job attitudes. Results of both studies demonstrated substantial equivalence of the computerized measure with the paper-and-pencil version. Implications for the practical use of computerized organizational attitude surveys are discussed.

Conditional Item-Exposure Control in Adaptive Testing Using Item-Ineligibility Probabilities - van der Linden, Wim J.; Veldkamp, Bernard P. (Journal of Educational and Behavioral Statistics, 2007)
Two conditional versions of the exposure-control method with item-ineligibility constraints for adaptive testing in van der Linden and Veldkamp (2004) are presented. The first version is for unconstrained item selection, the second for item selection with content constraints imposed by the shadow-test approach. In both versions, the exposure rates of the items are controlled using probabilities of item ineligibility given θ that adapt the exposure rates automatically to a goal value for the items in the pool. In an extensive empirical study with an adaptive version of the Law School Admission Test, the authors show how the method can be used to drive conditional exposure rates below goal values as low as 0.025. Obviously, the price to be paid for minimal exposure rates is a decrease in the accuracy of the ability estimates. This trend is illustrated with empirical data.

Cutscore location and classification accuracy in computerized classification testing - Ro, S.; Thompson, N. A. (Paper presented at the international meeting of the Psychometric Society, 2007)
{PDF file, 94 KB}

The design and evaluation of a computerized adaptive test on mobile devices - Triantafillou, E.; Georgiadou, E.; Economides, A. A. (Computers & Education, 2007)

The design of p-optimal item banks for computerized adaptive tests - Reckase, M. D. (2007)

Designing optimal item pools for computerized adaptive tests with Sympson-Hetter exposure control - Gu, L.; Reckase, M. D. (2007)
3 MB}

Designing templates based on a taxonomy of innovative items - Parshall, C. G.; Harmes, J. C. (2007)
{PDF file, 149 KB}

Detecting Differential Speededness in Multistage Testing - van der Linden, Wim J.; Breithaupt, Krista; Chuah, Siang Chee; Zhang, Yanwei (Journal of Educational Measurement, 2007)
A potential undesirable effect of multistage testing is differential speededness, which happens if some of the test takers run out of time because they receive subtests with items that are more time intensive than others. This article shows how a probabilistic response-time model can be used for estimating differences in time intensities and speed between subtests and test takers and detecting differential speededness. An empirical data set for a multistage test in the computerized CPA Exam was used to demonstrate the procedures. Although the more difficult subtests appeared to have items that were more time intensive than the easier subtests, an analysis of the residual response times did not reveal any significant differential speededness because the time limit appeared to be appropriate. In a separate analysis, within each of the subtests, we found minor but consistent patterns of residual times that are believed to be due to a warm-up effect, that is, use of more time on the initial items than they actually need.

Developing tailored instruments: item banking and computerized adaptive assessment - Bjorner, J. B.; Chang, C-H.; Thissen, D.; Reeve, B. B. (Quality of Life Research, 2007)
Item banks and Computerized Adaptive Testing (CAT) have the potential to greatly improve the assessment of health outcomes. This review describes the unique features of item banks and CAT and discusses how to develop item banks. In CAT, a computer selects the items from an item bank that are most relevant for and informative about the particular respondent; thus optimizing test relevance and precision. Item response theory (IRT) provides the foundation for selecting the items that are most informative for the particular respondent and for scoring responses on a common metric. The development of an item bank is a multi-stage process that requires a clear definition of the construct to be measured, good items, a careful psychometric analysis of the items, and a clear specification of the final CAT. The psychometric analysis needs to evaluate the assumptions of the IRT model such as unidimensionality and local independence; that the items function the same way in different subgroups of the population; and that there is an adequate fit between the data and the chosen item response models. Also, interpretation guidelines need to be established to help the clinical application of the assessment. Although medical research can draw upon expertise from educational testing in the development of item banks and CAT, the medical field also encounters unique opportunities and challenges.
Bjorner, Jakob BueChang, Chih-HungThissen, DavidReeve, Bryce B1R43NS047763-01/NS/United States NINDSAG015815/AG/United States NIAResearch Support, N.I.H., ExtramuralNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2007;16 Suppl 1:95-108. Epub 2007 Feb 15.
Keywords: *Health Status; *Health Status Indicators; *Mental Health; *Outcome Assessment (Health Care); *Quality of Life; *Questionnaires; *Software; Algorithms; Factor Analysis, Statistical; Humans; Models, Statistical; Psychometrics

Development and evaluation of a computer adaptive test for “Anxiety” (Anxiety-CAT) - Walter, O. B.; Becker, J.; Bjorner, J. B.; Fliege, H.; Klapp, B. F.; Rose, M. (Quality of Life Research, 2007)

The development of a computerized adaptive test for integrity - Egberink, I. J. L.; Veldkamp, B. P. (2007)
{PDf file, 290 KB}

Development of a multiple-component CAT for measuring foreign language proficiency (SIMTEST) - Sumbling, M.; Sanz, P.; Viladrich, M. C.; Doval, E.; Riera, L. (2007)
{PDF file, 258 KB}

The effect of including pretest items in an operational computerized adaptive test: Do different ability examinees spend different amounts of time on embedded pretest items? - Ferdous, A. A.; Plake, B. S.; Chang, S-R. (Educational Assessment, 2007)
The purpose of this study was to examine the effect of pretest items on response time in an operational, fixed-length, time-limited computerized adaptive test (CAT). These pretest items are embedded within the CAT, but unlike the operational items, are not tailored to the examinee's ability level. If examinees with higher ability levels need less time to complete these items than do their counterparts with lower ability levels, they will have more time to devote to the operational test questions. Data were from a graduate admissions test that was administered worldwide. Data from both quantitative and verbal sections of the test were considered. For the verbal section, examinees in the lower ability groups spent systematically more time on their pretest items than did those in the higher ability groups, though for the quantitative section the differences were less clear. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: ability; operational computerized adaptive test; pretest items; time

The Effect of Using Item Parameters Calibrated from Paper Administrations in Computer Adaptive Test Administrations - Pommerich, M (The Journal of Technology, Learning, and Assessment, 2007)
Computer administered tests are becoming increasingly prevalent as computer technology becomes more readily available on a large scale. For testing programs that utilize both computer and paper administrations, mode effects are problematic in that they can result in examinee scores that are artificially inflated or deflated. As such, researchers have engaged in extensive studies of whether scores differ across paper and computer presentations of the same tests. The research generally seems to indicate that the more complicated it is to present or take a test on computer, the greater the possibility of mode effects. In a computer adaptive test, mode effects may be a particular concern if items are calibrated using item responses obtained from one administration mode (i.e., paper), and those parameters are then used operationally in a different administration mode (i.e., computer). This paper studies the suitability of using parameters calibrated from a paper administration for item selection and scoring in a computer adaptive administration, for two tests with lengthy passages that required navigation in the computer administration. The results showed that the use of paper calibrated parameters versus computer calibrated parameters in computer adaptive administrations had small to moderate effects on the reliability of examinee scores, at fairly short test lengths. This effect was generally diminished for longer test lengths. However, the results suggest that in some cases, some loss in reliability might be inevitable if paper-calibrated parameters are used in computer adaptive administrations.

The effect of using item parameters calibrated from paper administrations in computer adaptive test administrations - Pommerich, M (Journal of Technology, Learning, and Assessment, 2007)
Computer administered tests are becoming increasingly prevalent as computer technology becomes more readily available on a large scale. For testing programs that utilize both computer and paper administrations, mode effects are problematic in that they can result in examinee scores that are artificially inflated or deflated. As such, researchers have engaged in extensive studies of whether scores differ across paper and computer presentations of the same tests. The research generally seems to indicate that the more complicated it is to present or take a test on computer, the greater the possibility of mode effects. In a computer adaptive test, mode effects may be a particular concern if items are calibrated using item responses obtained from one administration mode (i.e., paper), and those parameters are then used operationally in a different administration mode (i.e., computer). This paper studies the suitability of using parameters calibrated from a paper administration for item selection and scoring in a computer adaptive administration, for two tests with lengthy passages that required navigation in the computer administration. The results showed that the use of paper calibrated parameters versus computer calibrated parameters in computer adaptive administrations had small to moderate effects on the reliability of examinee scores, at fairly short test lengths. This effect was generally diminished for longer test lengths. However, the results suggest that in some cases, some loss in reliability might be inevitable if paper-calibrated parameters are used in computer adaptive administrations.
Keywords: Mode effects

Estimating the Standard Error of the Maximum Likelihood Ability Estimator in Adaptive Testing Using the Posterior-Weighted Test Information Function - Penfield, Randall D. (Educational and Psychological Measurement, 2007)
The standard error of the maximum likelihood ability estimator is commonly estimated by evaluating the test information function at an examinee's current maximum likelihood estimate (a point estimate) of ability. Because the test information function evaluated at the point estimate may differ from the test information function evaluated at an examinee's true ability value, the estimated standard error may be biased under certain conditions. This is of particular concern in adaptive testing because the height of the test information function is expected to be higher at the current estimate of ability than at the actual value of ability. This article proposes using the posterior-weighted test information function in computing the standard error of the maximum likelihood ability estimator for adaptive test sessions. A simulation study showed that the proposed approach provides standard error estimates that are less biased and more efficient than those provided by the traditional point estimate approach.

Evaluation of computer adaptive testing systems - Economides, A. A.; Roupas, C (International Journal of Web-Based Learning and Teaching Technologies, 2007)
Many educational organizations are trying to reduce the cost of the exams, the workload and delay of scoring, and the human errors. Also, they try to increase the accuracy and efficiency of the testing. Recently, most examination organizations use computer adaptive testing (CAT) as the method for large scale testing. This article investigates the current state of CAT systems and identifies their strengths and weaknesses. It evaluates 10 CAT systems using an evaluation framework of 15 domains categorized into three dimensions: educational, technical, and economical. The results show that the majority of the CAT systems give priority to security, reliability, and maintainability. However, they do not offer to the examinee any advanced support and functionalities. Also, the feedback to the examinee is limited and the presentation of the items is poor. Recommendations are made in order to enhance the overall quality of a CAT system. For example, alternative multimedia items should be available so that the examinee would choose a preferred media type. Feedback could be improved by providing more information to the examinee or providing information anytime the examinee wished. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: computer adaptive testing systems; examination organizations; systems evaluation

An exploration and realization of computerized adaptive testing with cognitive diagnosis - Haijing, L.; Shuliang, D. (Acta Psychologica Sinica, 2007)
An increased attention paid to “cognitive bugs behavior,” appears to lead to an increased research interests in diagnostic testing based on Item Response Theory（IRT）that combines cognitive psychology and psychometrics. The study of cognitive diagnosis were applied mainly to Paper-and-Pencil (P&P) testing. Rarely has it been applied to computerized adaptive testing CAT）, To our knowledge, no research on CAT with cognitive diagnosis has been conducted in China. Since CAT is more efficient and accurate than P&P testing, there is important to develop an application technique for cognitive diagnosis suitable for CAT. This study attempts to construct a preliminary CAT system for cognitive diagnosis.With the help of the methods for “ Diagnosis first, Ability estimation second ”, the knowledge state conversion diagram was used to describe all the possible knowledge states in a domain of interest and the relation among the knowledge states at the diagnosis stage, where a new strategy of item selection based-on the algorithm of Depth First Search was proposed. On the other hand, those items that contain attributes which the examinee has not mastered were removed in ability estimation. At the stage of accurate ability estimation, all the items answered by each examinee not only matched his/her ability estimated value, but also were limited to those items whose attributes have been mastered by the examinee.We used Monte Carlo Simulation to simulate all the data of the three different structures of cognitive attributes in this study. These structures were tree-shaped, forest-shaped, and some isolated vertices (that are related to simple Q-matrix). Both tree-shaped and isolated vertices structure were derived from actual cases, while forest-shaped structure was a generalized simulation. 3000 examinees and 3000 items were simulated in the experiment of tree-shaped, 2550 examinees and 3100 items in forest-shaped, and 2000 examinees and 2500 items in isolated vertices. The maximum test length was all assumed as 30 items for all those experiments. The difficulty parameters and the logarithm of the discrimination were drawn from the standard normal distribution N(0,1). There were 100 examinees of each attribute pattern in the experiment of tree-shaped and 50 examinees of each attribute pattern in forest-shaped. In isolated vertices, 2000 examinees are students come from actual case.To assess the behaviors of the proposed diagnostic approach, three assessment indices were used. They are attribute pattern classification agreement rate (abr.APCAR), the Recovery (the average of the absolute deviation between the estimated value and the true value) and the average test length (abr. Length).Parts of results of Monte Carlo study were as follows.For the attribute structure of tree-shaped, APCAR is 84.27%,Recovery is 0.17,Length is 24.80.For the attribute structure of forest-shaped, APCAR is 84.02%,Recovery is 0.172,Length is 23.47.For the attribute structure of isolated vertices, APCAR is 99.16%,Recorvery is 0.256,Length is 27.32.As show the above, we can conclude that the results are favorable. The rate of cognitive diagnosis accuracy has exceeded 80% in each experiment, and the Recovery is also good. Therefore, it should be an acceptable idea to construct an initiatory CAT system for cognitive diagnosis, if we use the methods for “Diagnosis first, Ability estimation second ” with the help of both knowledge state conversion diagram and the new strategy of item selection based-on the algorithm of Depth First Search

Exploring potential designs for multi-form structure computerized adaptive tests with uniform item exposure - Edwards, M. C.; Thissen, D. (2007)
{PDF file, 295 KB}

The future of outcomes measurement: item banking, tailored short-forms, and computerized adaptive assessment - Cella, D.; Gershon, R. C.; Lai, J-S.; Choi, S. W. (Quality of Life Research, 2007)
The use of item banks and computerized adaptive testing (CAT) begins with clear definitions of important outcomes, and references those definitions to specific questions gathered into large and well-studied pools, or “banks” of items. Items can be selected from the bank to form customized short scales, or can be administered in a sequence and length determined by a computer programmed for precision and clinical relevance. Although far from perfect, such item banks can form a common definition and understanding of human symptoms and functional problems such as fatigue, pain, depression, mobility, social function, sensory function, and many other health concepts that we can only measure by asking people directly. The support of the National Institutes of Health (NIH), as witnessed by its cooperative agreement with measurement experts through the NIH Roadmap Initiative known as PROMIS (www.nihpromis.org), is a big step in that direction. Our approach to item banking and CAT is practical; as focused on application as it is on science or theory. From a practical perspective, we frequently must decide whether to re-write and retest an item, add more items to fill gaps (often at the ceiling of the measure), re-test a bank after some modifications, or split up a bank into units that are more unidimensional, yet less clinically relevant or complete. These decisions are not easy, and yet they are rarely unforgiving. We encourage people to build practical tools that are capable of producing multiple short form measures and CAT administrations from common banks, and to further our understanding of these banks with various clinical populations and ages, so that with time the scores that emerge from these many activities begin to have not only a common metric and range, but a shared meaning and understanding across users. In this paper, we provide an overview of item banking and CAT, discuss our approach to item banking and its byproducts, describe testing options, discuss an example of CAT for fatigue, and discuss models for long term sustainability of an entity such as PROMIS. Some barriers to success include limitations in the methods themselves, controversies and disagreements across approaches, and end-user reluctance to move away from the familiar.

Hypothetischer Einsatz adaptiven Testens bei der Messung von Bildungsstandards in Mathematik [Hypothetical use of adaptive testing for the measurement of educational standards in mathematics] . - Frey, A.; Ehmke, T. (Zeitschrift für Erziehungswissenschaft, 2007)

ICAT: An adaptive testing procedure to allow the identification of idiosyncratic knowledge patterns - Kingsbury, G. G.; Houser, R.L. (2007)
{PDF file, 161 KB}

Implementing the Graduate Management Admission Test® computerized adaptive test - Rudner, L. M. (2007)
{PDF file, 135 KB}

Improving patient reported outcomes using item response theory and computerized adaptive testing - Chakravarty, E. F.; Bjorner, J. B.; Fries, J.F. (Journal of Rheumatology, 2007)
OBJECTIVE: Patient reported outcomes (PRO) are considered central outcome measures for both clinical trials and observational studies in rheumatology. More sophisticated statistical models, including item response theory (IRT) and computerized adaptive testing (CAT), will enable critical evaluation and reconstruction of currently utilized PRO instruments to improve measurement precision while reducing item burden on the individual patient. METHODS: We developed a domain hierarchy encompassing the latent trait of physical function/disability from the more general to most specific. Items collected from 165 English-language instruments were evaluated by a structured process including trained raters, modified Delphi expert consensus, and then patient evaluation. Each item in the refined data bank will undergo extensive analysis using IRT to evaluate response functions and measurement precision. CAT will allow for real-time questionnaires of potentially smaller numbers of questions tailored directly to each individual's level of physical function. RESULTS: Physical function/disability domain comprises 4 subdomains: upper extremity, trunk, lower extremity, and complex activities. Expert and patient review led to consensus favoring use of present-tense "capability" questions using a 4- or 5-item Likert response construct over past-tense "performance"items. Floor and ceiling effects, attribution of disability, and standardization of response categories were also addressed. CONCLUSION: By applying statistical techniques of IRT through use of CAT, existing PRO instruments may be improved to reduce questionnaire burden on the individual patients while increasing measurement precision that may ultimately lead to reduced sample size requirements for costly clinical trials.
Chakravarty, Eliza FBjorner, Jakob BFries, James FAr052158/ar/niamsConsensus Development ConferenceResearch Support, N.I.H., ExtramuralCanadaThe Journal of rheumatologyJ Rheumatol. 2007 Jun;34(6):1426-31.
Keywords: *Rheumatic Diseases/physiopathology/psychology; Clinical Trials; Data Interpretation, Statistical; Disability Evaluation; Health Surveys; Humans; International Cooperation; Outcome Assessment (Health Care)/*methods; Patient Participation/*methods; Research Design/*trends; Software

The initial development of an item bank to assess and screen for psychological distress in cancer patients - Smith, A. B.; Rush, R.; Velikova, G.; Wall, L.; Wright, E. P.; Stark, D.; Selby, P.; Sharpe, M. (Psycho-Oncology, 2007)
Psychological distress is a common problem among cancer patients. Despite the large number of instruments that have been developed to assess distress, their utility remains disappointing. This study aimed to use Rasch models to develop an item-bank which would provide the basis for better means of assessing psychological distress in cancer patients. An item bank was developed from eight psychological distress questionnaires using Rasch analysis to link common items. Items from the questionnaires were added iteratively with common items as anchor points and misfitting items (infit mean square > 1.3) removed, and unidimensionality assessed. A total of 4914 patients completed the questionnaires providing an initial pool of 83 items. Twenty items were removed resulting in a final pool of 63 items. Good fit was demonstrated and no additional factor structure was evident from the residuals. However, there was little overlap between item locations and person measures, since items mainly targeted higher levels of distress. The Rasch analysis allowed items to be pooled and generated a unidimensional instrument for measuring psychological distress in cancer patients. Additional items are required to more accurately assess patients across the whole continuum of psychological distress. (PsycINFO Database Record (c) 2007 APA ) (journal abstract)
10.1002/pon.1117Journal; Peer Reviewed Journal; Journal Article
Keywords: 3293 Cancer; cancer patients; Distress; initial development; Item Response Theory; Models; Neoplasms; Patients; Psychological; psychological distress; Rasch; Stress

Investigating CAT designs to achieve comparability with a paper test - Thompson, T.; Way, W. D. (2007)
{PDF file, 141 KB}

IRT health outcomes data analysis project: an overview and summary - Cook, K. F.; Teal, C. R.; Bjorner, J. B.; Cella, D.; Chang, C-H.; Crane, P. K.; Gibbons, L. E.; Hays, R. D.; McHorney, C. A.; Ocepek-Welikson, K.; Raczek, A. E.; Teresi, J. A.; et al. (Quality of Life Research, 2007)
BACKGROUND: In June 2004, the National Cancer Institute and the Drug Information Association co-sponsored the conference, "Improving the Measurement of Health Outcomes through the Applications of Item Response Theory (IRT) Modeling: Exploration of Item Banks and Computer-Adaptive Assessment." A component of the conference was presentation of a psychometric and content analysis of a secondary dataset. OBJECTIVES: A thorough psychometric and content analysis was conducted of two primary domains within a cancer health-related quality of life (HRQOL) dataset. RESEARCH DESIGN: HRQOL scales were evaluated using factor analysis for categorical data, IRT modeling, and differential item functioning analyses. In addition, computerized adaptive administration of HRQOL item banks was simulated, and various IRT models were applied and compared. SUBJECTS: The original data were collected as part of the NCI-funded Quality of Life Evaluation in Oncology (Q-Score) Project. A total of 1,714 patients with cancer or HIV/AIDS were recruited from 5 clinical sites. MEASURES: Items from 4 HRQOL instruments were evaluated: Cancer Rehabilitation Evaluation System-Short Form, European Organization for Research and Treatment of Cancer Quality of Life Questionnaire, Functional Assessment of Cancer Therapy and Medical Outcomes Study Short-Form Health Survey. RESULTS AND CONCLUSIONS: Four lessons learned from the project are discussed: the importance of good developmental item banks, the ambiguity of model fit results, the limits of our knowledge regarding the practical implications of model misfit, and the importance in the measurement of HRQOL of construct definition. With respect to these lessons, areas for future research are suggested. The feasibility of developing item banks for broad definitions of health is discussed.
Cook, Karon FTeal, Cayla RBjorner, Jakob BCella, DavidChang, Chih-HungCrane, Paul KGibbons, Laura EHays, Ron DMcHorney, Colleen AOcepek-Welikson, KatjaRaczek, Anastasia ETeresi, Jeanne AReeve, Bryce B1U01AR52171-01/AR/United States NIAMSR01 (CA60068)/CA/United States NCIY1-PC-3028-01/PC/United States NCIResearch Support, N.I.H., ExtramuralNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2007;16 Suppl 1:121-32. Epub 2007 Mar 10.
Keywords: *Data Interpretation, Statistical; *Health Status; *Quality of Life; *Questionnaires; *Software; Female; HIV Infections/psychology; Humans; Male; Neoplasms/psychology; Outcome Assessment (Health Care)/*methods; Psychometrics; Stress, Psychological

Item selection in computerized classification testing - Thompson, N. A. (Paper presented at the Conference on High Stakes Testing, 2007)
{PDF file, 87KB}

Methodological issues for building item banks and computerized adaptive scales - Thissen, D.; Reeve, B. B.; Bjorner, J. B.; Chang, C-H. (Quality of Life Research, 2007)
Abstract This paper reviews important methodological considerations for developing item banks and computerized adaptive scales (commonly called computerized adaptive tests in the educational measurement literature, yielding the acronym CAT), including issues of the reference population, dimensionality, dichotomous versus polytomous response scales, differential item functioning (DIF) and conditional scoring, mode effects, the impact of local dependence, and innovative approaches to assessment using CATs in health outcomes research.

Methods for restricting maximum exposure rate in computerized adaptative testing - Barrada, J; Olea, J.; Ponsoda, V. (Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 2007)
The Sympson-Hetter (1985) method provides a means of controlling maximum exposure rate of items in Computerized Adaptive Testing. Through a series of simulations, control parameters are set that mark the probability of administration of an item on being selected. This method presents two main problems: it requires a long computation time for calculating the parameters and the maximum exposure rate is slightly above the fixed limit. Van der Linden (2003) presented two alternatives which appear to solve both of the problems. The impact of these methods in the measurement accuracy has not been tested yet. We show how these methods over-restrict the exposure of some highly discriminating items and, thus, the accuracy is decreased. It also shown that, when the desired maximum exposure rate is near the minimum possible value, these methods offer an empirical maximum exposure rate clearly above the goal. A new method, based on the initial estimation of the probability of administration and the probability of selection of the items with the restricted method (Revuelta & Ponsoda, 1998), is presented in this paper. It can be used with the Sympson-Hetter method and with the two van der Linden's methods. This option, when used with Sympson-Hetter, speeds the convergence of the control parameters without decreasing the accuracy. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: computerized adaptive testing; item bank security; item exposure control; overlap rate; Sympson-Hetter method

The modified maximum global discrimination index method for cognitive diagnostic computerized adaptive testing - Cheng, Y; Chang, Hua-Hua (2007)
{PDF file, 172 KB}

A multiple objective test assembly approach for exposure control problems in computerized adaptive testing - Veldkamp, B. P.; Verschoor, Angela J.; Theo Eggen (Measurement and Research Department Reports, 2007)

Mutual Information Item Selection in Adaptive Classification Testing - Weissman, Alexander (Educational and Psychological Measurement, 2007)
A general approach for item selection in adaptive multiple-category classification tests is provided. The approach uses mutual information (MI), a special case of the Kullback-Leibler distance, or relative entropy. MI works efficiently with the sequential probability ratio test and alleviates the difficulties encountered with using other local- and global-information measures in the multiple-category classification setting. Results from simulation studies using three item selection methods, Fisher information (FI), posterior-weighted FI (FIP), and MI, are provided for an adaptive four-category classification test. Both across and within the four classification categories, it is shown that in general, MI item selection classifies the highest proportion of examinees correctly and yields the shortest test lengths. The next best performance is observed for FIP item selection, followed by FI.

An NCME instructional module on multistage testing - Hendrickson, A. (Educational Measurement: Issues and Practice, 2007)
#HE07044

A new delivery system for CAT - Park, J. (2007)
{PDF file, 248 KB}

Nonparametric online item calibration - Samejima, F. (2007)
8 MB}

Partial order knowledge structures for CAT applications - Desmarais, M. C.; Pu, X,; Blais, J-G. (2007)
{PDF file, 475 KB}

Patient-reported outcomes measurement and computerized adaptive testing: An application of post-hoc simulation to a diagnostic screening instrument - Immekus, J. C.; Gibbons, R. D.; Rush, J. A. (2007)
{PDF file, 203 KB}

Patient-reported outcomes measurement and management with innovative methodologies and technologies - Chang, C-H. (Quality of Life Research, 2007)
Successful integration of modern psychometrics and advanced informatics in patient-reported outcomes (PRO) measurement and management can potentially maximize the value of health outcomes research and optimize the delivery of quality patient care. Unlike the traditional labor-intensive paper-and-pencil data collection method, item response theory-based computerized adaptive testing methodologies coupled with novel technologies provide an integrated environment to collect, analyze and present ready-to-use PRO data for informed and shared decision-making. This article describes the needs, challenges and solutions for accurate, efficient and cost-effective PRO data acquisition and dissemination means in order to provide critical and timely PRO information necessary to actively support and enhance routine patient care in busy clinical settings.
Chang, Chih-HungR21CA113191/CA/NCI NIH HHS/United StatesResearch Support, N.I.H., ExtramuralNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2007;16 Suppl 1:157-66. Epub 2007 May 26.
Keywords: *Health Status; *Outcome Assessment (Health Care); *Quality of Life; *Software; Computer Systems/*trends; Health Insurance Portability and Accountability Act; Humans; Patient Satisfaction; Questionnaires; United States

The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years - Cella, D.; Yount, S.; Rothrock, N.; Gershon, R. C.; Cook, K. F.; Reeve, B.; Ader, D.; Fries, J.F.; Bruce, B.; Rose, M. (Medical Care, 2007)
The National Institutes of Health (NIH) Patient-Reported Outcomes Measurement Information System (PROMIS) Roadmap initiative (www.nihpromis.org) is a 5-year cooperative group program of research designed to develop, validate, and standardize item banks to measure patient-reported outcomes (PROs) relevant across common medical conditions. In this article, we will summarize the organization and scientific activity of the PROMIS network during its first 2 years.

A Practitioner’s Guide for Variable-length Computerized Classification Testing - Thompson, N. A. (Practical Assessment Research and Evaluation, 2007)

A practitioner's guide to variable-length computerized classification testing - Thompson, N. A. (Practical Assessment, Research and Evaluation, 2007)
Variable-length computerized classification tests, CCTs, (Lin & Spray, 2000; Thompson, 2006) are a powerful and efficient approach to testing for the purpose of classifying examinees into groups. CCTs are designed by the specification of at least five technical components: psychometric model, calibrated item bank, starting point, item selection algorithm, and termination criterion. Several options exist for each of these CCT components, creating a myriad of possible designs. Confusion among designs is exacerbated by the lack of a standardized nomenclature. This article outlines the components of a CCT, common options for each component, and the interaction of options for different components, so that practitioners may more efficiently design CCTs. It also offers a suggestion of nomenclature.
Keywords: CAT; classification; computer adaptive testing; computerized adaptive testing; Computerized classification testing

Prospective evaluation of the am-pac-cat in outpatient rehabilitation settings - Jette, A.,; Haley, S.; Tao, W.; Ni, P.; Moed, R.; Meyers, D.; Zurek, M. (Physical Therapy, 2007)

Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS) - Reeve, B. B.; Hays, R. D.; Bjorner, J. B.; Cook, K. F.; Crane, P. K.; Teresi, J. A.; Thissen, D.; Revicki, D. A.; Weiss, D. J.; Hambleton, R. K.; Liu, H.; Gershon, R. C.; et al. (Medical Care, 2007)
BACKGROUND: The construction and evaluation of item banks to measure unidimensional constructs of health-related quality of life (HRQOL) is a fundamental objective of the Patient-Reported Outcomes Measurement Information System (PROMIS) project. OBJECTIVES: Item banks will be used as the foundation for developing short-form instruments and enabling computerized adaptive testing. The PROMIS Steering Committee selected 5 HRQOL domains for initial focus: physical functioning, fatigue, pain, emotional distress, and social role participation. This report provides an overview of the methods used in the PROMIS item analyses and proposed calibration of item banks. ANALYSES: Analyses include evaluation of data quality (eg, logic and range checking, spread of response distribution within an item), descriptive statistics (eg, frequencies, means), item response theory model assumptions (unidimensionality, local independence, monotonicity), model fit, differential item functioning, and item calibration for banking. RECOMMENDATIONS: Summarized are key analytic issues; recommendations are provided for future evaluations of item banks in HRQOL assessment.
Reeve, Bryce BHays, Ron DBjorner, Jakob BCook, Karon FCrane, Paul KTeresi, Jeanne AThissen, DavidRevicki, Dennis AWeiss, David JHambleton, Ronald KLiu, HonghuGershon, RichardReise, Steven PLai, Jin-sheiCella, DavidPROMIS Cooperative GroupAG015815/AG/United States NIAResearch Support, N.I.H., ExtramuralUnited StatesMedical careMed Care. 2007 May;45(5 Suppl 1):S22-31.
Keywords: *Health Status; *Information Systems; *Quality of Life; *Self Disclosure; Adolescent; Adult; Aged; Calibration; Databases as Topic; Evaluation Studies as Topic; Female; Humans; Male; Middle Aged; Outcome Assessment (Health Care)/*methods; Psychometrics; Questionnaires/standards; United States

Psychometric properties of an emotional adjustment measure: An application of the graded response model - Rubio, V. J.; Aguado, D.; Hontangas, P. M.; Hernández, J. M. (European Journal of Psychological Assessment, 2007)
Item response theory (IRT) provides valuable methods for the analysis of the psychometric properties of a psychological measure. However, IRT has been mainly used for assessing achievements and ability rather than personality factors. This paper presents an application of the IRT to a personality measure. Thus, the psychometric properties of a new emotional adjustment measure that consists of a 28-six graded response items is shown. Classical test theory (CTT) analyses as well as IRT analyses are carried out. Samejima's (1969) graded-response model has been used for estimating item parameters. Results show that the bank of items fulfills model assumptions and fits the data reasonably well, demonstrating the suitability of the IRT models for the description and use of data originating from personality measures. In this sense, the model fulfills the expectations that IRT has undoubted advantages: (1) The invariance of the estimated parameters, (2) the treatment given to the standard error of measurement, and (3) the possibilities offered for the construction of computerized adaptive tests (CAT). The bank of items shows good reliability. It also shows convergent validity compared to the Eysenck Personality Inventory (EPQ-A; Eysenck & Eysenck, 1975) and the Big Five Questionnaire (BFQ; Caprara, Barbaranelli, & Borgogni, 1993). (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: computerized adaptive tests; Emotional Adjustment; Item Response Theory; Personality Measures; personnel recruitment; Psychometrics; Samejima's graded response model; test reliability; validity

Relative precision, efficiency and construct validity of different starting and stopping rules for a computerized adaptive test: The GAIN Substance Problem Scale - Riley, B. B.; Conrad, K. J.; Bezruczko, N.; Dennis, M. L. (Journal of Applied Measurement, 2007)
Substance abuse treatment programs are being pressed to measure and make clinical decisions more efficiently about an increasing array of problems. This computerized adaptive testing (CAT) simulation examined the relative efficiency, precision and construct validity of different starting and stopping rules used to shorten the Global Appraisal of Individual Needs’ (GAIN) Substance Problem Scale (SPS) and facilitate diagnosis based on it. Data came from 1,048 adolescents and adults referred to substance abuse treatment centers in 5 sites. CAT performance was evaluated using: (1) average standard errors, (2) average number of items, (3) bias in personmeasures, (4) root mean squared error of person measures, (5) Cohen’s kappa to evaluate CAT classification compared to clinical classification, (6) correlation between CAT and full-scale measures, and (7) construct validity of CAT classification vs. clinical classification using correlations with five theoretically associated instruments. Results supported both CAT efficiency and validity.
Keywords: My article

A review of item exposure control strategies for computerized adaptive testing developed from 1983 to 2005 - Georgiadou, E.; Triantafillou, E.; Economides, A. A. (Journal of Technology,Learning, and Assessment,, 2007)
Since researchers acknowledged the several advantages of computerized adaptive testing (CAT) over traditional linear test administration, the issue of item exposure control has received increased attention. Due to CAT’s underlying philosophy, particular items in the item pool may be presented too often and become overexposed, while other items are rarely selected by the CAT algorithm and thus become underexposed. Several item exposure control strategies have been presented in the literature aiming to prevent overexposure of some items and to increase the use rate of rarely or never selected items. This paper reviews such strategies that appeared in the relevant literature from 1983 to 2005. The focus of this paper is on studies that have been conducted in order to evaluate the effectiveness of item exposure control strategies for dichotomous scoring, polytomous scoring and testlet-based CAT systems. In addition, the paper discusses the strengths and weaknesses of each strategy group using examples from simulation studies. No new research is presented but rather a compendium of models is reviewed with an overall objective of providing researchers of this field, especially newcomers, a wide view of item exposure control strategies.
http://www.jtla.org. {PDF file, 326 KB}

The shadow-test approach: A universal framework for implementing adaptive testing - van der Linden, W. J. (2007)
{PDF file, 461 KB}

Some thoughts on controlling item exposure in adaptive testing - Lewis, C. (2007)
{PDF file, 357 KB}

Statistical aspects of adaptive testing - van der Linden, W. J.; Glas, C. A. W. (2007)

A system for interactive assessment and management in palliative care - Chang, C-H.; Boni-Saenz, A. A.; Durazo-Arvizu, R. A.; DesHarnais, S.; Lau, D. T.; Emanuel, L. L. (Journal of Pain Symptom Management, 2007)
The availability of psychometrically sound and clinically relevant screening, diagnosis, and outcome evaluation tools is essential to high-quality palliative care assessment and management. Such data will enable us to improve patient evaluations, prognoses, and treatment selections, and to increase patient satisfaction and quality of life. To accomplish these goals, medical care needs more precise, efficient, and comprehensive tools for data acquisition, analysis, interpretation, and management. We describe a system for interactive assessment and management in palliative care (SIAM-PC), which is patient centered, model driven, database derived, evidence based, and technology assisted. The SIAM-PC is designed to reliably measure the multiple dimensions of patients' needs for palliative care, and then to provide information to clinicians, patients, and the patients' families to achieve optimal patient care, while improving our capacity for doing palliative care research. This system is innovative in its application of the state-of-the-science approaches, such as item response theory and computerized adaptive testing, to many of the significant clinical problems related to palliative care.
Chang, Chih-HungBoni-Saenz, Alexander ADurazo-Arvizu, Ramon ADesHarnais, SusanLau, Denys TEmanuel, Linda LR21CA113191/CA/United States NCIResearch Support, N.I.H., ExtramuralReviewUnited StatesJournal of pain and symptom managementJ Pain Symptom Manage. 2007 Jun;33(6):745-55. Epub 2007 Mar 23.
Keywords: *Needs Assessment; Humans; Medical Informatics/*organization & administration; Palliative Care/*organization & administration

Test design optimization in CAT early stage with the nominal response model - Passos, V. L.; Berger, M. P. F.; Tan, F. E. (Applied Psychological Measurement, 2007)
The early stage of computerized adaptive testing (CAT) refers to the phase of the trait estimation during the administration of only a few items. This phase can be characterized by bias and instability of estimation. In this study, an item selection criterion is introduced in an attempt to lessen this instability: the D-optimality criterion. A polytomous unconstrained CAT simulation is carried out to evaluate this criterion's performance under different test premises. The simulation shows that the extent of early stage instability depends primarily on the quality of the item pool information and its size and secondarily on the item selection criteria. The efficiency of the D-optimality criterion is similar to the efficiency of other known item selection criteria. Yet, it often yields estimates that, at the beginning of CAT, display a more robust performance against instability. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: computerized adaptive testing; nominal response model; robust performance; test design optimization

Two-Phase Item Selection Procedure for Flexible Content Balancing in CAT - Ying Cheng,; Chang, Hua-Hua; Qing Yi, (Applied Psychological Measurement, 2007)
Content balancing is an important issue in the design and implementation of computerized adaptive testing (CAT). Content-balancing techniques that have been applied in fixed content balancing, where the number of items from each content area is fixed, include constrained CAT (CCAT), the modified multinomial model (MMM), modified constrained CAT (MCCAT), and others. In this article, four methods are proposed to address the flexible content-balancing issue with the a-stratification design, named STR_C. The four methods are MMM+, an extension of MMM; MCCAT+, an extension of MCCAT; the TPM method, a two-phase content-balancing method using MMM in both phases; and the TPF method, a two-phase content-balancing method using MMM in the first phase and MCCAT in the second. Simulation results show that all of the methods work well in content balancing, and TPF performs the best in item exposure control and item pool utilization while maintaining measurement precision.

Two-phase item selection procedure for flexible content balancing in CAT - Cheng, Y; Chang, Hua-Hua; Yi, Q. (Applied Psychological. Measurement, 2007)

Up-and-down procedures for approximating optimal designs using person-response functions - Sheng, Y.; Flournoy, N.; Osterlind, S. J. (2007)
{PDF file, 1,042 KB}

Use of CAT in dynamic testing - De Beer, M. (2007)
{PDF file, 133 KB}

The use of computerized adaptive testing to assess psychopathology using the Global Appraisal of Individual Needs - Conrad, K. J.; Riley, B. B.; Dennis, M. L. (American Evaluation Association, 2007)

Validity and decision issues in selecting a CAT measurement model - Olsen, J. B.; Bunderson, C. V (2007)
{PDF file, 977 KB}

Adaptive success control in computerized adaptive testing - Häusler, Joachim (Psychology Science, 2006)
In computerized adaptive testing (CAT) procedures within the framework of probabilistic test theory the difficulty of an item is adjusted to the ability of the respondent, with the aim of maximizing the amount of information generated per item, thereby also increasing test economy and test reasonableness. However, earlier research indicates that respondents might feel over-challenged by a constant success probability of p = 0.5 and therefore cannot come to a sufficiently high answer certainty within a reasonable timeframe. Consequently response time per item increases, which -- depending on the test material -- can outweigh the benefit of administering optimally informative items. Instead of a benefit, the result of using CAT procedures could be a loss of test economy. Based on this problem, an adaptive success control algorithm was designed and tested, adapting the success probability to the working style of the respondent. Persons who need higher answer certainty in order to come to a decision are detected and receive a higher success probability, in order to minimize the test duration (not the number of items as in classical CAT). The method is validated on the re-analysis of data from the Adaptive Matrices Test (AMT, Hornke, Etzel & Rettig, 1999) and by the comparison between an AMT version using classical CAT and an experimental version using Adaptive Success Control. The results are discussed in the light of psychometric and psychological aspects of test quality. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: adaptive success control; computerized adaptive testing; Psychometrics

Applying Bayesian item selection approaches to adaptive tests using polytomous items - Penfield, R. D. (Applied Measurement in Education, 2006)
This study applied the maximum expected information (MEI) and the maximum posterior- weighted information (MPI) approaches of computer adaptive testing item selection to the case of a test using polytomous items following the partial credit model. The MEI and MPI approaches are described. A simulation study compared the efficiency of ability estimation using the MEI and MPI approaches to the traditional maximal item information (MII) approach. The results of the simulation study indicated that the MEI and MPI approaches led to a superior efficiency of ability estimation compared with the MII approach. The superiority of the MEI and MPI approaches over the MII approach was greatest when the bank contained items having a relatively peaked information function. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: adaptive tests; Bayesian item selection; computer adaptive testing; maximum expected information; polytomous items; posterior weighted information

Assembling a computerized adaptive testing item pool as a set of linear tests - van der Linden, W. J.; Ariel, A.; Veldkamp, B. P. (Journal of Educational and Behavioral Statistics, 2006)
Test-item writing efforts typically results in item pools with an undesirable correlational structure between the content attributes of the items and their statistical information. If such pools are used in computerized adaptive testing (CAT), the algorithm may be forced to select items with less than optimal information, that violate the content constraints, and/or have unfavorable exposure rates. Although at first sight somewhat counterintuitive, it is shown that if the CAT pool is assembled as a set of linear test forms, undesirable correlations can be broken down effectively. It is proposed to assemble such pools using a mixed integer programming model with constraints that guarantee that each test meets all content specifications and an objective function that requires them to have maximal information at a well-chosen set of ability values. An empirical example with a previous master pool from the Law School Admission Test (LSAT) yielded a CAT with nearly uniform bias and mean-squared error functions for the ability estimator and item-exposure rates that satisfied the target for all items in the pool.
Keywords: Algorithms; computerized adaptive testing; item pool; linear tests; mathematical models; statistics; Test Construction; Test Items

Assessing CAT Test Security Severity - Yi, Q., Zhang, J.; Chang, Hua-Hua (Applied Psychological Measurement, 2006)

A CAT with personality and attitude - Hol, A. M. (2006)
#HO06-01 .

Comparing methods of assessing differential item functioning in a computerized adaptive testing environment - Lei, P-W.; Chen, S-Y.; Yu, L. (Journal of Educational Measurement, 2006)
Mantel-Haenszel and SIBTEST, which have known difficulty in detecting non-unidirectional differential item functioning (DIF), have been adapted with some success for computerized adaptive testing (CAT). This study adapts logistic regression (LR) and the item-response-theory-likelihood-ratio test (IRT-LRT), capable of detecting both unidirectional and non-unidirectional DIF, to the CAT environment in which pretest items are assumed to be seeded in CATs but not used for trait estimation. The proposed adaptation methods were evaluated with simulated data under different sample size ratios and impact conditions in terms of Type I error, power, and specificity in identifying the form of DIF. The adapted LR and IRT-LRT procedures are more powerful than the CAT version of SIBTEST for non-unidirectional DIF detection. The good Type I error control provided by IRT-LRT under extremely unequal sample sizes and large impact is encouraging. Implications of these and other findings are discussed. all rights reserved)
Keywords: computerized adaptive testing; educational testing; item response theory likelihood ratio test; logistic regression; trait estimation; unidirectional & non-unidirectional differential item functioning

Comparing Methods of Assessing Differential Item Functioning in a Computerized Adaptive Testing Environment - Lei, Pui-Wa; Chen, Shu-Ying; Yu, Lan (Journal of Educational Measurement, 2006)
Mantel-Haenszel and SIBTEST, which have known difficulty in detecting non-unidirectional differential item functioning (DIF), have been adapted with some success for computerized adaptive testing (CAT). This study adapts logistic regression (LR) and the item-response-theory-likelihood-ratio test (IRT-LRT), capable of detecting both unidirectional and non-unidirectional DIF, to the CAT environment in which pretest items are assumed to be seeded in CATs but not used for trait estimation. The proposed adaptation methods were evaluated with simulated data under different sample size ratios and impact conditions in terms of Type I error, power, and specificity in identifying the form of DIF. The adapted LR and IRT-LRT procedures are more powerful than the CAT version of SIBTEST for non-unidirectional DIF detection. The good Type I error control provided by IRT-LRT under extremely unequal sample sizes and large impact is encouraging. Implications of these and other findings are discussed.

The comparison among item selection strategies of CAT with multiple-choice items - Hai-qi, D.; De-zhi, C.; Shuliang, D.; Taiping, D. (Acta Psychologica Sinica, 2006)
The initial purpose of comparing item selection strategies for CAT was to increase the efficiency of tests. As studies continued, however, it was found that increasing the efficiency of item bank using was also an important goal of comparing item selection strategies. These two goals often conflicted. The key solution was to find a strategy with which both goals could be accomplished. The item selection strategies for graded response model in this study included: the average of the difficulty orders matching with the ability; the medium of the difficulty orders matching with the ability; maximum information; A stratified (average); and A stratified (medium). The evaluation indexes used for comparison included: the bias of ability estimates for the true; the standard error of ability estimates; the average items which the examinees have administered; the standard deviation of the frequency of items selected; and sum of the indices weighted. Using the Monte Carlo simulation method, we obtained some data and computer iterated the data 20 times each under the conditions that the item difficulty parameters followed the normal distribution and even distribution. The results were as follows; The results indicated that no matter difficulty parameters followed the normal distribution or even distribution. Every type of item selection strategies designed in this research had its strong and weak points. In general evaluation, under the condition that items were stratified appropriately, A stratified (medium) (ASM) had the best effect. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: CAT; computerized adaptive testing; graded response model; item selection strategies; multiple choice items

A comparison of online calibration methods for a CAT - Morgan, D. L.; Way, W. D.; Augemberg, K.E. (Presented at the National Council on Measurement on Education, 2006)

Comparison of the Psychometric Properties of Several Computer-Based Test Designs for Credentialing Exams With Multiple Purposes - Jodoin, Michael G.; Zenisky, April; Hambleton, Ronald K. (Applied Measurement in Education, 2006)

Computer adaptive testing improved accuracy and precision of scores over random item selection in a physical functioning item bank - Haley, S.; Ni, P.; Hambleton, R. K.; Slavin, M.; Jette, A. (Journal of Clinical Epidemiology, 2006)

Computer adaptive testing improved accuracy and precision of scores over random item selection in a physical functioning item bank - Haley, S. M.; Ni, P.; Hambleton, R. K.; Slavin, M. D.; Jette, A. M. (Journal of Clinical Epidemiology, 2006)
BACKGROUND AND OBJECTIVE: Measuring physical functioning (PF) within and across postacute settings is critical for monitoring outcomes of rehabilitation; however, most current instruments lack sufficient breadth and feasibility for widespread use. Computer adaptive testing (CAT), in which item selection is tailored to the individual patient, holds promise for reducing response burden, yet maintaining measurement precision. We calibrated a PF item bank via item response theory (IRT), administered items with a post hoc CAT design, and determined whether CAT would improve accuracy and precision of score estimates over random item selection. METHODS: 1,041 adults were interviewed during postacute care rehabilitation episodes in either hospital or community settings. Responses for 124 PF items were calibrated using IRT methods to create a PF item bank. We examined the accuracy and precision of CAT-based scores compared to a random selection of items. RESULTS: CAT-based scores had higher correlations with the IRT-criterion scores, especially with short tests, and resulted in narrower confidence intervals than scores based on a random selection of items; gains, as expected, were especially large for low and high performing adults. CONCLUSION: The CAT design may have important precision and efficiency advantages for point-of-care functional assessment in rehabilitation practice settings.
Haley, Stephen MNi, PengshengHambleton, Ronald KSlavin, Mary DJette, Alan MK02 hd45354-01/hd/nichdR01 hd043568/hd/nichdComparative StudyResearch Support, N.I.H., ExtramuralResearch Support, U.S. Gov't, Non-P.H.S.EnglandJournal of clinical epidemiologyJ Clin Epidemiol. 2006 Nov;59(11):1174-82. Epub 2006 Jul 11.
Keywords: *Recovery of Function; Activities of Daily Living; Adolescent; Adult; Aged; Aged, 80 and over; Confidence Intervals; Factor Analysis, Statistical; Female; Humans; Male; Middle Aged; Outcome Assessment (Health Care)/*methods; Rehabilitation/*standards; Reproducibility of Results; Software

Computer-based testing - F Drasgow; Chuah, S. C. (Handbook of multimethod measurement in psychology, 2006)
(From the chapter) There has been a proliferation of research designed to explore and exploit opportunities provided by computer-based assessment. This chapter provides an overview of the diverse efforts by researchers in this area. It begins by describing how paper-and-pencil tests can be adapted for administration by computers. Computerization provides the important advantage that items can be selected so they are of appropriate difficulty for each examinee. Some of the psychometric theory needed for computerized adaptive testing is reviewed. Then research on innovative computerized assessments is summarized. These assessments go beyond multiple-choice items by using formats made possible by computerization. Then some hardware and software issues are described, and finally, directions for future work are outlined. (PsycINFO Database Record (c) 2006 APA )
Using Smart Source ParsingHandbook of multimethod measurement in psychology. (pp. 87-100). Washington, DC : American Psychological Association, [URL:http://www.apa.org/books]. xiv, 553 pp
Keywords: Adaptive Testing computerized adaptive testing; Computer Assisted Testing; Experimentation; Psychometrics; Theories

Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: I. Activity outcomes - Haley, S. M.; Siebens, H.; Coster, W. J.; Tao, W.; Black-Schaffer, R. M.; Gandek, B.; Sinclair, S. J.; Ni, P. (Archives of Physical Medicine and Rehabilitation, 2006)
OBJECTIVE: To examine score agreement, precision, validity, efficiency, and responsiveness of a computerized adaptive testing (CAT) version of the Activity Measure for Post-Acute Care (AM-PAC-CAT) in a prospective, 3-month follow-up sample of inpatient rehabilitation patients recently discharged home. DESIGN: Longitudinal, prospective 1-group cohort study of patients followed approximately 2 weeks after hospital discharge and then 3 months after the initial home visit. SETTING: Follow-up visits conducted in patients' home setting. PARTICIPANTS: Ninety-four adults who were recently discharged from inpatient rehabilitation, with diagnoses of neurologic, orthopedic, and medically complex conditions. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Summary scores from AM-PAC-CAT, including 3 activity domains of movement and physical, personal care and instrumental, and applied cognition were compared with scores from a traditional fixed-length version of the AM-PAC with 66 items (AM-PAC-66). RESULTS: AM-PAC-CAT scores were in good agreement (intraclass correlation coefficient model 3,1 range, .77-.86) with scores from the AM-PAC-66. On average, the CAT programs required 43% of the time and 33% of the items compared with the AM-PAC-66. Both formats discriminated across functional severity groups. The standardized response mean (SRM) was greater for the movement and physical fixed form than the CAT; the effect size and SRM of the 2 other AM-PAC domains showed similar sensitivity between CAT and fixed formats. Using patients' own report as an anchor-based measure of change, the CAT and fixed length formats were comparable in responsiveness to patient-reported change over a 3-month interval. CONCLUSIONS: Accurate estimates for functional activity group-level changes can be obtained from CAT administrations, with a considerable reduction in administration time.
Haley, Stephen MSiebens, HilaryCoster, Wendy JTao, WeiBlack-Schaffer, Randie MGandek, BarbaraSinclair, Samuel JNi, PengshengK0245354-01/phsR01 hd043568/hd/nichdResearch Support, N.I.H., ExtramuralUnited StatesArchives of physical medicine and rehabilitationArch Phys Med Rehabil. 2006 Aug;87(8):1033-42.
Keywords: *Activities of Daily Living; *Adaptation, Physiological; *Computer Systems; *Questionnaires; Adult; Aged; Aged, 80 and over; Chi-Square Distribution; Factor Analysis, Statistical; Female; Humans; Longitudinal Studies; Male; Middle Aged; Outcome Assessment (Health Care)/*methods; Patient Discharge; Prospective Studies; Rehabilitation/*standards; Subacute Care/*standards

Computerized adaptive testing of diabetes impact: a feasibility study of Hispanics and non-Hispanics in an active clinic population - Schwartz, C.; Welch, G.; Santiago-Kelley, P.; Bode, R.; Sun, X. (Quality of Life Research, 2006)
BACKGROUND: Diabetes is a leading cause of death and disability in the US and is twice as common among Hispanic Americans as non-Hispanics. The societal costs of diabetes provide an impetus for developing tools that can improve patient care and delay or prevent diabetes complications. METHODS: We implemented a feasibility study of a Computerized Adaptive Test (CAT) to measure diabetes impact using a sample of 103 English- and 97 Spanish-speaking patients (mean age = 56.5, 66.5% female) in a community medical center with a high proportion of minority patients (28% African-American). The 37 items of the Diabetes Impact Survey were translated using forward-backward translation and cognitive debriefing. Participants were randomized to receive either the full-length tool or the Diabetes-CAT first, in the patient's native language. RESULTS: The number of items and the amount of time to complete the survey for the CAT was reduced to one-sixth the amount for the full-length tool in both languages, across disease severity. Confirmatory Factor Analysis confirmed that the Diabetes Impact Survey is unidimensional. The Diabetes-CAT demonstrated acceptable internal consistency reliability, construct validity, and discriminant validity in the overall sample, although subgroup analyses suggested that the English sample data evidenced higher levels of reliability and validity than the Spanish sample and issues with discriminant validity in the Spanish sample. Differential Item Function analysis revealed differences in responses tendencies by language group in 3 of the 37 items. Participant interviews suggested that the Spanish-speaking patients generally preferred the paper survey to the computer-assisted tool, and were twice as likely to experience difficulties understanding the items. CONCLUSIONS: While the Diabetes-CAT demonstrated clear advantages in reducing respondent burden as compared to the full-length tool, simplifying the item bank will be necessary for enhancing the feasibility of the Diabetes-CAT for use with low literacy patients.
Schwartz, CarolynWelch, GarrySantiago-Kelley, PaulaBode, RitaSun, Xiaowu1 r43 dk066874-01/dk/niddkResearch Support, N.I.H., ExtramuralNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2006 Nov;15(9):1503-18. Epub 2006 Sep 26.
Keywords: *Computers; *Hispanic Americans; *Quality of Life; Adult; Aged; Data Collection/*methods; Diabetes Mellitus/*psychology; Feasibility Studies; Female; Humans; Language; Male; Middle Aged

Computerized adaptive testing under nonparametric IRT models - Xu, X.; Douglas, J. (Psychometrika, 2006)

Constraints-weighted information method for item selection of severely constrained computerized adaptive testing - Cheng, Y; Chang, Hua-Hua; Wang, X. B. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2006)

Designing computerized adaptive tests - Davey, T.; Pitoniak, M. J. (2006)

Effects of Estimation Bias on Multiple-Category Classification With an IRT-Based Adaptive Classification Procedure - Yang, Xiangdong; Poggio, John C.; Glasnapp, Douglas R. (Educational and Psychological Measurement, 2006)
The effects of five ability estimators, that is, maximum likelihood estimator, weighted likelihood estimator, maximum a posteriori, expected a posteriori, and Owen's sequential estimator, on the performances of the item response theory–based adaptive classification procedure on multiple categories were studied via simulations. The following results were found. (a) The Bayesian estimators were more likely to misclassify examinees into an inward category because of their inward biases, when a fixed start value of zero was assigned to every examinee. (b) When moderately accurate start values were available, however, Bayesian estimators produced classifications that were slightly more accurate than was the maximum likelihood estimator or weighted likelihood estimator. Expected a posteriori was the procedure that produced the most accurate results among the three Bayesian methods. (c) All five estimators produced equivalent efficiencies in terms of number of items required, which was 50 or more items except for abilities that were less than -2.00 or greater than 2.00.

Equating scores from adaptive to linear tests - van der Linden, W. J. (Applied Psychological Measurement, 2006)
Two local methods for observed-score equating are applied to the problem of equating an adaptive test to a linear test. In an empirical study, the methods were evaluated against a method based on the test characteristic function (TCF) of the linear test and traditional equipercentile equating applied to the ability estimates on the adaptive test for a population of test takers. The two local methods were generally best. Surprisingly, the TCF method performed slightly worse than the equipercentile method. Both methods showed strong bias and uniformly large inaccuracy, but the TCF method suffered from extra error due to the lower asymptote of the test characteristic function. It is argued that the worse performances of the two methods are a consequence of the fact that they use a single equating transformation for an entire population of test takers and therefore have to compromise between the individual score distributions.
Keywords: computerized adaptive testing; equipercentile equating; local equating; score reporting; test characteristic function

Estimation of an examinee's ability in the web-based computerized adaptive testing program IRT-CAT - Lee, Y. H.; Park, J. H.; Park, I. Y. (J Educ Eval Health Prof, 2006)
We developed a program to estimate an examinee s ability in order to provide freely available access to a web-based computerized adaptive testing (CAT) program. We used PHP and Java Script as the program languages, PostgresSQL as the database management system on an Apache web server and Linux as the operating system. A system which allows for user input and searching within inputted items and creates tests was constructed. We performed an ability estimation on each test based on a Rasch model and 2- or 3-parametric logistic models. Our system provides an algorithm for a web-based CAT, replacing previous personal computer-based ones, and makes it possible to estimate an examinee's ability immediately at the end of test.
Lee, Yoon-HwanPark, Jung-HoPark, In-YongKorea (South)Journal of educational evaluation for health professionsJ Educ Eval Health Prof. 2006;3:4. Epub 2006 Nov 22.

An evaluation of a patient-reported outcomes found computerized adaptive testing was efficient in assessing osteoarthritis impact - Kosinski, M.; Bjorner, J.; Warejr, J.; Sullivan, E.; Straus, W. (Journal of Clinical Epidemiology, 2006)
BACKGROUND AND OBJECTIVES: Evaluate a patient-reported outcomes questionnaire that uses computerized adaptive testing (CAT) to measure the impact of osteoarthritis (OA) on functioning and well-being. MATERIALS AND METHODS: OA patients completed 37 questions about the impact of OA on physical, social and role functioning, emotional well-being, and vitality. Questionnaire responses were calibrated and scored using item response theory, and two scores were estimated: a Total-OA score based on patients' responses to all 37 questions, and a simulated CAT-OA score where the computer selected and scored the five most informative questions for each patient. Agreement between Total-OA and CAT-OA scores was assessed using correlations. Discriminant validity of Total-OA and CAT-OA scores was assessed with analysis of variance. Criterion measures included OA pain and severity, patient global assessment, and missed work days. RESULTS: Simulated CAT-OA and Total-OA scores correlated highly (r = 0.96). Both Total-OA and simulated CAT-OA scores discriminated significantly between patients differing on the criterion measures. F-statistics across criterion measures ranged from 39.0 (P < .001) to 225.1 (P < .001) for the Total-OA score, and from 40.5 (P < .001) to 221.5 (P < .001) for the simulated CAT-OA score. CONCLUSIONS: CAT methods produce valid and precise estimates of the impact of OA on functioning and well-being with significant reduction in response burden.

Evaluation parameters for computer adaptive testing - Georgiadou, E.; Triantafillou, E.; Economides, A. A. (British Journal of Educational Technology, 2006)

Expansion of a physical function item bank and development of an abbreviated form for clinical research - Bode, R. K.; Lai, J-S.; Dineen, K.; Heinemann, A. W.; Shevrin, D.; Von Roenn, J.; Cella, D. (Journal of Applied Measurement, 2006)
We expanded an existing 33-item physical function (PF) item bank with a sufficient number of items to enable computerized adaptive testing (CAT). Ten items were written to expand the bank and the new item pool was administered to 295 people with cancer. For this analysis of the new pool, seven poorly performing items were identified for further examination. This resulted in a bank with items that define an essentially unidimensional PF construct, cover a wide range of that construct, reliably measure the PF of persons with cancer, and distinguish differences in self-reported functional performance levels. We also developed a 5-item (static) assessment form ("BriefPF") that can be used in clinical research to express scores on the same metric as the overall bank. The BriefPF was compared to the PF-10 from the Medical Outcomes Study SF-36. Both short forms significantly differentiated persons across functional performance levels. While the entire bank was more precise across the PF continuum than either short form, there were differences in the area of the continuum in which each short form was more precise: the BriefPF was more precise than the PF-10 at the lower functional levels and the PF-10 was more precise than the BriefPF at the higher levels. Future research on this bank will include the development of a CAT version, the PF-CAT. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: clinical research; computerized adaptive testing; performance levels; physical function item bank; Psychometrics; test reliability; Test Validity

Factor analysis techniques for assessing sufficient unidimensionality of cancer related fatigue - Lai, J-S.; Crane, P. K.; Cella, D. (Quality of Life Research, 2006)
BACKGROUND: Fatigue is the most common unrelieved symptom experienced by people with cancer. The purpose of this study was to examine whether cancer-related fatigue (CRF) can be summarized using a single score, that is, whether CRF is sufficiently unidimensional for measurement approaches that require or assume unidimensionality. We evaluated this question using factor analysis techniques including the theory-driven bi-factor model. METHODS: Five hundred and fifty five cancer patients from the Chicago metropolitan area completed a 72-item fatigue item bank, covering a range of fatigue-related concerns including intensity, frequency and interference with physical, mental, and social activities. Dimensionality was assessed using exploratory and confirmatory factor analysis (CFA) techniques. RESULTS: Exploratory factor analysis (EFA) techniques identified from 1 to 17 factors. The bi-factor model suggested that CRF was sufficiently unidimensional. CONCLUSIONS: CRF can be considered sufficiently unidimensional for applications that require unidimensionality. One such application, item response theory (IRT), will facilitate the development of short-form and computer-adaptive testing. This may further enable practical and accurate clinical assessment of CRF.
0962-9343 (Print)Journal ArticleResearch Support, N.I.H., Extramural
Keywords: *Factor Analysis, Statistical; *Quality of Life; Aged; Chicago; Fatigue/*etiology; Female; Humans; Male; Middle Aged; Neoplasms/*complications; Questionnaires

A Feedback Control Strategy for Enhancing Item Selection Efficiency in Computerized Adaptive Testing - Weissman, Alexander (Applied Psychological Measurement, 2006)
A computerized adaptive test (CAT) may be modeled as a closed-loop system, where item selection is influenced by trait level (θ) estimation and vice versa. When discrepancies exist between an examinee's estimated and true θ levels, nonoptimal item selection is a likely result. Nevertheless, examinee response behavior consistent with optimal item selection can be predicted using item response theory (IRT), without knowledge of an examinee's true θ level, yielding a specific reference point for applying an internal correcting or feedback control mechanism. Incorporating such a mechanism in a CAT is shown to be an effective strategy for increasing item selection efficiency. Results from simulation studies using maximum likelihood (ML) and modal a posteriori (MAP) trait-level estimation and Fisher information (FI) and Fisher interval information (FII) item selection are provided.

How Big Is Big Enough? Sample Size Requirements for CAST Item Parameter Estimation - Chuah, Siang Chee; F Drasgow; Luecht, Richard (Applied Measurement in Education, 2006)

An Introduction to Multistage Testing - Alan D Mead (Applied Measurement in Education, 2006)

Item banks and their potential applications to health status assessment in diverse populations - Hahn, E. A.; Cella, D.; Bode, R. K.; Gershon, R. C.; Lai, J. S. (Medical Care, 2006)
In the context of an ethnically diverse, aging society, attention is increasingly turning to health-related quality of life measurement to evaluate healthcare and treatment options for chronic diseases. When evaluating and treating symptoms and concerns such as fatigue, pain, or physical function, reliable and accurate assessment is a priority. Modern psychometric methods have enabled us to move from long, static tests that provide inefficient and often inaccurate assessment of individual patients, to computerized adaptive tests (CATs) that can precisely measure individuals on health domains of interest. These modern methods, collectively referred to as item response theory (IRT), can produce calibrated "item banks" from larger pools of questions. From these banks, CATs can be conducted on individuals to produce their scores on selected domains. Item banks allow for comparison of patients across different question sets because the patient's score is expressed on a common scale. Other advantages of using item banks include flexibility in terms of the degree of precision desired; interval measurement properties under most circumstances; realistic capability for accurate individual assessment over time (using CAT); and measurement equivalence across different patient populations. This work summarizes the process used in the creation and evaluation of item banks and reviews their potential contributions and limitations regarding outcome assessment and patient care, particularly when they are applied across people of different cultural backgrounds.
0025-7079 (Print)Journal ArticleResearch Support, N.I.H., ExtramuralResearch Support, Non-U.S. Gov't

[Item Selection Strategies of Computerized Adaptive Testing based on Graded Response Model.] - Ping, Chen; Shuliang, Ding; Haijing, Lin; Jie, Zhou (Acta Psychologica Sinica, 2006)
Item selection strategy (ISS) is an important component of Computerized Adaptive Testing (CAT). Its performance directly affects the security, efficiency and precision of the test. Thus, ISS becomes one of the central issues in CATs based on the Graded Response Model (GRM). It is well known that the goal of IIS is to administer the next unused item remaining in the item bank that best fits the examinees current ability estimate. In dichotomous IRT models, every item has only one difficulty parameter and the item whose difficulty matches the examinee's current ability estimate is considered to be the best fitting item. However, in GRM, each item has more than two ordered categories and has no single value to represent the item difficulty. Consequently, some researchers have used to employ the average or the median difficulty value across categories as the difficulty estimate for the item. Using the average value and the median value in effect introduced two corresponding ISSs. In this study, we used computer simulation compare four ISSs based on GRM. We also discussed the effect of "shadow pool" on the uniformity of pool usage as well as the influence of different item parameter distributions and different ability estimation methods on the evaluation criteria of CAT. In the simulation process, Monte Carlo method was adopted to simulate the entire CAT process; 1,000 examinees drawn from standard normal distribution and four 1,000-sized item pools of different item parameter distributions were also simulated. The assumption of the simulation is that a polytomous item is comprised of six ordered categories. In addition, ability estimates were derived using two methods. They were expected a posteriori Bayesian (EAP) and maximum likelihood estimation (MLE). In MLE, the Newton-Raphson iteration method and the Fisher Score iteration method were employed, respectively, to solve the likelihood equation. Moreover, the CAT process was simulated with each examinee 30 times to eliminate random error. The IISs were evaluated by four indices usually used in CAT from four aspects--the accuracy of ability estimation, the stability of IIS, the usage of item pool, and the test efficiency. Simulation results showed adequate evaluation of the ISS that matched the estimate of an examinee's current trait level with the difficulty values across categories. Setting "shadow pool" in ISS was able to improve the uniformity of pool utilization. Finally, different distributions of the item parameter and different ability estimation methods affected the evaluation indices of CAT. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: computerized adaptive testing; item selection strategy

Kernel-smoothed DIF detection procedure for computerized adaptive tests (Computerized testing report 00-08) - Nandakumar, R.; Banks, J. C.; Roussos, L. A. (Law School Admission Council, 2006)

Maximum information stratification method for controlling item exposure in computerized adaptive testing - Barrada, J; Mazuela, P.; Olea, J. (Psicothema, 2006)
The proposal for increasing the security in Computerized Adaptive Tests that has received most attention in recent years is the a-stratified method (AS - Chang and Ying, 1999): at the beginning of the test only items with low discrimination parameters (a) can be administered, with the values of the a parameters increasing as the test goes on. With this method, distribution of the exposure rates of the items is less skewed, while efficiency is maintained in trait-level estimation. The pseudo-guessing parameter (c), present in the three-parameter logistic model, is considered irrelevant, and is not used in the AS method. The Maximum Information Stratified (MIS) model incorporates the c parameter in the stratification of the bank and in the item-selection rule, improving accuracy by comparison with the AS, for item banks with a and b parameters correlated and uncorrelated. For both kinds of banks, the blocking b methods (Chang, Qian and Ying, 2001) improve the security of the item bank.Método de estratificación por máxima información para el control de la exposición en tests adaptativos informatizados. La propuesta para aumentar la seguridad en los tests adaptativos informatizados que ha recibido más atención en los últimos años ha sido el método a-estratificado (AE - Chang y Ying, 1999): en los momentos iniciales del test sólo pueden administrarse ítems con bajos parámetros de discriminación (a), incrementándose los valores del parámetro a admisibles según avanza el test. Con este método la distribución de las tasas de exposición de los ítems es más equilibrada, manteniendo una adecuada precisión en la medida. El parámetro de pseudoadivinación (c), presente en el modelo logístico de tres parámetros, se supone irrelevante y no se incorpora en el AE. El método de Estratificación por Máxima Información (EMI) incorpora el parámetro c a la estratificación del banco y a la regla de selección de ítems, mejorando la precisión en comparación con AE, tanto para bancos donde los parámetros a y b correlacionan como para bancos donde no. Para ambos tipos de bancos, los métodos de bloqueo de b (Chang, Qian y Ying, 2001) mejoran la seguridad del banco.
Barrada, Juan RamonMazuela, PalomaOlea, JulioResearch Support, Non-U.S. Gov'tSpainPsicothemaPsicothema. 2006 Feb;18(1):156-9.
Keywords: *Artificial Intelligence; *Microcomputers; *Psychological Tests; *Software Design; Algorithms; Chi-Square Distribution; Humans; Likelihood Functions

Measurement precision and efficiency of multidimensional computer adaptive testing of physical functioning using the pediatric evaluation of disability inventory - Haley, S. M.; Ni, P.; Ludlow, L. H.; Fragala-Pinkham, M. A. (Archives of Physical Medicine and Rehabilitation, 2006)
OBJECTIVE: To compare the measurement efficiency and precision of a multidimensional computer adaptive testing (M-CAT) application to a unidimensional CAT (U-CAT) comparison using item bank data from 2 of the functional skills scales of the Pediatric Evaluation of Disability Inventory (PEDI). DESIGN: Using existing PEDI mobility and self-care item banks, we compared the stability of item calibrations and model fit between unidimensional and multidimensional Rasch models and compared the efficiency and precision of the U-CAT- and M-CAT-simulated assessments to a random draw of items. SETTING: Pediatric rehabilitation hospital and clinics. PARTICIPANTS: Clinical and normative samples. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Not applicable. RESULTS: The M-CAT had greater levels of precision and efficiency than the separate mobility and self-care U-CAT versions when using a similar number of items for each PEDI subdomain. Equivalent estimation of mobility and self-care scores can be achieved with a 25% to 40% item reduction with the M-CAT compared with the U-CAT. CONCLUSIONS: M-CAT applications appear to have both precision and efficiency advantages compared with separate U-CAT assessments when content subdomains have a high correlation. Practitioners may also realize interpretive advantages of reporting test score information for each subdomain when separate clinical inferences are desired.
Haley, Stephen MNi, PengshengLudlow, Larry HFragala-Pinkham, Maria AK02 hd45354-01/hd/nichdResearch Support, N.I.H., ExtramuralResearch Support, Non-U.S. Gov'tUnited StatesArchives of physical medicine and rehabilitationArch Phys Med Rehabil. 2006 Sep;87(9):1223-9.
Keywords: *Disability Evaluation; *Pediatrics; Adolescent; Child; Child, Preschool; Computers; Disabled Persons/*classification/rehabilitation; Efficiency; Humans; Infant; Outcome Assessment (Health Care); Psychometrics; Self Care

Multidimensional computerized adaptive testing of the EORTC QLQ-C30: basic developments and evaluations - Petersen, M. A.; Groenvold, M.; Aaronson, N. K.; Fayers, P.; Sprangers, M.; Bjorner, J. B. (Quality of Life Research, 2006)
OBJECTIVE: Self-report questionnaires are widely used to measure health-related quality of life (HRQOL). Ideally, such questionnaires should be adapted to the individual patient and at the same time scores should be directly comparable across patients. This may be achieved using computerized adaptive testing (CAT). Usually, CAT is carried out for a single domain at a time. However, many HRQOL domains are highly correlated. Multidimensional CAT may utilize these correlations to improve measurement efficiency. We investigated the possible advantages and difficulties of multidimensional CAT. STUDY DESIGN AND SETTING: We evaluated multidimensional CAT of three scales from the EORTC QLQ-C30: the physical functioning, emotional functioning, and fatigue scales. Analyses utilised a database with 2958 European cancer patients. RESULTS: It was possible to obtain scores for the three domains with five to seven items administered using multidimensional CAT that were very close to the scores obtained using all 12 items and with no or little loss of measurement precision. CONCLUSION: The findings suggest that multidimensional CAT may significantly improve measurement precision and efficiency and encourage further research into multidimensional CAT. Particularly, the estimation of the model underlying the multidimensional CAT and the conceptual aspects need further investigations.
Petersen, Morten AaGroenvold, MogensAaronson, NeilFayers, PeterSprangers, MirjamBjorner, Jakob BEuropean Organisation for Research and Treatment of Cancer Quality of Life GroupResearch Support, Non-U.S. Gov'tNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2006 Apr;15(3):315-29.
Keywords: *Quality of Life; *Self Disclosure; Adult; Female; Health Status; Humans; Male; Middle Aged; Questionnaires/*standards; User-Computer Interface

Multiple maximum exposure rates in computerized adaptive testing - Barrada, J; Veldkamp, B. P.; Olea, J. (Paper presented at the SMABS-EAM Conference, 2006)

Multistage Testing: Widely or Narrowly Applicable? - Stark, Stephen; Chernyshenko, Oleksandr S. (Applied Measurement in Education, 2006)

Optimal and nonoptimal computer-based test designs for making pass-fail decisions - Hambleton, R. K.; Xing, D. (Applied Measurement in Education, 2006)
Now that many credentialing exams are being routinely administered by computer, new computer-based test designs, along with item response theory models, are being aggressively researched to identify specific designs that can increase the decision consistency and accuracy of pass-fail decisions. The purpose of this study was to investigate the impact of optimal and nonoptimal multistage test (MST) designs, linear parallel-form test designs (LPFT), and computer adaptive test (CAT) designs on the decision consistency and accuracy of pass-fail decisions. Realistic testing situations matching those of one of the large credentialing agencies were simulated to increase the generalizability of the findings. The conclusions were clear: (a) With the LPFTs, matching test information functions (TIFs) to the mean of the proficiency distribution produced slightly better results than matching them to the passing score; (b) all of the test designs worked better than test construction using random selection of items, subject to content constraints only; (c) CAT performed better than the other test designs; and (d) if matching a TIP to the passing score, the MST design produced a bit better results than the LPFT design. If an argument for the MST design is to be made, it can be made on the basis of slight improvements over the LPFT design and better expected item bank utilization, candidate preference, and the potential for improved diagnostic feedback, compared with the feedback that is possible with fixed linear test forms. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: adaptive test; credentialing exams; Decision Making; Educational Measurement; multistage tests; optimal computer-based test designs; test form

Optimal Testing With Easy or Difficult Items in Computerized Adaptive Testing - Theo Eggen; Verschoor, Angela J. (Applied Psychological Measurement, 2006)
Computerized adaptive tests (CATs) are individualized tests that, from a measurement point of view, are optimal for each individual, possibly under some practical conditions. In the present study, it is shown that maximum information item selection in CATs using an item bank that is calibrated with the one or the two-parameter logistic model results in each individual answering about 50% of the items correctly. Two item selection procedures giving easier (or more difficult) tests for students are presented and evaluated. Item selection on probability points of items yields good results only with the one-parameter logistic model and not with the two-parameter logistic model. An alternative selection procedure, based on maximum information at a shifted ability level, gives satisfactory results with both models. Index terms: computerized adaptive testing, item selection, item response theory

Optimal testing with easy or difficult items in computerized adaptive testing - Theo Eggen; Verschoor, Angela J. (Applied Psychological Measurement, 2006)
Computerized adaptive tests (CATs) are individualized tests that, from a measurement point of view, are optimal for each individual, possibly under some practical conditions. In the present study, it is shown that maximum information item selection in CATs using an item bank that is calibrated with the one- or the two-parameter logistic model results in each individual answering about 50% of the items correctly. Two item selection procedures giving easier (or more difficult) tests for students are presented and evaluated. Item selection on probability points of items yields good results only with the one-parameter logistic model and not with the two-parameter logistic model. An alternative selection procedure, based on maximum information at a shifted ability level, gives satisfactory results with both models. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: computer adaptive tests; individualized tests; Item Response Theory; item selection; Measurement

Optimal Testlet Pool Assembly for Multistage Testing Designs - Ariel, Adelaide; Veldkamp, Bernard P.; Breithaupt, Krista (Applied Psychological Measurement, 2006)
Computerized multistage testing (MST) designs require sets of test questions (testlets) to be assembled to meet strict, often competing criteria. Rules that govern testlet assembly may dictate the number of questions on a particular subject or may describe desirable statistical properties for the test, such as measurement precision. In an MST design, testlets of differing difficulty levels must be created. Statistical properties for assembly of the testlets can be expressed using item response theory (IRT) parameters. The testlet test information function (TIF) value can be maximized at a specific point on the IRT ability scale. In practical MST designs, parallel versions of testlets are needed, so sets of testlets with equivalent properties are built according to equivalent specifications. In this project, the authors study the use of a mathematical programming technique to simultaneously assemble testlets to ensure equivalence and fairness to candidates who may be administered different testlets.

Overview of quantitative measurement methods. Equivalence, invariance, and differential item functioning in health applications - Teresi, J. A. (Medical Care, 2006)
BACKGROUND: Reviewed in this article are issues relating to the study of invariance and differential item functioning (DIF). The aim of factor analyses and DIF, in the context of invariance testing, is the examination of group differences in item response conditional on an estimate of disability. Discussed are parameters and statistics that are not invariant and cannot be compared validly in crosscultural studies with varying distributions of disability in contrast to those that can be compared (if the model assumptions are met) because they are produced by models such as linear and nonlinear regression. OBJECTIVES: The purpose of this overview is to provide an integrated approach to the quantitative methods used in this special issue to examine measurement equivalence. The methods include classical test theory (CTT), factor analytic, and parametric and nonparametric approaches to DIF detection. Also included in the quantitative section is a discussion of item banking and computerized adaptive testing (CAT). METHODS: Factorial invariance and the articles discussing this topic are introduced. A brief overview of the DIF methods presented in the quantitative section of the special issue is provided together with a discussion of ways in which DIF analyses and examination of invariance using factor models may be complementary. CONCLUSIONS: Although factor analytic and DIF detection methods share features, they provide unique information and can be viewed as complementary in informing about measurement equivalence.
Teresi, Jeanne AAG15294/AG/NIA NIH HHS/United StatesResearch Support, N.I.H., ExtramuralResearch Support, Non-U.S. Gov'tReviewUnited StatesMedical careMed Care. 2006 Nov;44(11 Suppl 3):S39-49.
Keywords: *Cross-Cultural Comparison; Data Interpretation, Statistical; Factor Analysis, Statistical; Guidelines as Topic; Humans; Models, Statistical; Psychometrics/*methods; Statistics as Topic/*methods; Statistics, Nonparametric

Sensitivity of a computer adaptive assessment for measuring functional mobility changes in children enrolled in a community fitness programme - Haley, S. M.; Fragala-Pinkham, M. A.; Ni, P. (Clin Rehabil, 2006)

Sequential Computerized Mastery Tests—Three Simulation Studies - Wiberg, Marie (International Journal of Testing, 2006)

SIMCAT 1.0: A SAS computer program for simulating computer adaptive testing - Raîche, G.; Blais, J-G. (Applied Psychological Measurement, 2006)
Monte Carlo methodologies are frequently applied to study the sampling distribution of the estimated proficiency level in adaptive testing. These methods eliminate real situational constraints. However, these Monte Carlo methodologies are not currently supported by the available software programs, and when these programs are available, their flexibility is limited. SIMCAT 1.0 is aimed at the simulation of adaptive testing sessions under different adaptive expected a posteriori (EAP) proficiency-level estimation methods (Blais & Raîche, 2005; Raîche & Blais, 2005) based on the one-parameter Rasch logistic model. These methods are all adaptive in the a priori proficiency-level estimation, the proficiency-level estimation bias correction, the integration interval, or a combination of these factors. The use of these adaptive EAP estimation methods diminishes considerably the shrinking, and therefore biasing, effect of the estimated a priori proficiency level encountered when this a priori is fixed at a constant value independently of the computed previous value of the proficiency level. SIMCAT 1.0 also computes empirical and estimated skewness and kurtosis coefficients, such as the standard error, of the estimated proficiency-level sampling distribution. In this way, the program allows one to compare empirical and estimated properties of the estimated proficiency-level sampling distribution under different variations of the EAP estimation method: standard error and bias, like the skewness and kurtosis coefficients. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: computer adaptive testing; computer program; estimated proficiency level; Monte Carlo methodologies; Rasch logistic model

Simulated computerized adaptive test for patients with lumbar spine impairments was efficient and produced valid measures of function - Hart, D.; Mioduski, J.; Werenke, M.; Stratford, P. (Journal of Clinical Epidemiology, 2006)

Simulated computerized adaptive test for patients with lumbar spine impairments was efficient and produced valid measures of function - Hart, D. L.; Mioduski, J. E.; Werneke, M. W.; Stratford, P. W. (Journal of Clinical Epidemiology, 2006)
Objective: To equate physical functioning (PF) items with Back Pain Functional Scale (BPFS) items, develop a computerized adaptive test (CAT) designed to assess lumbar spine functional status (LFS) in people with lumbar spine impairments, and compare discriminant validity of LFS measures (qIRT) generated using all items analyzed with a rating scale Item Response Theory model (RSM) and measures generated using the simulated CAT (qCAT). Methods: We performed a secondary analysis of retrospective intake rehabilitation data. Results: Unidimensionality and local independence of 25 BPFS and PF items were supported. Differential item functioning was negligible for levels of symptom acuity, gender, age, and surgical history. The RSM fit the data well. A lumbar spine specific CAT was developed that was 72% more efficient than using all 25 items to estimate LFS measures. qIRT and qCAT measures did not discriminate patients by symptom acuity, age, or gender, but discriminated patients by surgical history in similar clinically logical ways. qCAT measures were as precise as qIRT measures. Conclusion: A body part specific simulated CAT developed from an LFS item bank was efficient and produced precise measures of LFS without eroding discriminant validity.
Keywords: Back Pain Functional Scale; computerized adaptive testing; Item Response Theory; Lumbar spine; Rehabilitation; True-score equating

Simulated computerized adaptive test for patients with shoulder impairments was efficient and produced valid measures of function - Hart, D. L.; Cook, K. F.; Mioduski, J. E.; Teal, C. R.; Crane, P. K. (Journal of Clinical Epidemiology, 2006)
BACKGROUND AND OBJECTIVE: To test unidimensionality and local independence of a set of shoulder functional status (SFS) items, develop a computerized adaptive test (CAT) of the items using a rating scale item response theory model (RSM), and compare discriminant validity of measures generated using all items (theta(IRT)) and measures generated using the simulated CAT (theta(CAT)). STUDY DESIGN AND SETTING: We performed a secondary analysis of data collected prospectively during rehabilitation of 400 patients with shoulder impairments who completed 60 SFS items. RESULTS: Factor analytic techniques supported that the 42 SFS items formed a unidimensional scale and were locally independent. Except for five items, which were deleted, the RSM fit the data well. The remaining 37 SFS items were used to generate the CAT. On average, 6 items were needed to estimate precise measures of function using the SFS CAT, compared with all 37 SFS items. The theta(IRT) and theta(CAT) measures were highly correlated (r = .96) and resulted in similar classifications of patients. CONCLUSION: The simulated SFS CAT was efficient and produced precise, clinically relevant measures of functional status with good discriminating ability.
0895-4356 (Print)Journal ArticleValidation Studies
Keywords: *Computer Simulation; *Range of Motion, Articular; Activities of Daily Living; Adult; Aged; Aged, 80 and over; Factor Analysis, Statistical; Female; Humans; Male; Middle Aged; Prospective Studies; Reproducibility of Results; Research Support, N.I.H., Extramural; Research Support, U.S. Gov't, Non-P.H.S.; Shoulder Dislocation/*physiopathology/psychology/rehabilitation; Shoulder Pain/*physiopathology/psychology/rehabilitation; Shoulder/*physiopathology; Sickness Impact Profile; Treatment Outcome

Simulated computerized adaptive test for patients with shoulder impairments was efficient and produced valid measures of function - Hart, D. L.; Cook, K. F.; Mioduski, J. E.; Teal, C. R.; Crane, P. K. (Journal of Clinical Epidemiology, 2006)
Background and Objective: To test unidimensionality and local independence of a set of shoulder functional status (SFS) items, develop a computerized adaptive test (CAT) of the items using a rating scale item response theory model (RSM), and compare discriminant validity of measures generated using all items (qIRT) and measures generated using the simulated CAT (qCAT). Study Design and Setting: We performed a secondary analysis of data collected prospectively during rehabilitation of 400 patients with shoulder impairments who completed 60 SFS items. Results: Factor analytic techniques supported that the 42 SFS items formed a unidimensional scale and were locally independent. Except for five items, which were deleted, the RSM fit the data well. The remaining 37 SFS items were used to generate the CAT. On average, 6 items on were needed to estimate precise measures of function using the SFS CAT, compared with all 37 SFS items. The qIRT and qCAT measures were highly correlated (r 5 .96) and resulted in similar classifications of patients. Conclusion: The simulated SFS CAT was efficient and produced precise, clinically relevant measures of functional status with good discriminating ability.
Keywords: computerized adaptive testing; Flexilevel Scale of Shoulder Function; Item Response Theory; Rehabilitation

Técnicas para detectar patrones de respuesta atípicos [Aberrant patterns detection methods] - Núñez, R. M. N.; Pina, J. A. L. (Anales de Psicología, 2006)
La identificación de patrones de respuesta atípicos es de gran utilidad para la construcción de tests y de bancos de ítems con propiedades psicométricas así como para el análisis de validez de los mismos. En este trabajo de revisión se han recogido los más relevantes y novedosos métodos de ajuste de personas que se han elaborado dentro de cada uno de los principales ámbitos de trabajo de la Psicometría: el escalograma de Guttman, la Teoría Clásica de Tests (TCT), la Teoría de la Generalizabilidad (TG), la Teoría de Respuesta al Ítem (TRI), los Modelos de Respuesta al Ítem No Paramétricos (MRINP), los Modelos de Clase Latente de Orden Restringido (MCL-OR) y el Análisis de Estructura de Covarianzas (AEC).Aberrant patterns detection has a great usefulness in order to make tests and item banks with psychometric characteristics and validity analysis of tests and items. The most relevant and newest person-fit methods have been reviewed. All of them have been made in each one of main areas of Psychometry: Guttman's scalogram, Classical Test Theory (CTT), Generalizability Theory (GT), Item Response Theory (IRT), Non-parametric Response Models (NPRM), Order-Restricted Latent Class Models (OR-LCM) and Covariance Structure Analysis (CSA).
Spain: Universidad de Murcia
Keywords: aberrant patterns detection; Classical Test Theory; generalizability theory; Item Response; Item Response Theory; Mathematics; methods; person-fit; Psychometrics; psychometry; Test Validity; test validity analysis; Theory

A testlet assembly design for the uniform CPA Examination - Luecht, Richard; Brumfield, Terry; Breithaupt, Krista (Applied Measurement in Education, 2006)

Validitätssteigerungen durch adaptives Testen [Increasing validity by adaptive testing]. - Frey, A. (2006)

A variant of the progressive restricted item exposure control procedure in computerized adaptive testing systems based on the 3PL and the partial credit model - McClarty, L. K.; Sperling, R.; Dodd, B. G. (Paper presented at the annual meetings of the American Educational Research Association, 2006)

Adaptive orientation methods in computer adaptive testing - Economides, A. A. (2005)

Adaptive selection of personality items to inform a neural network predicting job performance - Thissen-Roe, A. (2005)
{PDF file, 488 KB}

Applications of item response theory to improve health outcomes assessment: Developing item banks, linking instruments, and computer-adaptive testing - Hambleton, R. K. (Outcomes assessment in cancer, 2005)
(From the chapter) The current chapter builds on Reise's introduction to the basic concepts, assumptions, popular models, and important features of IRT and discusses the applications of item response theory (IRT) modeling to health outcomes assessment. In particular, we highlight the critical role of IRT modeling in: developing an instrument to match a study's population; linking two or more instruments measuring similar constructs on a common metric; and creating item banks that provide the foundation for tailored short-form instruments or for computerized adaptive assessments. (PsycINFO Database Record (c) 2005 APA )
Using Smart Source ParsingOutcomes assessment in cancer: Measures, methods, and applications. (pp. 445-464). New York, NY : Cambridge University Press. xiv, 662 pp
Keywords: Computer Assisted Testing; Health; Item Response Theory; Measurement; Test Construction; Treatment Outcomes

Assessing Mobility in Children Using a Computer Adaptive Testing Version of the Pediatric Evaluation of Disability Inventory - Haley, S.; Raczek, A.; Coster, W.; Dumas, H.; Fragalapinkham, M. (Archives of Physical Medicine and Rehabilitation, 2005)

Assessing mobility in children using a computer adaptive testing version of the pediatric evaluation of disability inventory - Haley, S. M.; Raczek, A. E.; Coster, W. J.; Dumas, H. M.; Fragala-Pinkham, M. A. (Archives of Physical Medicine and Rehabilitation, 2005)
OBJECTIVE: To assess score agreement, validity, precision, and response burden of a prototype computerized adaptive testing (CAT) version of the Mobility Functional Skills Scale (Mob-CAT) of the Pediatric Evaluation of Disability Inventory (PEDI) as compared with the full 59-item version (Mob-59). DESIGN: Computer simulation analysis of cross-sectional and longitudinal retrospective data; and cross-sectional prospective study. SETTING: Pediatric rehabilitation hospital, including inpatient acute rehabilitation, day school program, outpatient clinics, community-based day care, preschool, and children's homes. PARTICIPANTS: Four hundred sixty-nine children with disabilities and 412 children with no disabilities (analytic sample); 41 children without disabilities and 39 with disabilities (cross-validation sample). INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Summary scores from a prototype Mob-CAT application and versions using 15-, 10-, and 5-item stopping rules; scores from the Mob-59; and number of items and time (in seconds) to administer assessments. RESULTS: Mob-CAT scores from both computer simulations (intraclass correlation coefficient [ICC] range, .94-.99) and field administrations (ICC=.98) were in high agreement with scores from the Mob-59. Using computer simulations of retrospective data, discriminant validity, and sensitivity to change of the Mob-CAT closely approximated that of the Mob-59, especially when using the 15- and 10-item stopping rule versions of the Mob-CAT. The Mob-CAT used no more than 15% of the items for any single administration, and required 20% of the time needed to administer the Mob-59. CONCLUSIONS: Comparable score estimates for the PEDI mobility scale can be obtained from CAT administrations, with losses in validity and precision for shorter forms, but with a considerable reduction in administration time.
Haley, Stephen MRaczek, Anastasia ECoster, Wendy JDumas, Helene MFragala-Pinkham, Maria AK02 hd45354-01a1/hd/nichdR43 hd42388-01/hd/nichdResearch Support, N.I.H., ExtramuralResearch Support, U.S. Gov't, P.H.S.United StatesArchives of physical medicine and rehabilitationArch Phys Med Rehabil. 2005 May;86(5):932-9.
Keywords: *Computer Simulation; *Disability Evaluation; Adolescent; Child; Child, Preschool; Cross-Sectional Studies; Disabled Children/*rehabilitation; Female; Humans; Infant; Male; Outcome Assessment (Health Care)/*methods; Rehabilitation Centers; Rehabilitation/*standards; Sensitivity and Specificity

An Authoring Environment for Adaptive Testing - Guzmán, E; Conejo, R; García-Hervás, E (Educational Technology & Society, 2005)
SIETTE is a web-based adaptive testing system. It implements Computerized Adaptive Tests. These tests are tailor-made, theory-based tests, where questions shown to students, finalization of the test, and student knowledge estimation is accomplished adaptively. To construct these tests, SIETTE has an authoring environment comprising a suite of tools that helps teachers create questions and tests properly, and analyze students’ performance after taking a test. In this paper, we present this authoring environment in the framework of adaptive testing. As will be shown, this set of visual tools, that contain some adaptable eatures, can be useful for teachers lacking skills in this kind of testing. Additionally, other systems that implement adaptive testing will be studied.
Keywords: Adaptability; Adaptive Testing; Authoring environment; Item Response Theory

Automated Simultaneous Assembly for Multistage Testing - Breithaupt, Krista; Ariel, Adelaide; Veldkamp, Bernard P. (International Journal of Testing, 2005)

A Bayesian student model without hidden nodes and its comparison with item response theory - Desmarais, M. C.; Pu, X. (International Journal of Artificial Intelligence in Education, 2005)
The Bayesian framework offers a number of techniques for inferring an individual's knowledge state from evidence of mastery of concepts or skills. A typical application where such a technique can be useful is Computer Adaptive Testing (CAT). A Bayesian modeling scheme, POKS, is proposed and compared to the traditional Item Response Theory (IRT), which has been the prevalent CAT approach for the last three decades. POKS is based on the theory of knowledge spaces and constructs item-to-item graph structures without hidden nodes. It aims to offer an effective knowledge assessment method with an efficient algorithm for learning the graph structure from data. We review the different Bayesian approaches to modeling student ability assessment and discuss how POKS relates to them. The performance of POKS is compared to the IRT two parameter logistic model. Experimental results over a 34 item Unix test and a 160 item French language test show that both approaches can classify examinees as master or non-master effectively and efficiently, with relatively comparable performance. However, more significant differences are found in favor of POKS for a second task that consists in predicting individual question item outcome. Implications of these results for adaptive testing and student modeling are discussed, as well as the limitations and advantages of POKS, namely the issue of integrating concepts into its structure. (PsycINFO Database Record (c) 2007 APA, all rights reserved)
Keywords: Bayesian Student Model; computer adaptive testing; hidden nodes; Item Response Theory

A closer look at using judgments of item difficulty to change answers on computerized adaptive tests - Vispoel, W. P.; Clough, S. J.; Bleiler, T. (Journal of Educational Measurement, 2005)

A comparison of adaptive mastery testing using testlets with the 3-parameter logistic model - Jacobs-Cassuto, M.S. (2005)

A comparison of item-selection methods for adaptive tests with content constraints - van der Linden, W. J. (Journal of Educational Measurement, 2005)

A comparison of item-selection methods for adaptive tests with content constraints - van der Linden, W. J. (Journal of Educational Measurement, 2005)
In test assembly, a fundamental difference exists between algorithms that select a test sequentially or simultaneously. Sequential assembly allows us to optimize an objective function at the examinee's ability estimate, such as the test information function in computerized adaptive testing. But it leads to the non-trivial problem of how to realize a set of content constraints on the test—a problem more naturally solved by a simultaneous item-selection method. Three main item-selection methods in adaptive testing offer solutions to this dilemma. The spiraling method moves item selection across categories of items in the pool proportionally to the numbers needed from them. Item selection by the weighted-deviations method (WDM) and the shadow test approach (STA) is based on projections of the future consequences of selecting an item. These two methods differ in that the former calculates a projection of a weighted sum of the attributes of the eventual test and the latter a projection of the test itself. The pros and cons of these methods are analyzed. An empirical comparison between the WDM and STA was conducted for an adaptive version of the Law School Admission Test (LSAT), which showed equally good item-exposure rates but violations of some of the constraints and larger bias and inaccuracy of the ability estimator for the WDM.
Keywords: Adaptive Testing; Algorithms; content constraints; item selection method; shadow test approach; spiraling method; weighted deviations method

Computer adaptive testing - Gershon, R. C. (Journal of Applied Measurement, 2005)
The creation of item response theory (IRT) and Rasch models, inexpensive accessibility to high speed desktop computers, and the growth of the Internet, has led to the creation and growth of computerized adaptive testing or CAT. This form of assessment is applicable for both high stakes tests such as certification or licensure exams, as well as health related quality of life surveys. This article discusses the historical background of CAT including its many advantages over conventional (typically paper and pencil) alternatives. The process of CAT is then described including descriptions of the specific differences of using CAT based upon 1-, 2- and 3-parameter IRT and various Rasch models. Numerous specific topics describing CAT in practice are described including: initial item selection, content balancing, test difficulty, test length and stopping rules. The article concludes with the author's reflections regarding the future of CAT.
Gershon, Richard CReviewUnited StatesJournal of applied measurementJ Appl Meas. 2005;6(1):109-27.
Keywords: *Internet; *Models, Statistical; *User-Computer Interface; Certification; Health Surveys; Humans; Licensure; Microcomputers; Quality of Life

Computer adaptive testing - Gershon, R. C. (Journal of Applied Measurement, 2005)

A computer adaptive testing approach for assessing physical functioning in children and adolescents - Haley, S. M.; Ni, P.; Fragala-Pinkham, M. A.; Skrinar, A. M.; Corzo, D. (Developmental Medicine and Child Neuropsychology, 2005)
The purpose of this article is to demonstrate: (1) the accuracy and (2) the reduction in amount of time and effort in assessing physical functioning (self-care and mobility domains) of children and adolescents using computer-adaptive testing (CAT). A CAT algorithm selects questions directly tailored to the child's ability level, based on previous responses. Using a CAT algorithm, a simulation study was used to determine the number of items necessary to approximate the score of a full-length assessment. We built simulated CAT (5-, 10-, 15-, and 20-item versions) for self-care and mobility domains and tested their accuracy in a normative sample (n=373; 190 males, 183 females; mean age 6y 11mo [SD 4y 2m], range 4mo to 14y 11mo) and a sample of children and adolescents with Pompe disease (n=26; 21 males, 5 females; mean age 6y 1mo [SD 3y 10mo], range 5mo to 14y 10mo). Results indicated that comparable score estimates (based on computer simulations) to the full-length tests can be achieved in a 20-item CAT version for all age ranges and for normative and clinical samples. No more than 13 to 16% of the items in the full-length tests were needed for any one administration. These results support further consideration of using CAT programs for accurate and efficient clinical assessments of physical functioning.
Haley, Stephen MNi, PengshengFragala-Pinkham, Maria ASkrinar, Alison MCorzo, DeyaniraComparative StudyResearch Support, Non-U.S. Gov'tEnglandDevelopmental medicine and child neurologyDev Med Child Neurol. 2005 Feb;47(2):113-20.
Keywords: *Computer Systems; Activities of Daily Living; Adolescent; Age Factors; Child; Child Development/*physiology; Child, Preschool; Computer Simulation; Confidence Intervals; Demography; Female; Glycogen Storage Disease Type II/physiopathology; Health Status Indicators; Humans; Infant; Infant, Newborn; Male; Motor Activity/*physiology; Outcome Assessment (Health Care)/*methods; Reproducibility of Results; Self Care; Sensitivity and Specificity

Computer adaptive testing quality requirements - Economides, A. A. (2005)

A computer-assisted test design and diagnosis system for use by classroom teachers - He, Q.; Tymms, P. (Journal of Computer Assisted Learning, 2005)
Computer-assisted assessment (CAA) has become increasingly important in education in recent years. A variety of computer software systems have been developed to help assess the performance of students at various levels. However, such systems are primarily designed to provide objective assessment of students and analysis of test items, and focus has been mainly placed on higher and further education. Although there are commercial professional systems available for use by primary and secondary educational institutions, such systems are generally expensive and require skilled expertise to operate. In view of the rapid progress made in the use of computer-based assessment for primary and secondary students by education authorities here in the UK and elsewhere, there is a need to develop systems which are economic and easy to use and can provide the necessary information that can help teachers improve students' performance. This paper presents the development of a software system that provides a range of functions including generating items and building item banks, designing tests, conducting tests on computers and analysing test results. Specifically, the system can generate information on the performance of students and test items that can be easily used to identify curriculum areas where students are under performing. A case study based on data collected from five secondary schools in Hong Kong involved in the Curriculum, Evaluation and Management Centre's Middle Years Information System Project, Durham University, UK, has been undertaken to demonstrate the use of the system for diagnostic and performance analysis. (PsycINFO Database Record (c) 2006 APA ) (journal abstract)
Keywords: Computer Assisted Testing; Computer Software; Diagnosis; Educational Measurement; Teachers

Computerized adaptive testing: a mixture item selection approach for constrained situations - Leung, C. K.; Chang, Hua-Hua; Hau, K. T. (British Journal of Mathematical and Statistical Psychology, 2005)
In computerized adaptive testing (CAT), traditionally the most discriminating items are selected to provide the maximum information so as to attain the highest efficiency in trait (theta) estimation. The maximum information (MI) approach typically results in unbalanced item exposure and hence high item-overlap rates across examinees. Recently, Yi and Chang (2003) proposed the multiple stratification (MS) method to remedy the shortcomings of MI. In MS, items are first sorted according to content, then difficulty and finally discrimination parameters. As discriminating items are used strategically, MS offers a better utilization of the entire item pool. However, for testing with imposed non-statistical constraints, this new stratification approach may not maintain its high efficiency. Through a series of simulation studies, this research explored the possible benefits of a mixture item selection approach (MS-MI), integrating the MS and MI approaches, in testing with non-statistical constraints. In all simulation conditions, MS consistently outperformed the other two competing approaches in item pool utilization, while the MS-MI and the MI approaches yielded higher measurement efficiency and offered better conformity to the constraints. Furthermore, the MS-MI approach was shown to perform better than MI on all evaluation criteria when control of item exposure was imposed.
Leung, Chi-KeungChang, Hua-HuaHau, Kit-TaiEnglandBr J Math Stat Psychol. 2005 Nov;58(Pt 2):239-57.
Keywords: *Computer-Aided Design; *Educational Measurement/methods; *Models, Psychological; Humans; Psychometrics/methods