2011 IACAT Conference

Asilomar Conference Center

Pacific Grove, California
October 3–5, 2011
 
President of IACAT
Theo J. H. M. Eggen, Cito, The Netherlands (Theo.Eggen@cito.nl)
 
Program Committee
Michelle Boyer, CTB/McGraw-Hill, USA (michelle_boyer@ctb.com)
Clifford Donath, Donath Group, USA (cliff@donathgroup.com)
Theo J. H. M. Eggen, Cito & University of Twente, The Netherlands (Theo.Eggen@cito.nl)
K. T. Hau, Chinese University, Hong Kong (kthau@cuhk.edu.hk)
Richard J. Patz, CTB/McGraw-Hill, USA (richard_patz@ctb.com)
Lawrence M. Rudner, Graduate Management Admission Council, USA (lrudner@gmac.com)
Kazuo Shigemasu, Teikyo University, Japan (kshige@main.teikyo-u.ac.jp)
Wim J. van der Linden, CTB/McGraw-Hill, USA (wim_vanderlinden@ctb.com)
David Weiss, University of Minnesota, USA (djweiss@mailbox.mail.umn.edu)

Program Outline

Workshops
Monday, 9:00 AM – 12:00 PM Room: Heather
A Primer on Adaptive Testing Wim J. van der Linden, CTB/McGraw-Hill

Monday, 9:00 AM – 12:00 PM Room: Acacia
Multidimensional Adaptive Testing Hua-Hua Chang, University of Illinois

Monday, 9:00 AM – 12:00 PM Room: Toyon
Adaptive Testing with Shadow Tests Using R Qi Diao, CTB/McGraw-Hill
 
Opening and Welcome
Monday, 1:00 PM – 1:15 PM Room: Merrill Hall
Theo J. H. M. Eggen, President, IACAT
Ellen Haley, President, CTB/McGraw-Hill
Wim J. van der Linden, Program Chair
 
Keynote Presentations
Monday, 1:15 PM – 2:45 PM Room: Merrill Hall
Chair: Richard J. Patz, CTB/McGraw-Hill

1:15 PM – 2:00 PM  Building CAT-Driven Assessment and Diagnosis to Address Today’s Challenges in Learning, Hua-Hua Chang(video recording), University of Illinois at Urbana-Champaign
2:00 PM – 2:45 PM  Measuring Complex Health Outcomes: A Multidimensional Hierarchical Computerized Adaptive Testing Approach, Seung Choi (video recording), PROMIS Project, Northwestern University

Breakout Sessions

Monday, 3:00 PM – 6:05 PM

Adaptive Testing for Classification and/or Diagnosis (invited symposium) Organizer & Chair: Ying Cheng, University of Notre Dame

3:00 PM-3:25 PM Likelihood-Ratio Based Computerized Classification Testing, Nathan A. Thompson, Assessment Systems Corporation

3:30 PM-3:55 PM Multiple Category Classification with the Sequential Probability Ratio Test with Multidimensional Adaptive Testing, Nicki-Nils Seitz* & Andreas Frey, Friedrich-Schiller University

4:00 PM-4:25 PM Item Calibration Error and Variable-Length Computerized Testing, Jeff Patton* & Ying Cheng, University of Notre Dame 

4:40 PM-5:05 PM Mutual Information Item Selection in Cognitive Diagnostic CAT, Chun Wang* & Hua-Hua Chang, University of Illinois

5:10 PM-5:35 PM Item Exposure Control in Cognitive Diagnostic Computerized Adaptive Testing, Haiyan Lin* & Hua-Hua Chang, University of Illinois

5:40 PM-6:05 PM Stage-Wise Adaptive Designs and Sequential Testing for Clinical Trials, Ying Cheng, Indiana University South Bend

What’s Next? A Discussion of Research Issues and Practical Requirements that May Influence the Next Generation of Adaptive Tests (invited symposium), Organizer & Chair: G. Gage Kingsbury, NWEA

3:00 PM-3:25 PM Dynamic Test Publishing, or, How to Use Rules and Filters to Decide which Item to Administer Next, G. Gage Kingsbury, NWEA

​3:30 PM-3:55 PM Walking the Tightrope: Using Better Content Control to Improve CAT, Kathleen A. Gialluca, Pearson

4:00 PM-4:25 PM Moving beyond Efficiency to Allow CAT to Provide Better Diagnostic Information, Brian D. Bontempo, Mountain Measurement

4:40 PM-5:05 PM A Paradigm for Multinational Adaptive Testing, Anthony R. Zara,Pearson

5:10 PM-5:35 PM From Reliability to Validity: Expanding Adaptive Testing Practice to Find the Most Valid Score for Each Test Taker, Steven L. Wise, NWEA

Item Selection and Analysis (individual papers), Chair: Otto B. Walter, Bielefeld University

3:00 PM-3:25 PM A Computerized Adaptive Test in Microsoft Excel with Different Item Selection Algorithms, Gilberto Sassi* & Mariana Cúri, University of São Paulo

3:30 PM-3:55 PM Shadow Testing for Small Testing Programs, Wallace Judd, Authentic Testing Corporation

4:00 PM-4:25 PM A Heuristic of CAT Item Selection Procedure for Testlets, Yuehmei Chien*, Chingwei David Shin, & Walter Denny Way, Pearson

4:40 PM-5:05 PM Detecting Differential Item Functioning between Conventional and Computerized Adaptive Testing: A Monte Carlo Study, Barth B. Riley*, Chestnut Health Systems, Adam C. Carle*, University of Cincinnati School of Medicine

5:10 PM-5:35 PM Practitioners’ Approach to Identifying Item Drift in CAT, Huijuan Meng* & Susan Steinkamp, Pearson, Paul Jones & Joy Matthews-Lopez, National Association of Boards of Pharmacy

5:40 PM-6:05 PM Impact of Item Drift on Candidate Ability Estimation, Sarah Hagge & Ada Woo*, National Council of State Boards of Nursing

Developing an Adaptive Testing Program (individual papers), Chair: Huan Wang, CTB/McGraw-Hill

3:00 PM-3:25 PM Improving a Short CAT Verification Test: Simulation versus Practice, Annette Maij-de Meij* & Lolle Schakel*, PiCompany

3:30 PM-3:55 PM A Monte Carlo Simulation Study on Measuring English Vocabulary Size via CAT, Wen-Ta Tseng, National Taiwan Normal University

4:00 PM-4:25 PM Characteristics of English Language Objective Test Items in School Certificate and Matriculation Examinations Conducted in Nigeria, Henry O. Owolabi, University of Ilorin

4:40 PM-5:05 PM Construction of an Adaptive Computerized System for Personality Assessment in Brazil, Carlos Henrique Sancineto da Silva Nunes*, Federal University of Santa Catarina, Maiana Farias Oliveira Nunes, Federal University of Rio Grande do Sul, Alexander Reinhard Rudolf Manfroi Melzer, Cássia Roettgers, Fábio Cesar Ariati, Luciane Guisso, & Micheli Sotili, Federal University of Santa Catarina

5:10 PM-5:35 PM A System for Creativity Assessment in Divergent Productions of Metaphors, Ricardo Primi*, University of São Francisco, Tatiana de Cássia Nakano, Pontificial Catholic University of Campinas

5:40 PM-6:05 PM The Importance of IRT and CAT, Madan L. Dutt, Institute of Banking Personnel Selection

Tuesday October 4th

Keynote Presentations

Tuesday, 8:30 AM – 10:00 AM Room: Merrill Hall, Chair: Cliff Donath, The Donath Group
 
8:30 AM – 9:15 AM, Innovative Items for Computerized Adaptive Testing Kathleen Scalise, University of Oregon, USA
 
9:15 AM – 10:00 AM, Designing Adaptive Tests with Rule-Based Item Generation Heinz Holling, Muenster University, Germany

 

Practical Challenges in CAT: Problems and Solutions to Test Calibration, Assembly, Administration and Monitoring Item Bias (invited symposium)
Organizer & Chair: Krista Breithaupt, Medical Council of Canada

10:15 AM – 10:40 AM Assessing the Impact of IRT Calibration Designs with Complex Clinical Decision Making Cases: An Application to the Medical Council of Canada’s Qualifying Examination Part I, André F. De Champlain, National Board of Osteopathic Medical Examiners

10:45 AM – 11:10 AM Multi-Stage Adaptive, Linear and Item Bundle Models for Administration: Possible Benefits in Item Usage, Comparability of Forms and Score Precision for a National Examination of Medical Knowledge, Timothy J. Wood*, University of Ottawa, Krista Breithaupt, Medical Council of Canada

11:15 AM – 11:40 AM Automated Systems and Algorithms Used in the Assembly of MST, Oliver Zhang, American Institute of Certified Public Accountants
 
11:45 AM – 12: 10 AM Monitoring Differential Item Functioning in CAT: An Extension of the CATSIB Procedure, Hollis Lai, University of Alberta
 
 
Issues Where Theory and Practice Meet (invited symposium), Organizer & Chair: Angela Verschoor, Cito
 
10:15 AM – 10:40 AM Item Selection Methods Based on Multiple Objective Approaches for Classification of Respondents Into Multiple Levels, Maaike van Groen*, Cito, Theo Eggen Cito/University of Twente, Bernard Veldkamp RCEC/University of Twente
 
10:45 AM – 11:10 AM Bayesian Models for Computerized Adaptive Testing, Bernard Veldkamp, RCEC/University of Twente
 
11:15 AM – 11:40 AM The Use of Decision Trees for Adaptive Item Selection and Score Estimation, Barth B. Riley*, Chestnut Health Systems, Rodney R. Funk, Chestnut Health Systems, Matthew Finkelman, Tufts University School of Dental Medicine, Michael L. Dennis, Chestnut Health Systems, Richard D. Lennox, Chestnut Global Partners
 
11:45 AM – 12: 10 AM A Test Assembly Model for Multistage Testing Using Simulations, Angela Verschoor*, Cito, Ingrid Radtke, VOX, Theo Eggen, Cito/University of Twente
 
 
Medical Applications and Innovative Modeling (individual papers) Chair: André F. De Champlain, National Board of Osteopathic Medical Examiners
 
10:15 AM – 10:40 AM Development of a Lower Extremity Physical Function CAT Bank, Man Hung* & Charles L. Saltzman, University of Utah
 
10:45 AM – 11:10 AM High-throughput Health Status Measurement using CAT in the Era of Personal Genomics: Opportunities and Challenges, Eswar Krishnan, Stanford University
 
11:15 AM – 11:40 AM Multidimensional Adaptive Testing for Likert Scales, Jonas M. Neuhengen* & Alan D. Mead*, Illinois Institute of Technology
 
11:45 AM – 12: 10 AM Penalized Spline Regression for Item Response Modeling and Adaptive Testing, Otto B. Walter, Bielefeld University
 
 
Implementation Aspects of CAT (individual papers), Chair: Xi Diao, CTB/McGraw-Hill
 
10:15 AM – 10:40 AM A Model for CAT Development, Nathan A. Thompson, Assessment Systems Corporation
 
10:45 AM – 11:10 AM How Can We Spread CAT?, Nathan A. Thompson*, Assessment Systems Corporation, Larry Rudner*, GMAC, Gage Kingsbury*, NWEA, Darrin Grelle*, PreVisor, Cliff Donath*, The Donath Group
 
11:15 AM – 11:40 AM Psychological Aspects of CAT: How Test-takers Feel about CAT, Tetsuo Kimura* & Keizo Nagaoka*, Waseda University
 
11:45 AM – 12: 10 AM Perceptions of Fairness and Opportunity to Perform on CAT in Personnel Selection, Sara L. Gutierrez*, PreVisor CAT
 
 

Invited Panel Discussion 1:30PM-2:30PM

Implementing CAT in Large Scale Testing Programs: Lessons Learned from a Variety of Perspectives (invited panel discussion), Organizer & Chair: Richard J. Patz, CTB/McGraw-Hill

Uniform Certified Public Accountant Examination, Craig N. Mills, American Institute of Certified Public Accountants

US Medical Licensing Exam, Ronald J. Nungester, National Board of Medical Examiners

Oregon Student Assessment Program, Anthony Alpert, Smarter Balanced Assessment Consortium

Armed Services Vocational Aptitude Test Battery, James McBride, Renaissance Learning

 

Breakout Sessions

How We Used CAT to Create a 33-Instrument Neurological Health Battery for Ages 3-85 Which Can Be Administered in Under 2 Hours (invited symposium), Organizer & Chair: Richard Gershon, Northwestern University

2:45 PM – 3:10 PM Introduction to the NIH Toolbox for the Assessment of Neurological and Behavioral Function, Richard Gershon, Northwestern University

3:15 PM – 3:40 PM The Development of a CAT to Assess Picture Vocabulary 3:3 Skills in English and Spanish, Richard Gershon, Northwestern University

3:45 PM – 4:10 PM The Development of an Oral Reading Fluency CAT, Dan Mungas, University of California at Davis

4:15 PM – 4:40 PM How We Assess 12 Areas of Emotional Health in under 15 Minutes using CAT

Item Calibration and Test Scoring (individual papers), Chair: Xinhui Xiong, CTB/McGraw-Hill

 
2:45 PM – 3:10 PM Recalibrating Items Using Adaptively Administered Response Data, Kyung T. Han*, Fanmin Guo, Eileen Talento-Miller, & Lawrence M. Rudner, GMAC
 
3:15 PM – 3:40 PM Adaptive Item Calibration and Norming: Unique Considerations of a Global Deployment, Alexander R. Schwall* & Evan F. Sinar*, Development Dimensions International
 
3:45 PM – 4:10 PM Differential Specifications of the Bi-factor Model and Improved Precision of Student Ability Scores in Computer Adaptive Tests, Yaacov Petscher, Florida Center for Reading Research
 
4:25 PM – 4:50 PM Automated Scoring in Computer Adaptive Testing: Examining the Impact of MC-only and CR-only Item Pools on the Measurement of Student Ability, Sue Lottridge* & E. Matt Schulz*, Pacific Metrics Corporation
 
4:55 PM – 5:20 PM The Effect of Fitting a Unidimensional IRT Model to Multidimensional Data in Content-Balanced Computerized Adaptive Testing, Tian Song*, Pearson, Mark D. Reckase*, Michigan State University
 

Developing CAT Systems (individual papers), Chair: Nathan A. Thompson, ASC

2:45 PM – 3:10 PM Rapid CAT Prototyping with OSCATS: The Open-Source Computerized Adaptive Testing System, Michael J. Culbertson*, University of Illinois
 
3:15 PM – 3:40 PM The Development of Concerto: An Open Source Online Adaptive Testing Platform, Michael S. Kosinski*,John N. Rust, & Tomoya Okubo, University of Cambridge
 
3:45 PM – 4:10 PM Haskell Possibilities for CAT, Xavier Lange*, TUREUS Inc., Rense Lange*, Integrated Knowledge Systems
 
4:25 PM – 4:50 PM Exploring the Use of Internet-Based, Computer Adaptive Tests as Formative Assessment Tools, Damon Bryant*, Adaptive Assessment Services/Humana- LifeSynch, Leonard Bethea, Adaptive Assessment Services, Joseph Hurston* & William Thornton, Dynamic Assessments
 
4:55 PM – 5:20 PM Development of an Internet-Based, Multidimensional Computer Adaptive Testing Platform, Damon Bryant*, Adaptive Assessment Services/Humana- LifeSynch, Kenneth James, Jr., Adaptive Assessment Services

Presidential Address

Tuesday, 5:15 PM – 6:00 PM Room: Merrill Hall
Chair: Wim J. van der Linden, CTB/McGraw-Hill
Theo J. H. M. Eggen, President IACAT
 
Conference Reception, Poster Presentations, and Bonfire
Tuesday, 7:00 PM – 10:00 PM Room: Woodlands North and South and BBQ Area (adjacent)
 
Poster Presentations
7:00 PM – 9:00 PM
 
Some Surprising Dynamics of “Technology Enhanced” Item Types: Taking Additional Time is Associated with Increased Student Performance on Some Types of Items, while Decreasing Performance on Others
Rense Lange*, Integrated Knowledge Systems, David Stevens & Paul Schwarz, Symphony Learning
 
The Effect of Overlap between a Routing Test Information Function and Population Ability Distribution in a Multistage Adaptive Test, J. Carl Setzer*, GED Testing Service, Erika Hall, Pearson
 
Theta Estimation Stability as a Function of Items Attempted, Matthew T. Schultz*, McCann Associates, Nathan A. Thompson*, Assessment Systems Corporation, David Vestal*, West Chester University of Pennsylvania
 
Pre-Operational CAT Simulation, Lianghua Shu* & Wim J. van der Linden, CTB
 
The Impact of Individual Differences across the Globe on Difficulty Parameter Ratings, Darrin Grelle* & Jolene Meyer, SHL
 
Facilitating Variable-Length Computerized Classification Testing in Instructional Contexts via Automatic Racing Calibration Heuristics, Andrew F. Barrett* & Theodore W. Frick*, Indiana University
 
Automated Assembly of Parallel Test Forms for Cognitive Diagnostic Models, Chuan-Ju Lin*, National University of Tainan
 
Creative Performance of Brazilian’s Children: Analysis of Rasch Item Maps, Tatiana de Cássia Nakano*, Pontificial Catholic University of Campinas, Ricardo Primi*, University of São Francisco
 
The National Internet Based English Test in Korea, Haedong Kim*, Hankuk University of Foreign Studies
 
Development of Items Databank for an Emotional Perception Test, Fabiano Koich Miguel*, State University of Londrina, Ricardo Primi*, University of São Francisco
 
Evaluation of an Internet-Based, Computer Adaptive Testing Algorithm, Damon Bryant*, Adaptive Assessment Services/Humana-LifeSynch, Larry Davis, Adaptive Assessment Services
 
Effectiveness of Incorporating Prior Information in the First Item Selection, Yong Luo, University of Maryland, Huan Wang, CTB/McGraw-Hill 

 

Wednesday October 5th

Keynote Presentations Chair: Theo J. H. M. Eggen, Cito/University of Twente

 

Self-Organizing Adaptive Learning Tools: An Application of Elo-Based Computerized Adaptive Testing, Han van der Maas, University of Amsterdam

Adaptive Test-Item Calibration, Wim J. van der Linden, CTB/McGraw-Hill 

 

Breakout Sessions

 

Advances in Multidimensional Adaptive Testing (invited symposium), Organizer & Chair: Andreas Frey, Friedrich-Schiller-University 

10:15 AM – 10:40 AM Constraining Item Exposure in Multidimensional Adaptive Testing, Qi Diao*, Wim J. van der Linden, & Hao Ren, CTB

10:45 AM – 11:10 AM Application of the Bifactor Model to Computerized Adaptive Testing, Dong Gi Seo, Michigan Department of Education

11:15 AM – 11:40 AM Computerized Adaptive Testing with the Conjunctive MIRT Model, Chun Wang & Hua-Hua Chang*, University of Illinois

11:45 AM – 12: 10 PM Multidimensional Adaptive Testing Environment (MATE) - Software for the Implementation of Computerized Adaptive Tests, Ulf Kröhne*, German Institute for International Educational Research


 

Impact and Control of Response Times (individual papers), Chair: Bernhard P. Veldkamp, University of Twente

10:15 AM – 10:40 AM Evaluation of Response Latency Times in a Nursing Assessment, Jason L. Meyers* & Julie A. Miles*, Pearson

10:45 AM – 11:10 AM The Impact of Testing Administration Time Expiration on CAT Ability Estimation, Jason L. Meyers* & Chingwei (David) Shin*, Pearson

11:15 AM – 11:40 AM Investigating the Effect of Item Response Times on Item Properties, Shu-chuan Kao* & Jerry Gorham*, Pearson

11:45 AM – 12: 10 PM Speededness and Adaptive Testing, Xinhui Xiong* & Wim J. van der Linden, CTB

 

Multistage Adaptive Testing (individual papers), Chair: Kentaro Yamamoto, ETS

10:15 AM – 10:40 AM A Two-Stage Procedure for Computerized Adaptive Mixed-Format Tests, Shuyan Sun, University of Cincinnati

10:45 AM – 11:10 AM Automatic on-the-fly Assembly for Computerized Adaptive Multistage Testing, Yi Zheng* & Hua Hua Chang, University of Illinois

11:15 AM – 11:40 AM Improving Proficiency Estimates through Reconfiguring Test Booklets in a Two-stage Adaptive Test Design, Brenda Siok-Hoon Tay-Lim*, UNESCO Institute for Statistics Jinming Zhang, University of Illinois

11:45 AM – 12: 10 PM The Effect of Differential Item Functioning on Multistage Testing, Yue Zhao, The Hong Kong Institute of Education

 

 

Intended and Unintended Multidimensionality in CAT: Some Issues and Findings (invited symposium), Organizer & Chair: Mary Pommerich, DMDC

1:30 PM – 1:55 PM A New CAT-ASVAB Scoring Framework, Based on Multidimensional Item Response Theory, Dan Segall, DMDC

2:00 PM – 2:25 PM Multidimensional CAT Item Selection Methods for Domain, Scores and Composite Scores with Item Exposure Control, Lihua Yao, DMDC

2:30 PM – 2:55 PM The Story of Assembling Objects: What Happens When Multidimensionality Persists in a Unidimensional CAT Environment, Mary Pommerich, DMDC

3:00 PM – 3:25 PM The Story of Assembling Objects Continued: Investigations into Sources and Solutions for Unintended Multidimensionality in CAT Data, Mary Pommerich, DMDC

 

Recent Advances in Multistage Testing Research and Applications at ETS (invited symposium), Organizer & Chair: Duanli Yan

1:30 PM – 1:55 PM Multistage Classification Testing: Using Bayesian Sequential Analysis to Make Multiple Classification Decisions, Charlie Lewis* & Robert Smith, ETS

2:00 PM – 2:25 PM Optimal Design and Scoring for Multistage Testing, Duanli Yan, ETS

2:30 PM – 2:55 PM A Design Process for Multistage Tests, Tim Davey* & Fred Robin, ETS

3:00 PM – 3:25 PM The International Literacy Assessment: PDQ and PIAAC, Kentaro Yamamoto, ETS

Statistical Methods in CAT (individual papers), Chair: Kyung Han, GMAC

1:30 PM – 1:55 PM Detection of Test Collusion in Computerized Adaptive Testing, Dmitry I. Belov, Law School Admission Council

2:00 PM – 2:25 PM Optimizing the Efficiency of Stratification Procedures with Exposure Control in Computerized Adaptive Testing, Hsiu-Yi Chao*, Jyun-Hong Chen, & Shu-Ying Chen, National Chung-Cheng University

2:30 PM – 2:55 PM Controlling General Test Overlap in Variable Length Computerized Adaptive Testing, Jyun-Hong Chen* & Shu-Ying Chen*, National Chung-Cheng University

3:00 PM – 3:25 PM Nine Outcomes of a MIMI CAT, Anne Thissen-Roe, Kronos