Psychological Science 2018, (3) 720-726 DOI:     ISSN: 0412-1961 CN: 21-1139/TG

Current Issue | Archive | Search                                                            [Print]   [Close]
Information and Service
This Article
Supporting info
PDF(0KB)
[HTML]
Reference
Service and feedback
Email this article to a colleague
Add to Bookshelf
Add to Citation Manager
Cite This Article
Email Alert
Keywords
adaptive multi-group testing for cognitive diagnosis
selection method
the safety of testing
time consuming per person
Authors
LUO Fen
YU Xiao-Qiang
DING Shu-Liang
XIONG Jian-Hua
PubMed
Article by Luo,f
Article by Yu,X.Q
Article by Ding,S.L
Article by Xiong,J.H

The Design and Selection Strategies of Adaptive Multi-group Testing for Cognitive Diagnosis

Abstract

It took incredible investment of time and effort to construct item bank. Selection strategy, which is the most significant component of Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT), should react quickly and pay attention to the utilization of item bank. The two widely used item selection methods in CD-CAT are Shannon Entropy (SHE) and Posterior Weighted Kullback-Leibler (PWKL). The characteristic of SHE method and PWKL method is higher classification accuracy, but the utilization rate of item bank is uneven. Learning from the idea of on-the-fly multistage adaptive testing, a new test named adaptive multi-group testing for cognitive diagnosis (AMST) was proposed. AMST is composed of several groups and each group has multiple items which were assembled by the interim knowledge state and its upper and lower bounds. The combination of AMST and SHE named AMST-SHE and AMST coupled with PWKL named AMST-PWKL, AMST-SHE and AMST-PWKL are the optimal design for AMST. Based on the Deterministic Inputs, Noisy-and-gate, a simulation study was operated to investigate the efficiency of the AMGT-SHE method and the AMGT-PWKL method compared with the SHE method, the PWKL method and the Random selection method for four item pools with different structures. Pattern correct rate, test length, average exposure rate and time consuming per person were calculated. Suppose that the attributes are mutually independent and the number of attributes was 5. Test length was fixed to 25, and the size of the item pool was fixed to 300. Variable length test stops when the largest posterior probability of knowledge state was not smaller than 0.9 and the second largest was larger than 0.1. The Monte Carlo simulation results showed that (1) the pattern correct rate and time consuming of the AGMT-SHE method were better than those of the AGMT-PWKL method, but the average exposure rate was opposite. For the AGMT-SHE method and the AGMT-PWKL method, the simpler the item types in item bank, the higher the pattern correct rate; (2) when the item types are rich in item bank, the average exposure rate and time consuming of the AMGT-PWKL method and the AMGT-SHE method are far better than those of the PWKL method and SHE method, especially on time consuming, the former is one-ninth of the latter, but test length would be increased; (3) the items of initial stage, which came from each column of the reachability matrix R replaced the random selection, contributes to improve pattern correct rate. AGMT, structured shadow pool by lattice theory, which combined AGMT - PWKL and AGMT - SHE, compared with the PWKL method and the SHE method, when item types are rich, at the price of increasing test length and losing a little pattern correct rate, it can greatly improve the uniformity of item bank and it is greatly beneficial for high-risk test. Paid attention to security in the test, the AMGT - PWKL method performed better and Paid attention to the item bank security and accuracy of the test, we should adopt the AMGT - SHE method. The AGMT-SHE method and AGMT-PWKL method are the locally optimal solution, not the globally optimal solution, so it can satisfy the high response speed request of CAT.

Keywords adaptive multi-group testing for cognitive diagnosis   selection method   the safety of testing   time consuming per person  
Received 2016-10-10 Revised 2017-09-12 Online: 2018-05-20 
DOI:
Fund:
Corresponding Authors: Shu-Liang DING
Email: ding06026@163.com
About author:

References:
Similar articles
1..Item Selection Strategies and Initial Items Selection Methods of CD_CAT[J]. Psychological Science, 2013,36(2): 469-474
2..Item Selection Methods for Balancing Test Efficiency with Item Bank Usage Efficiency in CD-CAT[J]. Psychological Science, 2014,37(1): 212-216
3.Guang MingLI Xin GuangWANG shan HU.A CD-CAT Item Selection Method Containing Strategy, Knowledge State and Ability[J]. Psychological Science, 2018,(2): 459-465

Comment for this article:

Copyright by Psychological Science