Psychological Science 2018, (2) 453-458 DOI:     ISSN: 0412-1961 CN: 21-1139/TG

Current Issue | Archive | Search                                                            [Print]   [Close]
Information and Service
This Article
Supporting info
PDF(1190KB)
[HTML]
Reference
Service and feedback
Email this article to a colleague
Add to Bookshelf
Add to Citation Manager
Cite This Article
Email Alert
Keywords
Authors
WANG Wen-Xi
SONG Li-Hong
DING Shu-Liang
PubMed
Article by Wang,W.X
Article by Song,L.H
Article by Ding,S.L

An Item Discrimination Index and its Application in Cognitive Diagnostic Assessment on a Classification-Oriented View

wenyi Wang, ,Shu-Liang DING

Abstract

The existing studies suggested that item quality is closely relevant to the number of attributes required by an item, item parameters, and the prior distribution of attribute patterns in cognitive diagnostic assessment. Several studies focused on the design of Q-matrix and showed that items required only one attribute are important for classification. There are some works provided two basic set of item discrimination index to measure discriminatory power of an item. The first one is based on descriptive measures from classical test theory, such as the global item discrimination index, and the second index is based on information measures from item response theory, including cognitive diagnosis index (CDI), attribute discrimination index (ADI), modified CDI and ADI. Results showed a strong relationship between these indices and the average correct classi?cation rates of attributes. But their relationship to the indices may change as a function of the distribution of attributes. There lacks an item quality index as a measure of item’s correct classification rates of attributes. The purpose of this study was to propose an item discrimination index as a measure of correct classification rate of attributes based on Q-matrix, item parameters, and the distribution of attributes. Firstly, an attribute-specific item discrimination index, called item expected attribute matched rate (EAMR), was introduced. Secondly, a heuristic method was presented using EAMR for test construction. The first simulation study was conducted to evaluate the performance of EAMR under the deterministic input noisy “and” gate (DINA) model. Several factors were manipulated for five independent attributes in this study. Four levels of correlation between latent attributes, ρ=.00, ρ=.50, ρ=.75, and ρ=.95, were considered. Items were categorized into five groups according to the number of attributes measured by each item. Item discrimination power was set at three levels, high, medium, and low. High level meant relatively smaller guessing and slip parameters, which were randomly generated from a uniform distribution U(.05,.25). Medium-level and low-level item parameters were randomly drawn from uniform distributions U(.05, .40) and U(.25, .45). Next, 1000 items were simulated with the q-vector randomly selected from all possible attribute patterns measuring at least one attribute. Results showed that the new index performed well in that their values matched closely with the simulated correct classification rates of attributes across different simulation conditions. The second simulation study was conducted to examinee the effectiveness of the heuristic method for test construction. The test length was fixed to 50 and simulation conditions are similar as used in the first study. Results showed that the heuristic method based on the sum of EAMRs yielded comparable performance to the famous CDI. These indices can provide test developers with a useful tool to evaluate the quality of the diagnostic items. The attribute-specific item discrimination index will provide researchers and practitioners a way to select the most appropriate item and test that they want to measure with greater accuracy. It will be valuable to explore the applications and advantages of using the EAMR for developing item selection algorithm or termination rule in cognitive diagnostic computerized adaptive testing.

Keywords
Received 2017-01-05 Revised 2017-07-27 Online: 2018-03-20 
DOI:
Fund:
Corresponding Authors:
Email: viviansong1981@163.com
About author:

References:
Similar articles

Comment for this article:

Copyright by Psychological Science