The existing studies suggested that item quality is closely relevant to the number of attributes required by an item, item parameters, and the prior distribution of attribute patterns in cognitive diagnostic assessment. Several studies focused on the design of Q-matrix and showed that items required only one attribute are important for classification. There are some works provided two basic set of item discrimination index to measure discriminatory power of an item. The first one is based on descriptive measures from classical test theory, such as the global item discrimination index, and the second index is based on information measures from item response theory, including cognitive diagnosis index (CDI), attribute discrimination index (ADI), modified CDI and ADI. Results showed a strong relationship between these indices and the average correct classi?cation rates of attributes. But their relationship to the indices may change as a function of the distribution of attributes.
There lacks an item quality index as a measure of item’s correct classification rates of attributes. The purpose of this study was to propose an item discrimination index as a measure of correct classification rate of attributes based on Q-matrix, item parameters, and the distribution of attributes. Firstly, an attribute-specific item discrimination index, called item expected attribute matched rate (EAMR), was introduced. Secondly, a heuristic method was presented using EAMR for test construction.
The first simulation study was conducted to evaluate the performance of EAMR under the deterministic input noisy “and” gate (DINA) model. Several factors were manipulated for five independent attributes in this study. Four levels of correlation between latent attributes, ρ=.00, ρ=.50, ρ=.75, and ρ=.95, were considered. Items were categorized into five groups according to the number of attributes measured by each item. Item discrimination power was set at three levels, high, medium, and low. High level meant relatively smaller guessing and slip parameters, which were randomly generated from a uniform distribution U(.05,.25). Medium-level and low-level item parameters were randomly drawn from uniform distributions U(.05, .40) and U(.25, .45). Next, 1000 items were simulated with the q-vector randomly selected from all possible attribute patterns measuring at least one attribute. Results showed that the new index performed well in that their values matched closely with the simulated correct classification rates of attributes across different simulation conditions.
The second simulation study was conducted to examinee the effectiveness of the heuristic method for test construction. The test length was fixed to 50 and simulation conditions are similar as used in the first study. Results showed that the heuristic method based on the sum of EAMRs yielded comparable performance to the famous CDI.
These indices can provide test developers with a useful tool to evaluate the quality of the diagnostic items. The attribute-specific item discrimination index will provide researchers and practitioners a way to select the most appropriate item and test that they want to measure with greater accuracy. It will be valuable to explore the applications and advantages of using the EAMR for developing item selection algorithm or termination rule in cognitive diagnostic computerized adaptive testing.