关键词: 认知诊断评估, 多分属性, 分类一致性, 分类准确性

Abstract: Classification consistency and accuracy are respectively deemed as significant criterions for evaluating the reliability and validity of test property in cognitive diagnostic assessment (CDA). Some indices, such as tetrachoric correlation, pattern-level classification consistency and accuracy indices, and attribute- and pattern-level classification consistency and accuracy indices have been introduced by several Psychometrians. However, all the existing indices focus on the scenario with dichotomous attributes. A CDA with polytomous attributes has received more and more attention in recent years due to additional diagnostic information that polytomous attributes can provide. This article first points out that the posterior probability distributions of attribute profiles and marginal posterior probability distributions of individual attributes under polytomous attribute scenario are rather different from those under dichotomous attribute scenario. Due to these distinctions, new classification consistency and accuracy indices that can represent the reliability and validity for a CDA with polytomous attributes are needed. This article proposes the attribute- and pattern-level classification consistency and accuracy indices based on binary method and develops procedures for the computation of the new indices specifically designed for CDAs with polytomous attributes. These indices make it possible to calculate the reliability and validity of CDAs with polytomous attributes. Simulation studies are conducted to evaluate the performance of the new indices using RPa-DINA model. In the simulation, the authors investigate the effects of five different factors: the hightest level of polytomous attributes, item quality, test length, number of polytomous attributes, and cutoff scores. Simulation results indicate that: (1) different values of cutoff score affect the performance of the reliability and validity indices. The RMSEs are almost similar when Ck and Cp are bounded between 0.1 and 0.6. Especially, the RMSE values are lowest when Ck and Cp are around 0.5 in most experiment conditions. So, the cutoff score taking 0.5 is recommended in practice. (2) the RMSEs of attribute- and pattern-level classification consistency and accuracy increase with the hightest level of polytomous attributes and the number of polytomous attributes. (3) the RMSEs decrease with item quality and test length. Finally, the authors discuss the essence of difference about the posterior probability distributions and marginal posterior probability distributions between dichotomous and polytomous attribute scenarios. Some practical suggestions for test construction are given and future research directions are proposed.

Key words: cognitive diagnostic assessment, polytomous attribute, classification consistency, classification accuracy