关键词: 认知诊断, 层级一致性指标, 多级计分, 拓展的层级一致性指标

Abstract: In educational practice of cognitive diagnostic assessment (CDA), it is crucial to validate the reasonability of the hierarchical structure of attributes, because it can affect the quality of CDA and the accuracy of classification of examinees directly. Several indices that validate the reasonability of the hierarchical structure of attributes have been developed by researchers. Based on attribute hierarchy method (AHM), Cui, Leighton, Gierl and Hunka (2006）developed hierarchy consistency index (HCI) to detect the misfit of examinees; also Ding, Mao, Wang and Luo (2011) modified the HCI, then developed a new index——Modified HCI（MHCI). In addition, Guo (2012) proposed the hierarchy misfit index (HMI) to detect the misfit of examinees’ misfit responses. Although these indexes can be used to detect misfit, all of these indexes are suitable for dichotomous. For polytomous items, in order to validate the reasonability of attribute hierarchy (AH), researchers, in general, transform these polytomous items into dichotomous items according to some prespecified rules, then calculate the HCI (Kang, Wu, Chen, & Zeng, 2015; Kang, Xin, & Tian, 2013) or MHCI (Ding et al., 2012). However, it will lose some details when transform polytomous items into dichotomous, therefore produce larger errors (Ding, Wang, & Luo, 2014) and underestimate the consistence of AH. The purpose of this study is extending the Modified HCI (MHCI) to a new HCI that suitable for both dichotomous and polytomous items, and we name the new HCI as generalized hierarchy consistency index (GHCI). To evaluate the suitability of GHCI, a simulation study and an empirical study are employed. For the simulation study, two independent variables are manipulated: type of attribute hierarchy (AH) and proportion of transformation. The AH has 4 levels (linear, convergent, divergent, and unstructured) and the proportion of transformation has 5 levels (0%，60%, 67%, 75%, 100%). The control variables are the number of attributes, K = 5, and the number of examinees, N = 2000. Results show that: (1) Both GHCI and MHCI are affected by type of AH, the linear AH has the best GHCI and MHCI, then is convergent AH, after that is divergent AH and the unstructured AH is worst. (2) GHCI has larger means than MHCI regardless of AH and proportion of transformation. In sum, lower accuracy are found when MHCI is used to calculate the consistence of AH for polytomous items, while GHCI produced high accuracy. To test the practical feasibility of GHCI, an empirical study is conducted. Results show the consistent pattern with simulation.

Key words: cognitive diagnostic assessment, hierarchy consistency index, polytomous extension, generalized hierarchy consistency index