关键词: 认知诊断, 计算机化自适应测验, 多值评分, GPDM模型

Abstract: The development of cognitive diagnostic computerized adaptive test (CD-CAT) provides a new perspective for obtaining information about students’ mastery or nonmastery of a set of skills in the field of knowledge. In recent years, CD-CAT has received more and more attention in the field of educational evaluation and psychological evaluation. Effective item selection algorithm is the key to the success of CD-CAT system. To date, various item selection methods of CD-CAT have been proposed based on dichotomous cognitive diagnosis model (CDM). There are few researches on the item selection algorithm of polytomous CD-CAT (PCD-CAT). However, in educational assessment, psychological evaluation and many other disciplines, there are a lot of polytomously-scored data. In this article, the authors explored the CD-CAT item selection algorithm based on general polytomous diagnosis model, and proposed two PCD-CAT item selection algorithms, namely, maximum expected posterior distribution variance (EPV) and maximum expected distance (MED). The performances of the proposed item selection algorithms were evaluated through two simulation studies and compared with the KL, PWKL, and HKL algorithms in fixed-length and variable-length PCD-CAT. In the simulation experiment, the size of the item bank was 350, and the maximum score of each item was fixed at 4. The number of attributes was fixed to K = 7. In the first study, we manipulated three factors: the test length (5, 10, 15 and 20), item bank quality (high vs. low), and item selection algorithms (KL, PWKL, HKL, EPV and MED). The results of Study 1 showed that the EPV and MED consistently resulted in the highest attribute pattern recovery rate in all the simulation conditions. The results of Study 1 showed that the pattern correct classification rate (PMR) of EPV and MED was significantly higher than that of KL, PWKL and HKL methods. The EPV and MED had similar PMR under most experimental conditions, but when the test length was short (for example, 5 items), regardless of the quality of the item bank, the PMR of EPV was higher than that of MED. Under all conditions, the KL method had the lowest PMR rate, while the difference in PMR rates between PWKL and HKL was almost negligible. Study 2 investigated the performance of two proposed new item selection algorithms under the condition of variable-length PCD-CAT. In Study 2, when the maximum posterior probability of the attribute vector reaches a prespecified value, the test is terminated. Three factors were manipulated in the Study 2: prespecified termination values (0.6, 0.7, 0.8 and 0.9), item bank quality (high and low) and five item selection algorithms. The results of Study 2 showed that the average test lengths of EPV and MED were roughly similar, and significantly smaller than the average test lengths of KL, PWKL and HKL. Although the results are encouraging, there are still some future research directions that deserve further study, such as, (a) how to use both confirmatory CDM and exploratory CDM in diagnostic evaluation to better analyze data; (b) item calibration technology for cognitive diagnostic computerized adaptive test; (c) using computer to realize automatic scoring of polytomously-scored items.

Key words: cognitive diagnosis, computerized adaptive testing, polytomously-scored items, GPDM