Journal of Psychological Science ›› 2022, Vol. 45 ›› Issue (4): 998-1007.

Previous Articles     Next Articles

DIF Detection Methods and Its Application of Polytomously Scored Items under the Framework of Cognitive Diagnosis

  

  • Received:2020-04-06 Revised:2020-11-26 Online:2022-07-20 Published:2022-07-20

认知诊断框架下多级评分题目的DIF检测方法及其应用

李秋云1,2,蔡艳2,汪大勋2,涂冬波2   

  1. 1. 常州大学
    2. 江西师范大学
  • 通讯作者: 蔡艳

Abstract: Currently, polytomous cognitive diagnosis test has attracted more and more researchers’ attention because it combines the advantages of polytomous items that can provide researchers with more information and the characteristics of cognitive diagnostic tests that can reflect the participants' inherent knowledge structure. However, some basic problems like differential item functioning (DIF) have not been solved, which greatly limits the practical application of the polytomous cognitive diagnosis test. Therefore, the main purpose of this study is to introduce several commonly used polytomous DIF detection procedures into polytomous cognitive diagnostic test to fill the gaps in this field. Firstly, the concept of DIF in the field of polytomous cognitive diagnosis test was defined and four DIF detection procedures include the mantel test, the logistic discriminant function analysis (LDFA), the likelihood ratio test (LR test), and Wald test were extended to polytomous cognitive diagnosis. Then simulation experiment was explored to investigate the performance of these polytomous DIF detection procedures in indetecting the uniform DIF under the estimation of four reduced models of seq-DINA, seq-DINO, seq-ACDM, seq-RRUM and a saturation model of seq-GDINA. In addition to the differences in the estimation model, some independent variables also included in the simulation experiment like dif size, sample size, matching variables, and the proportion of DIF items. Two levels of DIF size were .05 and .1. Two levels of sample size were 500 and 1000 examinees per group. Two levels of matching variables mainly for LDFA and the mantel test method were with KS as the matching variable and with the total score as the matching variable. Three levels of the proportion of DIF items were 10%, 20% and 30%. Finally, an empirical study was implemented to show the practical application of each method. The results are as follows: 1) Whether in the reduced model or the saturated model, each DIF detection procedures all can effectively detect the uniform DIF in the polytomous cognitive diagnostic tests, and the performance of each procedure is minimally affected by the model. 2) Two model-based DIF tests: Wald test and LR test have better Type I error control than two non-model-based DIF tests: LDFA method and the mantel test method. And within all methods, Wald test has the best Type I error control. 3) There are two matching variables for LDFA and the mantel test method. The mantel test and LDFA method with KS as the matching variable have higher power and lower type I error than the total score. 4) Among the other experimental conditions, DIF size has the greatest influence on the performance of DIF detection, followed by the size of the group, and finally the proportion of DIF items. From the method point of view, the mantel test method and LDFA method with total score as the matching variable are most affected by the changes of experimental conditions, while the other methods are less affected by the changes of experimental conditions. All in all, this article provided a preliminary definition of DIF in the field of polytomous cognitive diagnosis test, and extended four common used polytomous DIF detection procedures to the field of polytomous cognitive diagnosis. But, research of DIF in the field of polytomous cognitive diagnosis test is still insufficient, it is still needed for more in-depth research.

Key words: polytomous item, cognitive diagnosis, differential item functioning(DIF)

摘要: 本文对多级计分认知诊断测验的DIF概念进行了界定,并通过模拟实验以及实证研究对四种常见的多级计分DIF检验方法的适用性进行理论以及实践性的探索。研究结果表明:四种方法均能对多级计分认知诊断中的DIF进行有效的检验,且各方法的表现受模型的影响不大;相较于以总分为匹配变量,以KS为匹配变量时更利于DIF的检测;以KS为匹配变量的LDFA方法以及以KS为匹配变量的曼特尔检验方法在检测DIF题目时有着最高的检验力。

关键词: 多级计分, 认知诊断测验, 项目功能差异