Development of a Cognitive Diagnostic Model with Ability Covariate and Its Applications

doi:10.16719/j.cnki.1671-6981.20240423

Abstract

Abstract: The item response theory (IRT) is an important model to estimate the ability parameters, and the cognitive diagnosis model (CDM) is a vital model to diagnose the cognitive structure of the examinees. Because test item responses contain the information about the ability parameters and the knowledge states and their relationship, how to fully utilize the information to further enhance the accuracy of the knowledge states, becomes very important for helping to decrease the number of tests and test lengths.
Considering the strong relationship between higher-order DINA (HO-DINA) model and two-parameter logistic model (2PLM), this new statistical measurement model was proposed by regarding the general ability as higher-order ability and estimated by Markov Chain Monte Carlo. In the estimation of ability, the knowledge states are considered as response patterns in establishing the relationship between the ability and knowledge states. Five attributes were considered in the study. Test consists of two parts of items, one is fitted by 2PLM and the other is fitted by the DINA model. The simulation study examined the performance of the new model with four model parameter distributions and five different test lengths, and compared it with the DINA model and 2PLM respectively. In the analysis of Examination for the Certificate of Proficiency in English (ECPE) test data, the absolute fit indices, -2LL, AIC, BIC and DIC were provided as an example to evaluate the model-data fit.
The results show that: (a) The new proposed model in this study can obtain the ability parameters and cognitive structure parameters of the examinees in one test, and it has a good recovery for the estimation of attribute patterns, item parameters, and ability parameters. In particular, the accuracy of ability parameters has been greatly improved, and the MCMC algorithm is feasible; (b) The longer the CDM-based test length is, the higher the correct classification rate of knowledge state is, and the same is true for the ability estimation based on 2PLM; (c) Compared with lognormal distribution, the error of ability estimation is slightly smaller when the item discriminations followed from uniform distribution. When the slip parameters and the guessing parameters are small, the correct classification rate for attributes or attribute patterns are higher. In addition, the analysis of the ECPE data shows that the estimation accuracy of the new model is reasonable and has practical implications.
Because there is always a relationship between the levels of ability and the mastery of knowledge, the ability of estimation enriched in test items fitting by an item response theory model can provide information for the classification of knowledge state, and item only fitting the CDM can also indirectly provide the information to improve the precision of ability estimation. In addition, it does not need calibrate the attribute vector for all test items and not require all items fitted two models at the same time. The new model uses a general or high-order ability as bridge between item response theory and cognitive diagnostic models, utilizing information from different test items to improve the accuracy of abilities and knowledge, tates estimation.

Key words: ability, knowledge state, 2PLM, the DINA Model, MCMC

摘要： 新时代教育评价改革对评价的科学性、客观性和专业性提出了更高的要求。以一般（高阶）能力作为连接项目反应理论模型与认知诊断模型的桥梁,相互利用不同试题上的作答信息,开发出融入能力信息的认知诊断模型。模拟研究考查了新模型在四种参数分布和五种题量下的表现,并与DINA模型与2PLM进行了比较。结果显示：在相同题量下,新模型能力参数的均方误差优于2PLM的均方误差;新模型的模式判准率优于DINA模型的模式判准率;属性斜率越大,新模型的属性判准率和能力精度更高。在英语水平证书考试的实测数据上,新模型相对拟合指标优于DINA和HO-DINA等模型;新模型虽稍逊于2PLM,但两者所得能力与测验总分相关较高;新模型在属性2上分类准确性高于其他模型,分析发现新模型可利用能力信息提高Q阵中考查次数较少属性的分类准确性。

关键词: 能力, 知识状态, 2PL模型, DINA模型, MCMC算法

Song Lihong, Hu Haiyang, Wang Wenyi, Ding Shuliang, Yuan Siyu. Development of a Cognitive Diagnostic Model with Ability Covariate and Its Applications[J]. Journal of Psychological Science, 2024, 47(4): 947-958.

宋丽红, 胡海洋, 汪文义, 丁树良, 袁思玉. 融入能力信息的认知诊断模型开发与应用^*[J]. 心理科学, 2024, 47(4): 947-958.

References

[1] 丁树良, 罗芬, 汪文义, 熊建华. (2019). 0-1评分认知诊断测验设计. 江西师范大学学报(自然科学版), 43(5), 441-447.
[2] 丁树良, 汪文义, 杨淑群. (2011). 认知诊断测验蓝图的设计. 心理科学, 34(2), 258-265.
[3] 罗芬, 王晓庆, 蔡艳, 涂冬波. (2020). 基于基尼指数的双目标CD-CAT选题策略. 心理学报, 52(12), 1452-1465.
[4] 涂冬波, 蔡艳, 戴海琦, 丁树良. (2011). HO-DINA模型的MCMC参数估计及模型性能研究. 心理科学, 34(6), 1476-1481.
[5] 汪大勋, 涂冬波. (2021). 认知诊断计算机化自适应测量技术在心理障碍诊断与评估中的应用. 江西师范大学学报(自然科学版), 45(2), 111-117.
[6] 王璞珏, 刘红云. (2019). 让自适应测验更知人善选——基于推荐系统的选题策略. 心理学报, 51(9), 1057-1067.
[7] 韦来生. (2016). 贝叶斯统计. 高等教育出版社.
[8] 詹沛达, 潘艳方, 李菲茗. (2021). 面向“为学习而测评”的纵向认知诊断模型. 心理科学, 44(1), 214-222.
[9] 中共中央, 国务院. (2020-10-14). 中共中央国务院印发深化新时代教育评价改革总体方案. 人民日报.
[10] Akbay, L., & Kaplan, M. (2017). Transition to multidimensional and cognitive diagnosis adaptive testing: An overview of CAT. The Online Journal of New Horizons in Education, 7(1), 206-214.
[11] Ames, A. J., & Penfield, R. D. (2015). An NCME instructional module on item-fit statistics for item response theory models. Educational Measurement: Issues and Practice, 34(3), 39-48.
[12] Baker F. B.& Kim, S. H. (2004). Item response theory: Parameter estimation techniques CRC Press Parameter estimation techniques. CRC Press.
[13] Bao Y., Shen Y. W., Wang S. Y., & Bradshaw L. (2021). Flexible computerized adaptive tests to detect misconceptions and estimate ability simultaneously. Applied Psychological Measurement, 45(1), 3-21.
[14] Bradshaw, L., & Templin, J. (2013). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403-425.
[15] Chen, J. S., & de la Torre, J. (2018). Introducing the general polytomous diagnosis modeling framework. Frontiers in Psychology, 9, Article 1474.
[16] Dai B. Y., Zhang M. Q., & Li G. M. (2016). Exploration of item selection in dual-purpose cognitive diagnostic computerized adaptive testing: Based on the RRUM. Applied Psychological Measurement, 40(8), 625-640.
[17] de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353.
[18] Gelman A., Carlin J. B., Stern H. S., Dunson D. B., Vehtari A., & Rubin, D. B. (2013). Bayesian data analysis. CRC Press.
[19] Gitomer, D. H., & Yamamoto, K. (1991). Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28(2), 173-189.
[20] Givens, G. H., & Hoeting, J. A. (2013). Computational statistics. John Wiley & Sons, Inc.
[21] Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4), 301-321.
[22] Henson R. A., Templin J. L., & Willse J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191-210.
[23] Hong H., Wang C., Lim Y. S., & Douglas J. (2015). Efficient models for cognitive diagnosis with continuous and mixed-type latent variables. Applied Psychological Measurement, 39(1), 31-43.
[24] Hsu, C. L., & Wang, W. C. (2015). Variable-length computerized adaptive testing using the higher order DINA model. Journal of Educational Measurement, 52(2), 125-143.
[25] Huang, H. Y. (2017). Multilevel cognitive diagnosis models for assessing changes in latent attributes. Journal of Educational Measurement, 54(4), 440-480.
[26] Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
[27] Kang H. A., Zhang S. S., & Chang H. H. (2017). Dual-objective item selection criteria in cognitive diagnostic computerized adaptive testing. Journal of Educational Measurement, 54(2), 165-183.
[28] Lee, S. Y. (2017). Growth curve cognitive diagnosis models for longitudinal assessment(Unpublished doctoral dissertation). University of California, Berkeley.
[29] Ma, W. C., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275.
[30] Ma, W. C., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1-26.
[31] McGlohen, M., & Chang, H. H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40(3), 808-821.
[32] Patz, R. J. & Junker, B. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146-178.
[33] Rupp A. A., Templin J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications Guilford Press Theory, methods, and applications. Guilford Press.
[34] Sorrel M. A., Abad F. J., & Nájera P. (2021). Improving accuracy and usage by correctly selecting: The effects of model selection in cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 45(2), 112-129.
[35] Wang C., Chang H. H., & Douglas J. (2012). Combining CAT with cognitive diagnosis: A weighted item selection approach. Behavior Research Methods, 44(1), 95-109.
[36] Wang C., Zheng C. J., & Chang H. H. (2014). An enhanced approach to combine item response theory with cognitive diagnosis in adaptive testing. Journal of Educational Measurement, 51(4), 358-380.
[37] Wang S. Y., Yang Y., Culpepper S. A., & Douglas J. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57-87.
[38] Wang W. Y., Song L. H., Chen P., Meng Y. R., & Ding S. L. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457-476.
[39] Yan D. L., Mislevy R. J., & Almond R. G. (2003). Design and analysis in a cognitive assessment. ETS Research Report Series, 2003(2), i-47.
[40] Zhan P. D., Jiao H., Liao D. D., & Li F. M. (2019). A longitudinal higher-order diagnostic classification model. Journal of Educational and Behavioral Statistics, 44(3), 251-281.
[41] Zhan P. D., Ma W. C., Jiao H., & Ding S. L. (2020). A sequential higher order latent structural model for hierarchical attributes in cognitive diagnostic assessments. Applied Psychological Measurement, 44(1), 65-83.
[42] Zhang, S. S., & Chang, H. H. (2016). From smart testing to smart learning: How testing technology can assist the new generation of education. International Journal of Smart Technology and Learning, 1(1), 67-92.
[43] Zheng C. J., He G. R., & Gao C. L. (2018). The information product methods: A unified approach to dual-purpose computerized adaptive testing. Applied Psychological Measurement, 42(4), 321-324.