关键词: CD-CAT, 可修改答案, 题目口袋, Stocking设计

Abstract: Combining cognitive diagnosis with computerized adaptive testing, cognitive diagnostic computerized adaptive testing (CD-CAT) aims to more efficiently and accurately diagnose examinees’ mastery status of a group of discretely defined skills, or attributes than paper & pencil tests. While it is a natural thing for examinees to review their answers and possibly change them in paper and pencil-based tests, the same thing is less common to happen in CD-CAT since it could deteriorate the measurement efficiency. The absence of review opportunities on operational CD-CAT creates a dilemma for test developers as examinees need to review and change answers during the test in order to achieve more accurate estimates of their true ability. Item Pocket(Han, 2013) is a method of reviewable computerized adaptive testing(RCAT). This method provides test takers with Item Pocket(IP) into which they can place items for later review and response change. Test takers can skip answering items by putting them in the IP. but the shortcoming of IP is that the capacity is not easy to control, if the capacity is too large that will results a comparatively large estimation error. Based on IP method, the study proposes a new IP method called modified IP(MIP), employing a new scoring method in IP. Compared with IP, stocking (1997) design cause greater restrictions for examinee behavior. In stocking design I, examinees are instructed in advance that they will be permitted to revise answers to fixed number of items, under stocking design II, the testing is divided into separately sections and examinees are informed in advance of testing that they will be permitted to revise answers to items only within a section. The advantage of design II is that it simultaneously restricts examinee control over the actual item presented because revised responses from previous sections influence the section of items in subsequent sections. CD-CAT is a further development of the CAT, but they are very different in some ways. In order to verify the above methods in Reviewable CD-CAT(RCD-CAT), two Monte Carlo simulation studies with different experimental conditions were conducted here, the interim and final states of knowledge were estimated using the maximum likelihood estimation (MLE) method, a group of 5,000 examinees were simulated for this study, and the tests were then created from an item pool of 300 items. These experimental conditions were cognitive diagnosis model(DINA and R-RUM), the number of attributes(5 and 7), item selection strategies(KL, PWKL, HKL and MPWKL), and the fixed test length CD-CAT (10 and 20 items respectively) . Monte Carlo simulation results show that: (1) When using the DINA model, MIP and IP methods had very similar classification accuracy , however, while using R-RUM model MIP method had higher classification accuracy than IP method. Furthermore, both MIP and IP had low classification accuracy than traditional CD-CAT; (2) Stocking design had a higher classification accuracy than the other methods in all simulations, and stocking design II was slightly better than the stocking design I. In a word, RCD-CAT is more consistent with traditional examination habits, in addition, it can also improve classification accuracy. This study will help to provide theory and method support for future research and practical application.

Key words: CD-CAT, Answer Change, Item Pocket, Stocking Design