关键词: 序贯监测程序, 个人拟合指标, 计算机化自适应测验, 测验安全, 能力估计精度

Abstract: Computerized adaptive testing (CAT) was proposed to measure the trait levels of examinee with great precision than conventional tests by building an individualized test for each examinee. Test items are selected sequentially, according to the current performance of an examinee. So, the test can be tailored to each examinee’s trait level, thus matching the difficulties of the items to the examinee being measured. In the field of the CAT research, the security of item pool is greatly important for the fairness of test and the protection of item pool. So far, many studies have proposed different kinds of item exposure control techniques to limit the maximum exposure ratio of each item, such as a-stratified method, Sympson-Hetter (SH) strategy, restricted maximum information strategy, shadow test approach and so on. However, only emphasizing the item exposure control is far from being enough. This method cannot prevent some people to steal the items from the exam room. Actually, when a CAT item pool has been used in practice for a while, some items may be compromised due to either item overexposure or some contrived reasons. When the number of compromised items increases, both the reliability and validity of the continuous testing system and the estimate accuracy of examinees’ ability will be damaged. In order to guarantee and maintain the security of the CAT system, sequentially monitoring procedure (SMP; Zhang, 2013) was developed to detect whether the performance or statistical characteristic of individual items has gone through any significant changes during their lifetime in an item pool or not. Nevertheless, , the rate of Type I errors may occur when using the SMP in the process of monitoring. Moreover, there is no concern about how compromised items could affect the ability estimate accuracy in the Zhang’s study. Thus, this paper combining the person fit index which was calculated by Bayesian Method with the sequentially monitoring procedure, and proposed a new approach called SMP_PFI method to decrease the rate of Type I errors. The rationale and operating steps of the new method were illustrated. The new method could not noly examine whether the items which had already been flagged by SMP were truly compromised or not, but balance the relationship between ability estimate accuracy and the number of items that be freezed in the item pool. The simulation studies were conducted to investigate the performance of SMP_PFI method. Particularly, the correct response probability of examinees who had preknowledge of the receiving items was set at 0.8 or 0.9 artificially in Zhang’s study, which ignored the interaction between the examinee themselves and the property of items. So, we adopted the modified 3PL model to simulate responses when examinee received compromised items. Experimental results showed that the new method could effectively reduce the rate of Type I errors, and users also could set the criteria of CPFI index at any reasonable level to balance the relationship between ability estimate accuracy and the number of items that be freezed in the item pool.

Key words: sequentially monitoring procedure, person fit index, computerized adaptive testing, test security, ability estimate accuracy