Abstract:

From a psychological perspective, cognitive diagnostic models (CDMs) are divided into two basic categories called compensatory and noncompensatory diagnostic models depending on the interaction of skills. The interaction of skills in examinee’ item response behavior may sometimes be better modeled as disjunctive (e.g., psychological assessment) and other times be better modeled as conjunctive (e.g., mathematical assessment). In other words, the choice of mode of attribute interaction clearly depends on the diagnostic setting. The selection of an appropriate CDM is based on analyses of the cognitive interaction between the skills and the items on the test. It almost always requires consultation of the literature and close collaboration among psychometric and substantive experts, in addition to empirical checking and confirmation. In conjunctive condensation rule, hierarchy consistency index (HCI) or item consistency index (ICI) can be directly used to assess whether actual examinees’ or items’ response patterns match the expected response patterns. It should be noted that the HCI or the ICI cannot be used with disjunctive CDMs where the mastery of all the attributes measured by an item is not necessary for successful performance because of the assumption that high ability on one attribute can compensate for low ability on other attributes. This leads us to propose the new indices that are specifically designed to identify misfits of item response vectors relative to disjunctive models. HCI or ICI requires the assumption that, from a correct or incorrect response, one can infer that the examinee has or has not mastered all attributes required by the item. Such an inference is often unreasonable and it is likely to draw the possibly incorrect inference. Considering statistical inference is generally more precise than everyday inference, we introduce a consistency index based on hypothesis testing to help detect misfitting item response vectors under the disjunctive condensation rule. The new consistency index with the original ICI can be used in the selection and evaluation of conjunctive or disjunctive model for data analyses. We also proposed a method to estimate slipping and guessing parameters based on comparison of item responses. To investigate whether the new indices can work well under certain conditions, simulated data are generated with an independent structure using five attributes. Four important factors were included in the design of the simulation study: (a) cognitive diagnostic model including two conjunctive models and a disjunctive model; (b) the quality of test Q-matrix with the four error rates from 0 to .4 with step .1, (c) the quality of items, and (d) the number of examinees with N = 500 or 1,000. The results show that these indices can be used to evaluate cognitive assumptions, to assess the quality of test Q-matrix, to identify poor items with attribute misspecification and to estimate noise rate in response data. The new consistency index with original indices may provide better understanding of the nature of cognitive assumptions and they will help determine which psychometric model is most appropriate and interpretable for the intended diagnostic assessment.