Leveraging Machine Learning Technology to Improve Accuracy and Efficiency of Identification of Enemy Item Pairs

Abstract

The interpretations of test scores in secure, high-stakes environments are dependent on several assumptions, one of which is that examinee responses to items are independent and no enemy items are included on the same forms. This paper documents the development and implementation of a C#-based application that uses Natural Language Processing (NLP) and Machine Learning (ML) techniques to produce prioritized predictions of item enemy statuses within a large item bank, which can then be followed by medical editor review of the prioritized predictions as part of an iterative process. An item bank of 4130 items from a large-scale healthcare specialist certification exam was used, in which it was assumed that many unidentified enemy pairs existed. For each pair of items, cosine similarities using TF-IDF weights were computed for the stem and answer text separately, with additional dichotomous classification variables added indicating content and existing enemy relationships. Each item pairs’ existing enemy status (enemy or non-enemy) was the dependent variable for the supervised ML model, the coefficients of which were then used to generate probabilities that a given pair of items were enemies. Medical editors reviewed prioritized lists of the actual versus predicted enemy relationships in an iterative fashion. Of the 700 untagged enemy item pairs reviewed, 666 were confirmed and tagged by editors as enemies (95.1% accuracy). Thus, this application was successful in allowing editors to efficiently identify the most egregious uncoded enemy item pairs in a large item bank. The ultimate goal of this research is to inform discussion about the potential for NLP and ML applications to greatly improve accuracy and efficiency of human expert work in test construction.