Bridging the Standard Setting Gap via Assessment Engineering

Abstract

Standard setting is the process of identifying the point(s) on a scale that serve to differentiate between individuals of distinct proficiency levels. While standard setting is ultimately a policy decision, most of the process is carried out by subject matter experts who are tasked with reconciling item-level or examinee-level information (e.g. content of the item, examinee data, performance data) with performance level descriptors, borderline candidate conceptualizations, etc. The ability of subject matter experts to recommend a cut score that accurately differentiates between levels of proficiency is directly impacted by the level of alignment between the conceptualization of the construct and the more concrete test or examinee content upon which they are making judgments. Assessment engineering is a principled assessment framework that, in addition to infusing manufacturing and quality control principles into the test development cycle, builds in a high level of connectivity between the claims we wish to make about examinees and the items they are administered. This paper focuses on the role assessment engineering infrastructure can play in helping standard setting-panelists bridge the gap between the claims testing bodies wish to make about examinees and the item-level information they are typically provided, and, in doing so, strengthen the validity argument for an instrument’s score interpretations.