Matching the Judgmental Task with Standard Setting Panelist Expertise: the Item-descriptor (id) Matching Method

Matching the Judgmental Task with Standard Setting Panelist Expertise: the Item-descriptor (id) Matching Method

Authors

  • Steve Ferrara
     CTB/McGraw-Hill
  • Marianne Perie
     National Center for the Improvement of Educational Assessment
  • Eugene Johnson
     Independent Consultant

Abstract

Psychometricians continue to introduce new approaches to setting cut scores for educational assessments in an attempt to improve on current methods. In this paper we describe the Item- Descriptor (ID) Matching method, a method based on IRT item mapping. In ID Matching, test content area experts match items (i.e., their judgments about the knowledge and skills required to respond to an item) to the knowledge and skills described in performance level descriptors that are used for reporting test results. We argue that the cognitive-judgmental task of matching item response requirements to performance level descriptors is aligned closely with the experience and expertise of standard setting panelists, who are typically classroom teachers and other content area experts. Unlike other popular standard setting methods, ID Matching does not require panelists to make errorprone probability judgments, predict student performance, or imagine examinees who are just barely in a performance level. We describe applications of ID Matching in two educational testing programs and provide evidence of the effectiveness of this method. The entire process is described in the first section of the paper. Subsequent sections describe applications of ID Matching for two operational testing programs.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Downloads

Published

2014-04-09

How to Cite

Ferrara, S., Perie, M., & Johnson, E. (2014). Matching the Judgmental Task with Standard Setting Panelist Expertise: the Item-descriptor (id) Matching Method. Journal of Applied Testing Technology, 9(1), 1–20. Retrieved from http://www.jattjournal.net/index.php/atp/article/view/48346

Issue

Volume 9, Issue 1, 2008

Section

Articles

References

Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.

Ferrara, S., & DeMauro, G. E. (2006). Standardized assessment of individual achievement in K–12. In R.L. Brennan (Ed.), Educational measurement (4th ed., pp. 579–621). Westport, CT: American Council on Education/Praeger.

Ferrara, S., Phillips, G., Williams, P., Leinwand, S., Mahoney, S., & Ahadi, S. (in press). Vertically articulated performance standards: An exploratory study of inferences about achievement and growth. In R. Lissitz (Ed.), Assessing and modeling cognitive development in school: Intellectual growth and standard setting.

Glass, G. V. (1978). Standards and criteria. Journal of Educational Measurement, 15, 237–261.

Green, D. R., Trimble, C. S., & Lewis, D. M. (2003). Interpreting the results of three standard-setting procedures. Educational Measurement: Issues and Practice, 22(1), 22–32.

Hambleton, R. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.

Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: American Council on Education/Praeger Publishers.

Huynh, H. (2006). A clarification on the response probability criterion RP67 for standard settings based on Bookmark and item mapping. Educational Measurement: Issues and Practice, 25(2), 19–20.

Impara, J. C., & Plake, B. S. (1998). Teachers’ ability to estimate item difficulty: A test of the assumptions in the Angoff standard setting method. Journal of Educational Measurement, 35(1), 69–81.

Kane, M. T. (2001). So much remains the same: Conception and status of validation in setting standards. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.

Karantonis, A., & Sireci, S. G. (2006). The Bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4–12.

Kingston, N. M., Kahl, S. R., Sweeney, K. P., & Bay, L. (2001). Setting performance standards using the Body of Work method. In G.J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.

Mills, C. N., & Jaeger, R. M. (1998). Creating descriptions of desired student achievement when setting performance standards. In L. Hansche (Ed.), Handbook for the development of performance standards. Washington, DC: U.S. Department of Education and Council of Chief State School Officers.

National Academy of Education. (1997). Assessment in transition: Monitoring the nation’s educational progress. Palo Alto, CA: Stanford University.

Nickerson, R. S. (2004). Cognition and chance: The psychology of probabilistic reasoning. Mahwah, NJ: Lawrence Erlbaum Associates.

Perie, M. (2005). Angoff and Bookmark methods. Workshop presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.

Plous, S. (1993). The psychology of judgment and decision making. New York: McGraw-Hill.

Reckase, M. D. (2006a). A conceptual framework for a psychometric theory for standard setting with examples of its use for evaluating the functioning of two standard setting methods. Educational Measurement: Issues and Practice, 25(2), 4–18.

Reckase, M. D. (2006b). Rejoinder: Evaluating standard setting methods using error models proposed by Schulz. Educational Measurement: Issues and Practice, 25(3), 14–17.

Schulz, E. M. (2006). Commentary: A response to Reckase’s conceptual framework and examples for evaluating standard setting methods. Educational Measurement: Issues and Practice, 25(3), 4–13.

Sireci, S. G., Baldwin, P., Martone, D., & Han, K. T. (2007). Establishing achievement levels on a multistage computerized-adaptive test: An application of the Item Descriptor Matching method. In B. Plake (Chair), Innovations in Standard Setting, a symposium conducted at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

Wang, N. (2003). Use of the Rasch IRT model in standard setting: An item mapping method. Journal of Educational Measurement, 40(3), 231–253.

Zieky, M., Perie, M., & Livingston, S. (in press). Cutscores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.

Zwick, R., Senturk, D., Wang, J., & Loomis, S. C. (2001). An investigation of alternative methods for item mapping in the National Assessment of Educational Progress. Educational Measurement: Issues and Practice, 20(2), 15–25.