Evaluating the Content Validity of Multistage - Adaptive Tests
Authors
-
University of Massachusetts Amherst
Katrina Crotts -
University of Massachusetts Amherst Stephen G. Sireci
-
University of Massachusetts Amherst April Zenisky
Keywords:
No keywordsAbstract
Validity evidence based on test content is important for educational tests to demonstrate the degree to which they fulfill their purposes. Most content validity studies involve subject matter experts (SMEs) who rate items that comprise a test form. In computerized-adaptive testing, examinees take different sets of items and test "forms" do not exist, which makes it difficult to evaluate the content validity of different tests taken by different examinees. In this study, we evaluated content validity of a multistage-adaptive test (MST) using SMEs' content validity ratings of all items in the MST bank. Analyses of these ratings across the most common "paths" taken by examinees were conducted. The results indicated the content validity ratings across the different tests taken by examinees were roughly equivalent. The method used illustrates how content validity can be evaluated in an MST context.Downloads
Metrics
Downloads
Published
How to Cite
Issue
Section
References
Adult and Community Learning Services (2005). Massachusetts adult basic education curriculum framework for the English language arts. Malden: MA: Massachusetts Department of Education. Available at http://www.doe.mass.edu/acls/frameworks.
American Council on Education. (1995). Guidelines for computerized adaptive test development and use in education. Washington, DC: American Council on Education.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Bhola, D.S., Impara, J.C., & Buckendahl, C.W. (2003). Aligning tests with states’ content standards: Methods and issues. Educational Measurement: Issues and Practice, 22(3), 21-29.
Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, 26(2), 44-52.
Huff, K. L. & Sireci, S.G. (2005). Validity issues in computer-based testing. Educational Measurement: Issues and Practice, 20(3), 16-25.
Kaira, L. T., & Sireci, S. G. (2010). Evaluating content validity in multistage adaptive testing. CLEAR Exam Review, 21(2), 15-23.
Luecht, R.M. (2005). Some useful cost-benefit criteria for evaluating computer-based test delivery models and systems. Journal of Applied Testing Technology, 7(2).
Luecht, R.M. Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage tests. Applied Measurement in Education, 19(3), 189-202.
Luecht, R. M., de Champlain, A., & Nungester, R. J. (1998). Maintaining content validity in computerized adaptive testing. Advances in Health Sciences Education, 3, 29-41.
Martone, A., & Sireci, S. G. (2009). Evaluating alignment among curriculum, assessments, and instruction, Review of Educational Research 4, 1332-1361.
Messick, S. (1989). Validity. In R. Linn (Ed.), Educational Measurement, (3rd ed., pp. 13-103). Washington, D.C.: American Council on Education.
Reese, L.M., Schnipke, D.L., & Luebke, S.W. (1999). Incorporating content constraints into a multi-stage adaptive testlet design (LSAC Computerized Test Report 97-02). Newtown, PA: Law School Admission Council.
Sireci, S. G. (1998). The construct of content validity. Social Indicators Research, 45, 83-117.
Sireci, S.G., Baldwin, P., Martone, A., Zenisky, A.L., Kaira, L., Lam, W., Shea, C.L., Han, K.T., Deng, N., Delton, J., & Hambleton, R.K. (2008). Massachusetts Adult Proficiency Tests Technical Manual, Version 2. Center for Educational Assessment Research Report No. 677. Amherst, MA: Center for Educational Assessment, University of Massachusetts.
Thissen, D. & Mislevy, R.J. (1990). Testing algorithms. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 103-136). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Wainer, H. (1993). Some practical considerations when converting a linearly administered test to an adaptive format. Educational Measurement: Issues and Practice, 12(1), 15-20.
Zenisky, A. L., Sireci, S. G., Ryan, A., Baldwin, P., Colvin, K., Li, X., Semerjian, A., & Hambleton, R. K. (2012). The Massachusetts Adult Proficiency Tests technical manual: Version 3. Amherst, MA: Center for Educational Assessment, University of Massachusetts Amherst.