Improving the Quality of Innovative Item Types: Four Tasks for Design and Development

Improving the Quality of Innovative Item Types: Four Tasks for Design and Development

Authors

  • Cynthia G. Parshall
     Measurement Consultant
  • J. Christine Harmes
     James Madison University

Abstract

Many exam programs have begun to include innovative item types in their operational assessments. While innovative item types appear to have great promise for expanding measurement, there can also be genuine challenges to their successful implementation. In this paper we present a set of four activities that can be beneficially incorporated into the design and development of innovative item types. These tasks are: template design, item writing guidelines, item writer training, and usability studies. When these four tasks are fully incorporated in the test development process then the potential for improved measurement through innovative item types is much greater.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Downloads

Published

2014-04-09

How to Cite

Parshall, C. G., & Christine Harmes, J. (2014). Improving the Quality of Innovative Item Types: Four Tasks for Design and Development. Journal of Applied Testing Technology, 10(1), 1–20. Retrieved from http://www.jattjournal.net/index.php/atp/article/view/48349

Issue

Volume 10, Issue 1, 2009

Section

Articles

References

Ballas, J. (1994). Delivery of information through sound. In G. Kramer (Ed.), Auditory Display (pp. 79-94). Reading, MA: Addison-Wesley.

Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice, 17, 9-17.

Bias, R. G., & Mayhew, D. J. (Eds.). (1994). Cost-justifying usability. Boston: Academic Press.

Bunderson, V. C., Inouye, D. I., & Olsen, J. B. (1989). The four generations of computerized educational measurement. In R. Linn (Ed.) Educational Measurement. 3rd edition. New York: American Council on Education and Macmillan Publishing Co.

Content Consumer. (2008). The great Ubuntu-girlfriend experiment. Retrieved April 28, 2008 from http://contentconsumer.wordpress.com/2008/04/27/is-ubuntu-useable-enough-for-my-girlfriend/

Downing, S. M. (2006). Twelve steps for effective test development. In S. M. Downing & T. M. Haladyna (Eds.). Handbook of Test Development (pp. 3-25). Mahwah, NJ: Lawrence Erlbaum Associates.

Dumas, J. S., & Redish, J. C. (1999). A practical guide to usability testing, Revised Edition. Exeter, England: Intellect.

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Revised edition). Cambridge, MA: MIT Press.

Gaver, W. W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4, 67-94.

Gould, J. D., Bois, F. J., & Ukelson, J. (1997). How to design usable systems. In Helander, M., & Landauer, T.K., & Prabhu, P. (Eds.), Handbook of human-computer interaction, 2nd, completely revised edition. (pp. 231-254). New York: Elsevier Science Publishers.

Haladyna, T. M. (1996). Writing test items to evaluate higher order thinking. Needham Heights, MA: Allyn & Bacon.

Harmes, J. C., Kaliski, P. K., & Barry, C. L. (2007, November). Are they really more memorable? Implications of innovative items for test security. Paper presented at the annual meeting of the Florida Educational Research Association, Tampa, FL.

Harmes, J. C. & Parshall, C. G. (2000, November). An iterative process for computerized test development: Integrating usability methods. Paper presented at the annual meeting of the Florida Educational Research Association, Tallahassee.

Harmes, J. C., & Parshall, C. G. (2007, February). Development and evaluation of an innovative computer-based assessment. Poster presented at the annual meeting of the Association of Test Publishers, Palm Springs, CA.

Harmes, J. C., Parshall, C. G., Rendina-Gobioff, G., Jones, P. K., Githens, M. P., & Dennard, A. (2004, November). Integrating usability methods into the CBT development process: Case study of a technology literacy assessment. Paper presented at the annual meeting of the Florida Educational Research Association, Tampa, FL.

Harms, M., Burling, K., Way, W., Hanna, E., & Dolon, R. (2006, April). Constructing innovative computer-administered tasks and items according to Universal Design: Establishing guidelines for test developers. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.

Hoffman, D. J., Harmes, J. C., & Erb, J. P. (2007, April). Usability evaluation for computer-based testing software: Comparing method effects on information acquisition. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.

Jodoin, M. G. (2003). Measurement efficiency of innovative item formats in computer-based testing. Journal of Educational Measurement, 40(1), 1-15.

Johnstone, C. J., Thompson, S. J., Bottsford-Miller, N. A., & Thurlow, M. L. (2008). Universal design and multimethod approaches to item review. Educational Measurement: Issues and Practice, 27(1), 25-36.

Karat, C. (1997). Cost-justifying usability engineering in the software life cycle. In Helander, M., & Landauer, T.K., & Prabhu, P. (Eds.). Handbook of human-computer interaction, 2nd, completely revised edition. (pp. 231-254). New York: Elsevier Science Publishers.

Kayser, M., & Parshall, C. G. (2008, March). Building a global innovative test. Presented at the annual meeting of the Association of Test Publishers, Dallas, TX.

Kirakowski, J. & Corbett, M. (1990). Effective methodology for the study of HCI. New York: North-Holland.

Landauer, T. K. (1995). The trouble with computers: Usefulness, usability, and productivity. Cambridge, MA: MIT Press.

Millman, J. & Greene, J. (1989). The specification and development of tests of achievement and ability. In Linn, R. (Ed.). Educational Measurement. 3rd edition. New York: American Council on Education and Macmillan Publishing Co.

Nielsen, J. (2000). Why you only need to test with 5 users. Retrieved November 4, 2004, from: http://www.useit.com/alertbox/20000319.html

Nielsen, J. (2003). Usability 101: Introduction to usability. Retrieved April 4, 2008 from http://www.useit.com/alertbox/20030825.html

Nielsen, J. (2007). Fast, cheap, and good: Yes, you can have it all. Retrieved April 4, 2008 from http://www.useit.com/alertbox/quantitative_testing.html

Nielsen, J. (2006). Quantitative studies: How many users to test. Retrieved April 25, 2008 from http://www.useit.com/alertbox/fast-methods.html

Parshall, C. G., & Balizet, S. (2001). Audio CBTs: An initial framework for the use of sound in computerized tests. Educational Measurement: Issues & Practice, 20, 5-15.

Parshall, C. G. & Becker, K. A. (2008, July). Beyond the technology: Developing innovative items. Presented at the bi-annual meeting of the International Test Commission, Manchester, UK.

Parshall, C. G., & Harmes, J. C. (2008). The design of innovative item types: Targeting constructs, selecting innovations, and refining prototypes. CLEAR Exam Review, 19(2).

Parshall, C. G. & Harmes, J. C. (2008, March). Stages in designing innovative item types. Presented at the annual meeting of the Association of Test Publishers, Dallas, TX.

Parshall, C. G., & Harmes, J. C. (2005, February). Tools for improving the CBT user interface: Paper prototyping, expert review and user testing. Presented at the annual meeting of the Association of Test Publishers, Phoenix, AZ.

Parshall, C. G., Harmes, J. C., Davey, T., & Pashley, P. (In press). Innovative items for computerized testing. In W. J. van der Linden & C. A. W. Glas (Eds.). Computerized adaptive testing: Theory and practice, 2nd Edition, Norwell, MA: Kluwer Academic Publishers.

Parshall, C. G., Spray, J. A., Kalohn, J. C., & Davey, T. (2002). Practical considerations in computer-based testing. New York: Springer-Verlag.

Roid, G. H., & Haladyna, T. M. (1982). A technology of test-item writing. New York: Academic Press.

Schmeiser, C. B. & Welch, C. J. (2006). Test development. In R. Brennan (Ed.). Educational Measurement 4th edition, (pp. 307-353). Westport, CT: Praeger Publishers.

Shneiderman, B., & Plaisant, C. (2005). Designing the user interface: Strategies for effective human-computer interaction. Boston: Pearson/Addison Wesley.

Sireci, S. G., & Zenisky, A. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representations. In S. M. Downing & T. M. Haladyna, (Eds.), Handbook of Test Development (pp. 329-347). Mahwah, NJ: Lawrence Earlbaum Associates.

Tullis, T. (1997). Screen design. In Helander, M., & Landauer, T.K., & Prabhu, P. (Eds.), Handbook of human-computer interaction, 2nd, completely revised edition. (pp. 503-531). New York: Elsevier Science Publishers.

Wendt, A., & Harmes, J. C. (in press). Developing and evaluating innovative items: Part II, item characteristics and cognitive processing. Nurse Educator.

Wendt, A., Harmes, J. C., Wise, S. L., & Jones, A. T. (2008, March). Development and evaluation of innovative test items for a computerized nursing licensure exam. Paper presented at the annual meeting of the American Educational Research Association, New York.

Wendt, A., Kenny, L. E., & Marks, C. (2007). Assessing critical thinking using a talk-aloud protocol. CLEAR Exam Review, 18(1), 18-27.

Zenisky, A. L., & Sireci, S. G. (2001). Feasibility review of selected performance assessment item types of the Computerized Uniform CPA Exam. (AICPA Research Consortium- Examinations Team. Technical Report) AICPA: Author.