A Process for Automatically Generating Algebra Items
Authors
-
MetaMetrics, Inc. 1000 Park Forty Plaza # 120, Durham, NC 27713
Audra E. Kosh
Keywords:
Assessment Design, Item Generation, Test DevelopmentAbstract
Developing high-quality test items is a costly and time-consuming endeavor, yet the need for large item banks increases as testing becomes more frequent and as computer-adaptive tests call for many items of varying difficulty. To address this need, some test developers have begun to implement Automatic Item Generation (AIG) to supplement item writing efforts. Research and practice on AIG continue to grow as technology and psychometric methods develop, but the field lacks content-specific processes for implementing AIG. The purpose of this paper is to present a process for creating AIG item models to assess middle school students’ algebraic reasoning by building upon existing AIG methodology and literature regarding the cognitive complexity of algebra tasks.Downloads
Metrics
Downloads
Published
How to Cite
Issue
Section
References
Baranes, R., Perry, M. & Stigler, J. (1989). Activation of realworld knowledge in the solution of word problems. Cognition and Instruction, 6(4), 287−318. https://doi.org/10.1207/s1532690xci0604_1.
Bassok, M. C. & Martin, S. A. (1998). Adding apples and oranges: Alignment of semantic and formal knowledge.Cognitive Psychology, 35, 99−134. https://doi.org/10.1006/ cogp.1998.0675. PMid: 9570897.
Bejar, I. I. (2010). Recent development and prospects in item generation. In: S. Embretson (Ed.), Measuring psychological constructs (pp. 201-226). Washington, DC: American Psychological Association. https://doi.org/10.1037/12074-009.
Bickel, L., Simpson, M. A., Sanford-Moore, E., Price, R., Durakovic, L., Totten, S. & Kosh, A. E. (2015). Family group generation theory: Large scale implementation. Paper presented at the 2015 National Council on Measurement inEducation Conference, Chicago, IL.
Blanton, M. L. & Kaput, J. J. (2005). Characterizing a classroom practice that promotes algebraic reasoning. Journal for Research in Mathematics Education, 36(5), 412−446.
Carraher, T. N., Carraher, D. W. & Schliemann, A. D. (1985).Mathematics in the streets and schools. British Journal of Developmental Psychology, 3, 21−29. https://doi.org/10.1111/j.2044-835X.1985.tb00951.x.
Clement, J. (1982). Algebra word problem solutions: Thought processes underlying a common misconception. Journal for Research in Mathematics Education, 13(1), 16−30. https://doi.org/10.2307/748434.
Common Core State Standards Initative (2015). Standards for Mathematical Practice. Retrieved from http://www.corestandards. org/Math/Practice/.
Daniel, R. C. & Embretson, S. E. (2010). Designing cognitive complexity in mathematical problem-solving items.Applied Psychological Measurement, 34(5), 348−364.https://doi.org/10.1177/0146621609349801.
Drasgow, F., Luecht, R. M. & Bennett, R. E. (2006). Technology and testing. In: Brennan, R. L. (Ed.), EducationalMeasurement, Westport, CT: Praeger Publishers. p. 471−516.
Embretson, S. E. (1999). Generating items during testing: Psychometric issues and models. Psychometrika, 64, 407−433. https://doi.org/10.1007/BF02294564.
Embretson, S. E. (2002). Generating abstract reasoning items with cognitive theory. In: Irvine, S. H. & Kyllonen, P. C.(Eds.), Item generation for test development. Mahwah, NJ: Erlbaum. p. 219-250.
Embretson, S. E. & Daniel, R. C. (2008). Understanding and quantifying cognitive complexity level in mathematical problem solving items. Psychology Science, 50(3), 328.
Embretson, S. & Yang, X. (2006). Automatic item generation and cognitive psychology. Handbook of Statistics, 26, 747−768. https://doi.org/10.1016/S0169-7161(06)26023-1.
Enright, M. K. & Sheehan, K. M. (2002). Modeling the difficulty of quantitative reasoning items: Implications for item generation. In: S. Irvine & P. Kyllonen (Eds.), Item generation for test development. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. p. 129−157.
Enright, M. K., Morley, M. & Sheehan, K. M. (2002). Items by design: The impact of systematic feature variation on item statistical characteristics. Applied Measurement in Education, 15(1), 49−74. https://doi.org/10.1207/ S15324818AME1501_04.
Fischer, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37(6), 359−374. https://doi.org/10.1016/0001-6918(73)90003-6.
Fuson, K. (1984). More complexities in subtraction. Journal for Research in Mathematics Education, 15, 214−225. https:// doi.org/10.2307/748350.
Geerlings, H., Glas, C. A. & van der Linden, W. J. (2011).Modeling rule-based item generation. Psychometrika, 76(2), 337−359. https://doi.org/10.1007/s11336-011-9204-x.
Gierl, M. J. & Haladyna, T. M. (2013). Automatic item generation: An introduction. In: Gierl, M. J. & Haladyna (Eds.), Automatic Item Generation: Theory and Practice. New York: Routledge. p. 3-12.
Gierl, M. J. & Lai, H. (2013). Using weak and strong theory to create item models for automatic item generation: Some practical guidelines with examples. In: Gierl, M. J. & Haladyna (Eds.), Automatic item generation: Theory and practice. New York: Routledge. p. 26−39.
Gierl, M. J. & Lai, H. (2016). Automatic item generation. In: Lane, S., Raymond, M., & Haladyna, T. M. (Eds.), Second Handbook of Test Development. New York: Routledge. p. 410−429.
Gierl, M. J., Lai, H., Hogan, J. B. & Matovinovic, D. (2015). A method for generating educational test items that are aligned to the common core state standards. Journal for Applied Testing Technology, 16(1), 1-18.
Gierl, M. J., Zhou, J. & Alves, C. (2008). Developing a taxonomy of item model types to promote assessment engineering. The Journal of Technology, Learning and Assessment, 7(2).
Glas, C. A. W. & van der Linden, W. J. (2003). Computerized adaptive testing with item cloning. Applied Psychological Measurement, 27, 247−261. https://doi.org/10.1177/01466 21603027004001.
Goldman, S. & Booker, A. (2009). Making math a definition of the situation: Families as sites for mathematical practices.Anthropology and Education Quarterly, 40(4), 369−387. https://doi.org/10.1111/j.1548-1492.2009.01057.x.
Gorin, J. S. & Embretson, S. E. (2013). Using cognitive psychology to generate items and predict item characteristics. In: Gierl, M. J. & Haladyna (Eds.), Automatic Item Generation: Theory and Practice. New York: Routledge. p. 136-156.
Graf, E. A. & Fife, J. H. (2013). Difficulty modeling and automatic generation of quantitative items: Recent advances and possible next steps. In: Gierl, M. J. & Haladyna (Eds.), Automatic Item Generation: Theory and Practice. New York: Routledge. p. 157−179.
Graf, E. A. (2009). Defining mathematics competency in the service of cognitively based assessment for grades 6 through 8 (ETS Research Report. No. RR-09-42). Princeton, NJ: Educational Testing Service.
Haladyna, T. & Rodriguez, M. (2013). Developing the test item. In: T. Haladyna & M. Rodriguez (Eds.), Developing and Validating Test Items. New York: Routledge. p. 17−27. https://doi.org/10.4324/9780203850381.
Halford, G. S., Wilson, W. H. & Phillips, S. (1998). Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and Brain Sciences, 21, 803−865. https://doi.org/10.1017/S0140525X98001769. PMid: 10191879.
Hombo, C. & Dresher, A. (2001). A simulation study of the impact of automatic item generation under NAEP-like data conditions. Paper presented at the annual meeting of the National Council of Measurement of Education, Seattle, WA.
Hudson, T. (1983). Correspondences and numerical differences between disjoint sets. Child Development, 54(1), 84−90. https://doi.org/10.2307/1129864.
Kahneman, D. (2011). Thinking, Fast and Slow.New York, NY: Farrar, Straus and Giroux.
PMid: 21714386.
Koedinger, K. R. & Nathan, M. J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129−164. https://doi.org/10.1207/s15327809jls1302_1.
Koedinger, K. R., Alibali, M. W. & Nathan, M. J. (2008). Tradeâ€offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science, 32(2),366−397. https://doi.org/10.1080/03640210701863933.PMid: 21635340.
Kosh, A. E., Simpson, M. A., Bickel, L. (2015). Validity of task models in automatically generated mathematics items. Poster presented at the Research Symposium Honoring the Legacy of Carol E. Malloy, Chapel Hill, NC.
Landauer, T. K., McNamara, D. S., Dennis, S. & Kintsch, W.(Eds.). (2013). Handbook of Latent Semantic Analysis. Psychology Press. https://doi.org/10.4324/9780203936399.
Lave, J. (1988). Cognition in Practice: Mind, Mathematics and Culture in Everyday Life. Cambridge University Press.https://doi.org/10.1017/CBO9780511609268.
Leighton, J. P. & Gierl, M. J. (2011). The learning sciences in educational assessment: The role of cognitive models. Cambridge: Cambridge University Press. https://doi.org/10.1017/ CBO9780511996276.
Martin, S. A. & Bassok, M. (2005). Effects of semantic cues on mathematical modeling: Evidence from word-problem solving and equation construction tasks. Memory & Cognition, 33(3), 471−478. https://doi.org/10.3758/BF03193064.
Mayer, R. E. (1981). Frequency norms and structural analysis of algebra story problems into families, categories, and templates.Instructional Science, 10(2), 135−175. https://doi.org/10.1007/BF00132515.
Mayer, R. E. (1982). Different problem-solving strategies for algebra word and equation problems. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8(5), 448−462. https://doi.org/10.1037/0278-7393.8.5.448.
Mayer, R. E., Larkin, J., & Kadane, J. B. (1984). A cognitive analysis of mathematical problem solving ability. In: R. Sternberg (Ed.), Advances in the Psychology of Human Intelligence, Hillsdale, NJ: Lawrence Erlbaum. Vol. 2, p. 231−273.
Mislevy, R. J., Almond, R. G. & Lukas, J. F. (2003). A brief introduction to evidence-centered design (ETS Research Report.No. RR-03-16). Princeton, NJ: Educational Testing Service.
Nathan, M. J. & Koedinger, K. R. (2000). Teachers’ and researchers’ beliefs about the development of algebraic reasoning. Journal for Research in Mathematics Education, 168-90. https://doi.org/10.2307/749750.
Nathan, M. J., Long, S. D. & Alibali, M. W. (2002). Symbol precedence in mathematics textbooks: A corpus analysis.Discourse Processes, 33, 1−21. https://doi.org/10.1207/ S15326950DP3301_01.
National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common Core State Standards for Mathematics. Washington, DC: Authors.
O’Rourke, E., Andersen, E., Gulwani, S. & Popovic, Z. (2015, April). A process for automatically generating interactive instructional scaffolding. In: Begole, B. & Kim, Jinwoo (Eds.), Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery. p. 1545−1554.
Penuel, W. R., Confrey, J., Maloney, A. & Rupp, A. A. (2014). Design decisions in developing learning trajectories–based assessments in mathematics: A case study. Journal of the Learning Sciences, 23(1), 47−95. https://doi.org/10.1080/10 508406.2013.866118.
Saxe, G. B. (1988a). Candy selling and math learning.Educational Researcher, 17(6), 14−21.
https://doi.org/10.3102/0013189X017006014.
Saxe, G. B. (1988b). The mathematics of child street vendors.Child Development, 59(5), 1415−25. https://doi.org/10.2307/1130503.
Sebrechts, M. M., Enright, M., Bennett, R. E. & Martin, K.(1996). Using algebra word problems to assess quantitative ability: Attributes, strategies, and errors. Cognition and Instruction, 14(3), 285−343. https://doi.org/10.1207/ s1532690xci1403_2.
Simpson, M. A., Kosh, A. E., Bickel, L., Elmore, J., SanfordMoore, E., Koons, H. & Enoch-Marx, M. (2015, April). The effects of varying grain size on the exchangeability of item isomorphs. Paper presented at the 2015 National Council on Measurement in Education Conference, Chicago, IL.
Singley, M. K. & Bennett, R. E. (2002). Item generation and beyond: Applications to schema theory to mathematics assessment. In: Irvine, S. H. & Kyllonen, P. C. (Eds.), Item Generation for Test Development. Mahwah, NJ: Erlbaum. p. 361−384.
Sinharay, S. & Johnson, M. (2013). Statistical modeling of automatically generated items. In: Gierl, M. J. & Haladyna (Eds.), Automatic item generation: Theory and practice. New York: Routledge. p. 183−195
Star, J. R., Foegen, A., Larson, M. R., McCallum, W. G., Porath, J. & Zbiek, R. M. (2015). Teaching strategies for improving algebra knowledge in middle and high school students. U.S.Department of Education.
Taylor, E. V. (2009). The purchasing practice of low-income students: The relationship to mathematical development. Journal of the Learning Sciences, 18(3), 370−415. https://doi.org/10.1080/10508400903013462.
U.S. Department of Education (2015). Fact sheet: Testing action plan. Retrieved from https://www.ed.gov/news/pressreleases/ fact-sheet-testing-action-plan
Zieky, M. J. (2014). An introduction to the use of evidencecentered design in test development. PsicologÃa Educativa, 20(2), 79−87. https://doi.org/10.1016/j.pse.2014.11.003.