Taming the Firehose: Unsupervised Machine Learning for Syntactic Partitioning of Large Volumes of Automatically Generated Items to Assist Automated Test Assembly

Taming the Firehose: Unsupervised Machine Learning for Syntactic Partitioning of Large Volumes of Automatically Generated Items to Assist Automated Test Assembly

Authors

  • Brian S. Cole
     Content and Innovation, Elsevier, Amsterdam, Netherlands
  • Elia Lima-Walton
     Content and Innovation; Elsevier, Houston, TX, USA
  • Kim Brunnert
     HESI, Elsevier, Houston, TX, USA
  • Winona Burt Vesey
     HESI, Elsevier, Houston, TX, USA
  • Kaushik Raha
     Content and Innovation, Elsevier, Philadelphia, PA, USA

Keywords:

Automated Item Generation, Automated Test Assembly, Machine Learning, Natural Language Processing

Abstract

Automatic item generation can rapidly generate large volumes of exam items, but this creates challenges for assembly of exams which aim to include syntactically diverse items. First, we demonstrate a diminishing marginal syntactic return for automatic item generation using a saturation detection approach. This analysis can help users of automatic item generation to generate more diverse item banks. We then develop a pipeline that uses an unsupervised machine learning method for partitioning of a large, automatically generated item bank into syntactically distinct clusters. We explore applications to test assembly and conclude that machine learning methods can provide utility in harnessing the large datasets achievable by automatic item generation.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Downloads

Published

2019-08-30

How to Cite

Cole, B. S., Lima-Walton, E., Brunnert, K., Vesey, W. B., & Raha, K. (2019). Taming the Firehose: Unsupervised Machine Learning for Syntactic Partitioning of Large Volumes of Automatically Generated Items to Assist Automated Test Assembly. Journal of Applied Testing Technology, 21(1), 1–11. Retrieved from http://www.jattjournal.net/index.php/atp/article/view/146483

Issue

Volume 21, Issue 1, 2020

Section

Articles

References

Al-Yahya, M. (2014). Ontology-based multiple choice question generation. The Scientific World Journal. https:// doi.org/10.1155/2014/274949. PMid: 24982937, PMCid: PMC3984768.

Arendasy, M. E. and Sommer, M. (2012). Using automatic item generation to meet the increasing item demands of high-stakes educational and occupational assessment. Learning and Individual Differences, 22(1), 112−17. https:// doi.org/10.1016/j.lindif.2011.11.005.

Arthur, D. and Vassilvitskii, S. (2007). k-means++: The advantages of careful seeding. In: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms. p.1027−35.

Bahmani, B., Moseley, B., Vattani, A., Kumar, R. and Vassilvitskii, S. (2012). Scalable k-means++. Proceedings of the VLDB Endowment, 5(7), 622−33. https://doi.org/10.14778/2180912.2180915.

Bejar, I. I. (2002). Generative testing: From conception to implementation.Item Generation for Test Development, 199−217.

Bejar, I. I., Division, P. S., Authority, C., Number, I., Investigator, P. and Service, E. T. (1986). A psychometric analysis of a three-dimensional spatial task, (June). https://doi.org/10.1002/j.2330-8516.1986.tb00174.x.

Deane, P. and Sheehan, K. (2003). Automatic Item Generation via Frame Semantics : Natural Language Generation of Math Word Problems. Language, 26. Retrieved from http:// ccl.pku.edu.cn/doubtfire/semantics/AutomaticItemGenera tionViaFrameSemantics-by-deane.pdf.

Drasgow, F. (2015). Technology and testing: Improving educational and psychological measurement. Routledge. https:// doi.org/10.4324/9781315871493.

Foulonneau, M. (2012). Generating educational assessment items from linked open data: The case of DBpedia. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). https://doi.org/10.1007/978-3-642-259531_2. https://doi.org/10.1007/978-3-642-25953-1_2.

Gierl, M. J. and Haladyna, T. M. (2012). Automatic item generation: Theory and practice. Routledge. https://doi.org/10.4324/9780203803912.

Gierl, M. J. & Lai, H. (2012). The Role of Item Models in Automatic Item Generation. International Journal of Testing, 12(3), 273−298. https://doi.org/10.1080/15305058.2011.635830

Gierl, M. J., Lai, H. and Turner, S. R. (2012). Using automatic item generation to create multiple-choice test items.Medical Education, 46(8), 757−65. https://doi.org/10.1111/ j.1365-2923.2012.04289.x. PMid: 22803753.

Jordan, M. I. and Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255−60. https://doi.org/10.1126/science.aaa8415. PMid: 26185243.

Morley, M. E., Bridgeman, B. and Lawless, R. R. (2004).Transfer between variants of quantitative items. ETS Research Report Series, 2004(2), 1−27. https://doi.org/10.1002/j.2333-8504.2004.tb01963.x.

Newstead, S., Handley, S. and Evans, J. (2002). Using the psychology of reasoning to predict the difficulty of analyticalreasoning problems. Item Generation for Test Development, 35.

Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5), 503−20. https://doi.org/10.1108/00220410410560582.

Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53−65.https://doi.org/10.1016/0377-0427(87)90125-7.

Royal, K. D. and Hedgpeth, M.-W. (2017). The prevalence of item construction flaws in medical school examinations and innovative recommendations for improvement. EMJ Innov., 1(1), 61-66.

Rudner, L. M. (2009). Implementing the graduate management admission test computerized adaptive test. In: Elements of Adaptive Testing. Springer, p. 151−65. https://doi.org/10.1007/978-0-387-85461-8_8.

Silva, C. and Fonseca, J. (2017). Educational data mining: a literature review. In: Europe and MENA Cooperation Advances in Information and Communication Technologies. Springer. p. 87−94. https://doi.org/10.1007/978-3-319-46568-5_9.

von Davier, M. (2018). Automated item generation with recurrent neural networks. Psychometrika. https://doi. org/10.1007/s11336-018-9608-y. PMid: 29532403

Wainer, H. (2002). On the automatic generation of test items: Some whens, whys, and hows. Item Generation for Test Development, 287−314.