Using Machine Learning to Predict Bloom’s Taxonomy Level for Certification Exam Items
Authors
-
Chief Psychometrician, Certiverse, 4803 N. Milwaukee Avenue, Suite B, Unit 103, Chicago, IL 60630
Alan D. Mead -
Psychometrician, Certiverse, 4803 N. Milwaukee Avenue, Suite B, Unit 103, Chicago, IL 60630 Chenxuan Zhou
Keywords:
Naïve Bayesian classifier, Bloom's Taxonomy, Machine LearningAbstract
This study fit a Naïve Bayesian classifier to the words of exam items to predict the Bloom’s taxonomy level of the items. We addressed five research questions, showing that reasonably good prediction of Bloom’s level was possible, but accuracy varies across levels. In our study, performance for Level 2 was poor (Level 2 items were misclassified and other items were classified as Level 2), but the performance of the model in distinguishing Level 1 from all other levels was quite good. Applying a model developed on an IT certification exam domain to a more diverse set of items showed poor performance, suggesting that models may generalize poorly. Finally, we showed what features of items the classifier was using. Examples and implications for practice are discussed.Downloads
Metrics
Downloads
Published
How to Cite
Issue
Section
References
Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s Taxonomy of Educational Objectives (Abridged Edition). New York: Longman.
Athanassiou, N., McNett, J., & Harvey C. (2003). Critical thinking in the management classroom: Bloom’s taxonomy as a learning tool. Journal of Management Education, 27(5), 555-575. https://doi.org/10.1177/1052562903252515
Billings, M. S., DeRuchie, K., Hussie, K., Kulesher, A., Merrell, J., Morales, A., Paniagua, M. A., Sherlock, J., Swygert, K.A., & Tyson, J. (2020). Constructing Written Test Questions for the Basic and Clinical Sciences (6th ed). Philadelphia, PA: National Board of Medical Examiners
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives; the classification of educational goals; Handbook I: Cognitive domain. New York, NY: Longmans, Green.
Buckwalter, J. A., Schumacher, R., Albright, J. P., & Cooper, R. R. (1981). Use of an educational taxonomy for evaluation of cognitive performance. Journal of Medical Education, 56(2), 115-21. https://doi.org/10.1097/00001888198102000-00006
Chang, W., & Chung, M. (2009). Automatic applying Bloom’s taxonomy to classify and analysis the cognition level of English question items. 2009 Joint Conferences on Pervasive Computing (JCPC), 727-734. https://doi.org/10.1109/JCPC.2009.5420087
Cilliers, F. J., Schuwirth, L. W., Herman, N., Adendorff, H. J., & van der Vleuten, C. P. (2012). A model of the preassessment learning effects of summative assessment in medical education. Advances in Health Sciences Education: Theory and Practice, 17(1), 39-53. https://doi.org/10.1007/s10459-011-9292-5
Crowe, A., Dirks, C., & Wenderoth, M. (2008). Biology in bloom: Implementing Bloom’s taxonomy to enhance student learning in biology. CBE Life Sciences Education, 7(4), 368-381. https://doi.org/10.1187/cbe.08-05-0024
Haris, S. S., & Omar, N. (2015). Bloom’s taxonomy question categorization using rules and n-gram approach. Journal of Theoretical and Applied Information Technology, 7(3), 401-407.
Jensen, J. L., McDaniel, M. A., Woodard, S. M., & Kummer, T. (2014). Teaching to the test… or testing to teach: Requiring higher order thinking skills encourage greater conceptual understanding. Educational Psychology Review, 26, 307-329. https://doi.org/10.1007/s10648013-9248-9
Jurafsky, D., & Martin, J. H. (2000). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Upper Saddle River, NJ: Prentice Hall.
Karpen, S. C., & Welch, A. C. (2016). Assessing the interrater reliability and accuracy of pharmacy faculty’s Bloom’s Taxonomy classifications. Currents in Pharmacy Teaching and Learning, 8, 885-888. https://doi.
org/10.1016/j.cptl.2016.08.003
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. https://doi.org/10.2307/2529310
Manning, C. D., Raghavan, P., & Schütze, H. (2009). An introduction to information retrieval (Online Edition). Cambridge University Press: Cambridge, England. https://nlp.stanford.edu/IR-book/pdf/ irbookonlinereading.pdf
Mesic, V., & Muratovic, H. (2011). Identifying predictors of physics item difficulty: A linear regression approach. Physical Review Special Topics - Physics Education Research, 7. https://doi.org/10.1103/ PhysRevSTPER.7.010110
Mohammed, M., & Omar, N. (2020). Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec. PLoS ONE. 15(3). https://doi.org/10.1371/journal.pone.0230442
Nkanginieme, K. E. O. (1997). Clinical diagnosis as a dynamic cognitive process: Application of Bloom’s Taxonomy for educational objectives in the cognitive domain. Medical Education Online, 2:1. https://doi.
org/10.3402/meo.v2i.4288
Orrù, G., Monaro, M., Conversano, C., Gemignani, A., & Sartori, G. (2020). Machine learning in psychometrics and psychological research. Frontiers in Psychology, 10, 2970. https://doi.org/10.3389/fpsyg.2019.02970
Plack, P.M., Driscoll, M., Marquez, M., Cuppernull, L., Maring, J., & Greenberg, L. (2007). Assessing reflective writing on a pediatric clerkship by using a modified Bloom’s Taxonomy. Ambulatory Pediatrics, 7, 285-291. https://doi.org/10.1016/j.ambp.2007.04.006
Porter, M.F. (1980). An algorithm for suffix stripping. Program, 14, 130-137. https://doi.org/10.1108/eb046814
Putka, D. J., Beatty, A. S., & Reeder, M. C. (2018). Modern prediction methods: New perspectives on a common problem. Organizational Research Methods, 21, 689-732. https://doi.org/10.1177/1094428117697041
Rosca, C. V. (2004). What makes a science item difficult? A study of TIMSS -R items using regression and the linear logistic test model. Unpublished doctoral dissertation, Boston College.
Sinharay, S. (2016). An NCME Instructional Module on Data Mining Methods for Classification and Regression. Educational Measurement: Issues and Practice, 35, 38-54.
https://doi.org/10.1111/emip.12115
Stephens, C. R., Huerta, H. F., & Linares, A. R. (2018). When is the Naïve Bayes approximation not so naïve? Machine Learning, 107, 397-441. https://doi.org/10.1007/s10994017-5658-0
Tan, Y. T., & Othman, A. (2013). The relationship between complexity (taxonomy) and difficulty. AIP Conference Proceedings, 1522, 596. https://doi.org/10.1063/1.4801179
Thompson, E., Luxton-Reilly, A., Whalley, J., Hu, M. & Robbins, P. (2008). Bloom’s taxonomy for CS assessment. Paper presented at Conference on Australasian computing education.
Yahya, A. A., Osman, A., Taleb, A., & Alattab, A. A. (2013). Analyzing the cognitive level of classroom questionsn using machine learning techniques. Procedia – Social and Behavioral Sciences, 97, 587-595. https://doi.org/10.1016/j.sbspro.2013.10.277.
Yahya, A. A., & Osman, A. (2011). Automatic classification of questions into Bloom’s cognitive levels using support vector machines. Proceedings of the International Arab Conference on Information Technology. Riyadh, Saudi Arabia, 335-342. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.657.25&rep=rep1&type=pdf
Yusof, N., & Hui, C.J. (2010). Determination of Bloom’s cognitive level of question items using artificial neural network. 2010 10th International Conference on Intelligent Systems Design and Applications, 866-870. https://doi.org/10.1109/ISDA.2010.5687152
Zaidi, N. L. B., Grob, K. L., Monrad, S. M., Kurtz, J. B., Tai, A., Ahmed, A. Z., Gruppen, L. D., & Santen, S. A. (2018). Pushing critical thinking skills with multiple-choice questions: Does Bloom’s Taxonomy work? Academic Medicine, 93(6), 856-859. https://doi.org/10.1097/ACM.0000000000002087