Using Pupillometry to Validate a KSA-Mitigated Model of Cognitive Processes
Authors
-
Lead Content Specialist, Science, ACT, Iowa City, Iowa
Jay Thomas
Keywords:
Graphic Literacy, Index of Cognitive Activity, KSA-mitigated Effort, Pupillometry, Total Cognitive EffortAbstract
A model of cognition and a construct, such as a concept map (Wilson, 2009), is critical in designing assessments of that construct. The Knowledge, Skills and Abilities (KSAs) in the construct must be put to use in order to assess what test takers know and can do (National Research Council, 2001). In order to validate a construct map for graphic literacy, a model of cognitive processes involving exerted cognitive effort and the mitigating effects of KSAs is explored. Data from pupillometry was used to quantify cognitive effort so that the KSA-mitigated model of cognition could be validated along with the construct map of cognitive processes related to graphic literacy and its assessment.
Downloads
Metrics
Downloads
Published
How to Cite
Issue
Section
References
Aberg-Bengtsson, L., Ottosson, T. (2006). What lies behind graphicacy? Relating students’ results on a test of graphically represented quantitative information to formal academic achievement. Journal of Research in Science Teaching. 43(1):43–62. https://doi.org/10.1002/ tea.20087.
ACT (2019).WorkKeys Graphic Literacy technical manual. March 21, 2022. https://www.act.org/content/dam/act/ unsecured/documents/WorkKeys-Graphic-LiteracyTechnical-Manual.pdf.
American Educational Research Association (AERA), American Psychological Association (APA) and National Council for Educational Measurement (NCME). (2014). Standards for educational and psychological testing (2014 ed.). Washington, DC: AERA Publications.
Aminihajibashi, S., Hagen, T., Foldal, M. D., Laeng, B., Espeseth, T. (2019). Individual differences in resting-state pupil size: Evidence for association between working memory capacity and pupil size variability. International Journal of Psychophysiology. 140:1–7. PMid: 30894328. https://doi.org/10.1016/j.ijpsycho.2019.03.007.
Ahern, S., Beatty, J. (1979). Pupillary responses during information processing vary with Scholastic Aptitude Test scores. Science. 205(4412):1289–92. PMid: 472746. https://doi.org/10.1126/science.472746.
Andrzejewska, M., Stolinska, A. (2016). Comparing the difficulty of tasks using eye tracking combined with subjective and behavioural criteria. Journal of Eye Movement Research. 9(3). https://doi.org/10.16910/jemr.9.3.3.
Ayaz, H., Shewokis, P. A., Bunce, S., Onaral, B. (2011, August). An optical brain computer interface for environmental control. In Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE (pp. 6327–30). IEEE. PMid: 22255785. https://doi.org/10.1109/IEMBS.2011.6091561.
Bailey, B. P., Iqbal, S. T. (2008). Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management. ACM Transactions on Computer-Human Interaction (TOCHI). 14(4):1–28. https://doi.org/10.1145/1314683.1314689.
Bartels, M., Marshall, S.P. (2006, March). Eye tracking insights into cognitive modeling. In Proceedings of the 2006 symposium on Eye Tracking Research and Applications (pp. 141–7). https://doi.org/10.1145/1117309.1117358.
Beatty, J. (1982). Task-evoked pupillary responses, processing load and the structure of processing resources. Psychological Bulletin. 91:276–92. PMid: 7071262. https://doi.org/10.1037/0033-2909.91.2.276.
Beatty, J., Kahneman, D. (1966). Pupillary changes in two memory tasks. Psychonomic Science. 5(10):371–2. https://doi.org/10.3758/BF03328444.
Beatty, J. Lucero-Wagoner, B. (2000). The pupillary system. In J. T. Cacioppo, L. G. Tassinary and G. G.
Berntson (Eds), Handbook of Psychophysiology (2nd ed.; pp. 142–62). Cambridge, UK: Cambridge University Press.
Beauchamp, M. R. (2016). Disentangling motivation from self-efficacy: Implications for measurement, theorydevelopment and intervention. Health Psychology Review. 10(2):129–32. PMid: 26953186. https://doi.org/ 10.1080/17437199.2016.1162666.
Blessing, S. B., Ross, B. H. (1996). Content effects in problem categorization and problem solving. Journal of Experimental Psychology: Learning, Memory and Cognition. 22(3):792–810. https://doi.org/10.1037/02787393.22.3.792.
Boehm-Davis, D. A., Gray, W. D., Adelman, L., Marshall, S., Pozos, R. (2003). Understanding and measuring cognitive workload: A coordinated multidisciplinary approach. George Mason University Fairfox VA Department of Psychology. Retrieved March 22, 2022. https://apps.dtic.mil/sti/pdfs/ADA417743.pdf https:// doi.org/10.21236/ADA417743.
Bryant, P. E., Somerville, S. (1986). The spatial demands of graphs. British Journal of Psychology. 77:187–97. PMid: 3730726. https://doi.org/10.1111/j.2044-8295.1986.tb01993.x.
Chen, F., Zhou, J., Wang, Y., Yu, K., Arshad, S. Z., Khawaji, A., Conway, D. (2016). Robust multimodal cognitive load measurement. Switzerland: Springer International Publishing; https://doi.org/10.1007/978-3-319-31700-7.
Chi, M. T., Feltovich, P. J., Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science. 5(2):121–52. https://doi.org/10.1207/s15516709cog0502_2.
Curcio, F.R. (1987). Comprehension of mathematical relationships expressed in graph. Journal for Research in Mathematics Education. 18(5):382–93. https://doi.org/10.5951/jresematheduc.18.5.0382.
Curcio, F.R., Artz, A.F. (1997). Assessing students’ statistical problem solving behaviours in small-group setting. In Gal, I., and Garfield, J.B. (Eds.). The assessment challenge in statistics education Amsterdam: IOS Press; p. 123–38.
Ericsson, K. A., Simon, H. A. (1980). Verbal reports as data. Psychological Review. 87(3):215–50. https://doi.org/10.1037/0033-295X.87.3.215.
EyeTracking, Inc. (2014). Workload Module Manual.
Friel, S. N., Bright, G. W. (1996). Building a theory of graphicacy: How do students read graphs? Retrieved September 30, 2021. http://eric.ed.gov/?id=ED395277.
Friel, S. N., Curcio, F. R., Bright, G. W. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education. 32(2):124–58. https://doi.org/10.2307/749671.
Garner, R. (1987). Metacognition and reading comprehension. Norwood, NJ: Ablex Publishing.
Gotwals, A. W., Songer, N. B. (2009). Reasoning up and down a food chain: Using an assessment framework to investigate students’ middle knowledge. Science Education. 94(2):259–81. https://doi.org/10.1002/sce.20368.
Gotwals, A. W., Songer, N. B., Bullard, L. (2012). Assessing students’ progressing abilities to construct scientific explanations. A.C. Alonzo and A.W. Gotwals (Eds.). Learning progressions in science. Rotterdam: Sense Publishers; p. 183–210. https://doi.org/10.1007/978-946091-824-7_9.
Gutirrez, R. S., Shapiro, L. P. (2010). Measuring the time course of sentence processing with pupillometry. Cuny conference on human sentence processing.
Hess, E. H., Polt, J. R. (1964). Pupil size in relation to mental activity during simple problem-solving.
Science. 143(3611):1190–2. https://doi.org/10.1037/0033295X.87.3.215.
Hmelo-Silver, C. E., Pfeffer, M. G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors and functions. Cognitive Science. 28(1):127–38. https://doi.org/10.1207/s15516709cog2801_7.
Just, M.A., Carpenter, P.A., Woolley, J.D. (1982). Paradigms and processes in reading comprehension. Journal of Experimental Psychology. 111(2):228–38. PMid: 6213735. https://doi.org/10.1037/0096-3445.111.2.228.
Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus, and Giroux;.
Kahneman, D., Beatty, J. (1966). Pupil diameter and load on memory. Science. 154(3756):1583–5. PMid: 5924930. https://doi.org/10.1126/science.154.3756.1583.
Kliewer, K., Langenfeld, T., Thomas, J. (2018, October). Observing cognitive processes and identifying reading difficulties through eyetracking analysis. Poster session presented at the Education Technology and Computational Psychometrics Symposium in Iowa City, IA.
Langenfeld, T., Thomas, J., and Gao, X. (2019, April). Principled assessment design: Applications and tools for assessment updates. Paper presented at the meeting of the National Council for Educational Measurement, Toronto, Ontario.
Langenfeld, T., Thomas, J., Zhu, R., Morris, C. A. (2020). Integrating multiple sources of validity evidence for an assessmentâ€based cognitive model. Journal of Educational Measurement. 57(2):159–84. https://doi.org/10.1111/jedm.12245.
Lee, S. C., Nathan, B. R. (1997). Obtaining reliable job analysis information: A progress report from the Work Keys System TM: ACT’s nationwide program for improving workplace skills. March 22, 2022. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.679 &rep=rep1&type=pdf.
Manseta, K., Khwaja, A. M., Sultan, E., Daruwalla, P., Pourrezaei, K., Najafizadeh, L., Gandjbakhche, A., Daryoush, A. S. (2011, June). Untethered helmet mounted Functional Near Infrared (fNIR) biomedical imaging? Microwave Symposium Digest (MTT), 2011 IEEE MTT-S International (pp. 1-4). IEEE. https://doi.org/10.1109/MWSYM.2011.5972982.
Marshall, S. P. (2000). U.S. Patent No. 6,090,051. Washington, DC: U.S. Patent and Trademark Office.
Marshall, S. P. (2002). The index of cognitive activity: Measuring cognitive workload. Human factors and power plants, 2002, proceedings of the 2002 IEEE 7th Conference. Scottsdale, AZ: IEEE. p. 75–9. https://doi.org/10.1109/HFPP.2002.1042860.
Marshall, S. P. (2007). Identifying cognitive state from eye metrics. Aviation, Space and Environmental Medicine. 78(5Sec II, Suppl):B165–75.
Marshall, S. P., Davis, C. L., Knust, S. R. (2004). The index of cognitive activity: Estimating cognitive effort from pupil dilation. San Diego, CA: EyeTracking, Inc.; Technical Report ETI-0401.
Marshall, S. P., Pleydell-Pearce, C. W., Dickson, B. T. (2003). Integrating psycho-physiological measures of cognitive workload and eye movements to detect strategy shifts. Proceedings of the 36th Annual Hawaii International Conference on System Sciences. https://doi.org/10.1109/HICSS.2003.1174298.
Martinez, M. E., Katz, I. R. (1995). Cognitive processing requirements of constructed figural response and multiplechoice items in architecture assessment. Educational Assessment. 3(1):83–98. https://doi.org/10.1207/ s15326977ea0301_4.
Morrison, J. G., Marshall, S. P., Kelly, R. T., Moore, R. A. (1997). Eye tracking in tactical decision making environments: Implications for decision support evaluation. Third International Command and Control Research and Technology Symposium, National Defense University, June 17-20, 1997. May 14, 2022. http://www.all.net/journal/deception/www-tadmus.spawar.navy.mil/www-tadmus.spawar.navy.mil/Eye_Trkr.pdf.
National Research Council (2001). Knowing what students know: The science and design of educational assessment. National Academies Press.
NGSS Lead States (2013). Next Generation Science Standards: For states, by states. Washington, DC: The National Academies Press.
Paas, F. G. W. C, Van Merrienboer J. J. G. (1994). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review. 6:351–71. https://doi.org/10.1007/BF02213420.
Paas, F., Tuovinen, J. E., Tabbers, H., Van Gerven, P. W. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist. 38(1):63–71. https://doi.org/10.1007/BF02213420.
Palmer, H., Valet, W. (2001). Job analysis: Targeting needed skills. Employment Relations Today. 28(3):85–92. https://doi.org/10.1002/ert.1029.
Pomplun, M., Sunkara, S. (2003). Pupil dilation as an indicator of cognitive workload in human-computer interaction. Proceedings of the 10th International Conference on Human Computer Interaction, V.D.D.
Harris, M. Smith, and C. Stephanidis, Eds.
Poole, A., Ball, L. J. (2006). Eye tracking in HCI and usability research. C. Ghaoui (Ed.) Encyclopedia of Human-Computer Interaction. Pennsylvania: Idea Group; p. 211–9. PMid16524094. https://doi.org/10.4018/978-1-59140-562-7.ch034.
Schiff, M. (1875). La pupille considérée comme esthésiomètre. Baillière.
Sensori Motoric Instruments (SMI). (2016). iView User Guide: Version 2.7.1. Boston, MA: Author.
Shah, P., Freedman, E. G. (2011). Bar and line graph comprehension: An interaction of top-down and bottom-up processes. Topics in Cognitive Science. 3(3):560–78. PMid: 25164403. https://doi.org/10.1111/ j.1756-8765.2009.01066.x.
Shah, P., Hoeffner, J. (2002). Review of graph comprehension research: Implications for instruction.
Educational Psychology Review. 14(1):47–69. https://doi.org/10.1023/A:1013180410169.
van Someren, M. W., Barnard, Y. F., Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modelling cognitive processes. London: Academic Press.
Tan, J. K., Benbasat, I. (1990). Processing of graphical information: A decomposition taxonomy to match data extraction tasks and graphical representations. Information Systems Research. 1(4):416–39. https://www.jstor.org/stable/23010666.
Tanaka, Y., Yamaoka, K. (1993). Blink activity and task difficulty. Perceptual and Motor Skills. 77(1):55–66. PMid: 8367265. https://doi.org/10.2466/pms.1993.77.1.55.
Taylor, H. A., Renshaw, C. E., Choi, E. J. (2004). The effect of multiple formats on understanding complex visual displays. Journal of Geoscience Education. 52(2):115–21. https://doi.org/10.5408/1089-9995-52.2.115.
Thomas, J., Langenfeld, T. L. (2017a, April). Analyzing think-aloud and eye-tracking data to support score interpretations. Paper presented at the meeting of the National Council on Measurement in Education, San Antonio, TX.
Thomas, J., Langenfeld, T. (2017b, November). Using multiple lines of evidence to validate a construct
and assessment. Poster presented at the Education Technology and Computational Psychometrics Symposium, Iowa City, IA.
Thomas, J., Langenfeld, T. L. (2018). Using eye tracking and pupillometry to validate a Graphic Literacy assessment. Presentation at ITC 2018, Montreal, Canada.
Thomas, J., Langenfeld, T. (2019, April). Using eye tracking data to validate the cognitive processes of foundational workplace skills. Paper presented at the meeting of the National Council on Measurement in Education, Toronto, Ontario.
Tobii Pro, Inc (2016). Tobii Pro Glasses 2 User’s Manual.
Veltman, J. A., Jansen, C. (2006). The role of operator state assessment in adaptive automation. TNO Defense, Security and Safety. March 22, 2022. https://apps.dtic.mil/sti/pdfs/ADA455055.pdf.
Vygotsky, L. (1987). Zone of proximal development. Mind in society: The development of higher psychological processes, 5291:157.
Wainer, H. (1992). Understanding graphs and tables. Educational Researcher. 21(1):14–23. https://doi.
org/10.3102/0013189X021001014.
Wilson, M. (2009). Measuring progressions: Assessment structures underlying a learning progression. Journal of Research in Science Teaching: The Official http://sanshodhanstaging.jattjournal.net/ Journal of Applied Testing Technology 23 National Association for Research in Science Teaching. 46(6):716–30. https://doi.org/10.1002/tea.20318.
Wise, S. L. (2017). Rapidâ€guessing behavior: Its identification, interpretation and implications. Educational Measurement: Issues and Practice. 36(4):52–61. https://doi.org/10.1111/emip.12165.
Wise, S. L., Kuhfeld, M. R. (2020). A cessation of measurement: Identifying test taker disengagement using response time. M.J. Margolis and R.A. Feinberg (Eds.). Integrating timing considerations to improve testing practices. p. 150–64. https://doi.org/10.4324/9781351064781-11.
Wise, S. L., DeMars, C. E. (2010). Examinee noneffort and the validity of program assessment results.
Educational Assessment. 15(1):27–41. https://doi.org/10.1080/10627191003673216.
Wise, S. L., Pastor, D. A., Kong, X. J. (2009). Correlates of rapid-guessing behavior in low-stakes testing: Implications for test development and measurement practice. Applied Measurement in Education. 22(2):185–205. https://doi.org/10.1080/08957340902754650.
Wise, S. L., Smith, L. F. (2011). A model of examinee test-taking effort. J. A. Bovaird, K. F. Geisinger, amd C. W. Buckendahl (Eds.). High-stakes testing in education: Science and practice in K-12 settings. American Psychological Association. p. 139–53. https://doi.org/10.1037/12330-009.
Young, M. S., Stanton, N. A. (2001). Mental workload: Theory, measurement and application. Karwowski, W. (Ed.). International Encyclopedia of Ergonomics and Human Factors. London: Taylor and Francis; Vol. 1. p. 507–9.