The Show Must Go On: Lessons Learned from using Remote Proctoring in a High-stakes Medical Licensing Exam Program in Response to Severe Disruption

The Show Must Go On: Lessons Learned from using Remote Proctoring in a High-stakes Medical Licensing Exam Program in Response to Severe Disruption

Authors

  • Maxim Morin
     Psychometrics and Assessment Services, Medical Council of Canada, 1021 Thomas Spratt Place. Ottawa, ON Canada K1G 5L5
  • Cecilia Alves
     Psychometrics and Assessment Services, Medical Council of Canada, Ottawa
  • André De Champlain
     Digital Health R&D, AstraZeneca, 1 Medimmune Way, Gaithersburg, MD 2087

Keywords:

Remote Proctoring, Remote Online Proctoring, High-Stake Testing, Licensing Examination, Computer-based Testing, Exam Comparability, Exam Equivalence

Abstract

The COVID-19 pandemic severely disrupted assessment models that were commonplace in the testing industry for decades. As a response to this disturbance, remote proctoring has emerged as a promising and potentially sound alternative to offer examinations, while adhering to public health authority guidelines. However, validity evidence in support of such a delivery modality with high-stake examinations, in comparison with traditional in-person testing, is still limited. The purpose of this study was to evaluate, compare and share lessons learned from the delivery of a high-stake medical licensing examination that was offered under two proctoring modalities. Over a few months, a total of 2070 candidates (46%) completed the exam in test centers, while 2475 (54%) completed it remotely. The analysis of technical difficulties suggests that the candidate experience was different across the two conditions. However, findings pertaining to candidate performance were encouraging and suggest that these discrepancies did not impact exam or examinee characteristics in any meaningful fashion. We believe that our findings therefore provide some support for the use of remote proctoring as a defensible alternative for completing a high-stakes examination.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Downloads

Published

2021-11-30

How to Cite

Morin, M., Alves, C., & De Champlain, A. (2021). The Show Must Go On: Lessons Learned from using Remote Proctoring in a High-stakes Medical Licensing Exam Program in Response to Severe Disruption. Journal of Applied Testing Technology, 23, 15–35. Retrieved from http://www.jattjournal.net/index.php/atp/article/view/165790

Issue

Volume 23, Special Issue, 2022

Section

Articles

References

Alessio, H. M., Malay, N., Maurer, K., Bailer, A. J., Rubin, B. (2017). Examining the effect of proctoring on online test scores. Online Learning. 21(1):146–61. https://doi.org/10.24059/olj.v21i1.885.

American Educational Research Association, American Psychology Association, National Council on Measurement in Education, and Joint Committee on Standards for Educational and Psychological Testing (U.S.). (2014). Standards for Educational and Psychological Testing. American Educational Research Association..

Attia, M. A. (2014). Postgraduate students’ perceptions toward online assessment: The case of the faculty of education, Umm Al-Qura university. In A. W. Wiseman, N. H. Alromi, and S. A. Alshumrani (Eds.), Education for a Knowledge Society in Arabian Gulf Countries (International Perspectives on Education and Society. 24:151–73). Emerald Group Publishing Limited. https://doi.org/10.1108/S1479-367920140000024015.

Beaty, J. C., Nye, C. D., Borneman, M. J., Kantrowitz, T. M., Drasgow, F., Grauer, E. (2011). Proctored versus unproctored internet tests: Are unproctored noncognitive tests as predictive of job performance? International Journal of Selection and Assessment. 19(1):1–10. https://doi.org/10.1111/j.1468-2389.2011.00529.x.

Butler-Henderson, K., Crawford, J. (2020). A systematic review of online examinations: A pedagogical innovation for scalable authentication and integrity. Computers and Education. 159:104024. https://doi.org/https://doi.org/10.1016/j.compedu.2020.104024.

Camara, W. (2020). Never let a crisis go to waste: Large-scale assessment and the response to COVID-19. Educational Measurement: Issues and Practice. 39(3):10–8. https://doi.org/https://doi.org/10.1111/emip.12358.

Cohen, J. (1988). Statistical power analysis for the behavorial sciences (2nd ed.). Routledge. https://doi.org/https://doi.org/10.4324/9780203771587.

College Board. (2020). Students take more than 4 million advanced placement exams online for the first time. Working to Claim College Credit. https://newsroom.collegeboard.org/students-take-more-4-million-advanced-placementexamsonline-first-time-working-claim-college.

Daffin Jr., L. W., Jones, A. A. (2018). Comparing student performance on proctored and non-proctored exams in online psychology courses. Online Learning. 22(1). https:// doi.org/10.24059/olj.v22i1.1079.

Davis, A. B., Rand, R., Seay, R. (2016). Remote proctoring: The effect of proctoring on grades. Advances in Accounting Education: Teaching and Curriculum Innovations. 18.

https://doi.org/10.1108/S1085-462220160000018002.

Dendir, S., Maxwell, R. S. (2020). Cheating in online courses: Evidence from online proctoring. Computers in Human Behavior Reports. 2:100033. https://doi.org/10.1016/j.chbr.2020.100033.

Elsalem, L., Al-Azzam, N., Jum’ah, A. A., Obeidat, N., Sindiani, A. M., Kheirallah, K. A. (2020). Stress and behavioral changes with remote e-exams during the COVID-19 pandemic: A cross-sectional study among undergraduates of medical sciences. Annals of Medicine and Surgery. 60:271–9. https://doi.org/10.1016/j.amsu.2020.10.058.

Foster, D., Layman, H. (2013). Online proctoring systems compared. Caveon Test Security. https://www.caveon.com/ wp-content/uploads/2014/03/Online-Proctoring-SystemsCompared-Mar-13-2013.pdf.

Hense, R., Golden, J. H., Burnett, J. (2009). Making the case for unproctored internet testing: Do the rewards outweigh the risks? Industrial and Organizational Psychology. 2(1):20–3. https://doi.org/10.1111/j.17549434.2008.01100.x.

Hollister, K. K., Berenson, M. L. (2009). Proctored versus unproctored online exams: Studying the impact of exam environment on student performance. Decision Sciences Journal of Innovative Education. 7(1):271–94. https://doi.org/10.1111/j.1540-4609.2008.00220.x.

Howard, D. (2019). Online testing integrity in a general education math course: A correlational study. https:// search.proquest.com/openview/b0235bd6a6ffa3d3864cdd 00a8848d3c/1?pq-origsite=gscholar&cbl=2026366&diss=y.

Hylton, K., Levy, Y., Dringus, L. P. (2016). Utilizing webcambased proctoring to deter misconduct in online exams. Computers and Education. 92-93:53–63. https:// doi.org/10.1016/j.compedu.2015.10.002.

Jiao, H., Lissitz, R. W. (2020). What hath the Coronavirus brought to assessment? Unprecedented Challenges in Educational Assessment in 2020 and Years to Come. Educational Measurement: Issues and Practice. 39(3):45–8. https://doi.org/10.1111/emip.12363.

Jodoin, M. G., Rubright, J. D. (2020). When examinees cannot test: The Pandemic’s Assault on Certification and Licensure. Educational Measurement: Issues and Practice. 39(3):31–3. https://doi.org/10.1111/emip.12361.

Karim, M. N., Kaminsky, S. E., Behrend, T. S. (2014). Cheating, reactions and performance in remotely proctored testing: An exploratory experimental study. Journal of Business and Psychology. 29(4):555–72. https://doi.org/10.1007/s10869-014-9343-z.

Langenfeld, T. (2020). Internet-based proctored assessment: Security and fairness issues. Educational Measurement: Issues and Practice. 39(3):24–7. https://doi.org/10.1111/emip.12359.

McCallin, R. C. (2016). Test administration. In Handbook of test development. Routledge. (Second Ed, pp. 567–84). . 23. Michel, R. S. (2020). Remotely proctored K-12 high stakes standardized testing during COVID-19: Will it Last? Educational Measurement: Issues and Practice. 39(3):28– 30. https://doi.org/10.1111/emip.12364 .

Milone, A. S., Cortese, A. M., Balestrieri, R. L., Pittenger, A. L. (2017). The impact of proctored online exams on the educational experience. Currents in Pharmacy Teaching and Learning. 9(1):108–14. https://doi.org/10.1016/j.cptl.2016.08.037.

Mozes-Carmel, A., Gold, S. S. (2009). A comparison of online vs. proctored final exams in online classes. Journal of Educational Technology. 6(1):76–81. https://doi.org/10.26634/jet.6.1.212.

National Association for College Admission Counseling. (2020). NACAC Statement on standardized testing during the COVID-19 emergency. National Association of Collegiate Admissions Counselors. https://www.nacacnet.org/newspublications/%0Anewsroom/press-releases/ Testing_Statement/%0A.

Plaus, K., Boren, L., Brazell, T. P., Wickett, J., Weber, A. (2015). Remote proctoring test delivery: A report on options and considerations. https://www.credentialingexcellence.org/blog/rsrch15-remote-proctoring-test-delivery-a-reportonoptions-and-considerations.

R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/.

Rios, J. A., Liu, O. L. (2017). Online proctored versus unproctored low-stakes Internet test administration: Is there differential test-taking behavior and performance? American Journal of Distance Education. 31(4):226–41. https://doi.org/10.1080/08923647.2017.1258628.

Schmidt, S. M. P., Ralph, D. L., Buskirk, B. (2009). Utilizing online exams: A case study. Journal of College Teaching and Learning (TLC). 6(8). https://doi.org/10.19030/tlc.v6i8.1108.

Sinharay, S., Johnson, M. S. (2020). The use of item scores and response times to detect examinees who may have benefited from item preknowledge. British Journal of Mathematical and Statistical Psychology. 73(3):397–419. https://doi.org/10.1111/bmsp.12187.

Sireci, S. G., Zenisky, A. L. (2016). Computerized innovative item formats: Achievement and credentialing. In Handbook of test development, Routledge/Taylor and Francis Group. 2nd ed. (pp. 313–34).

Stack, S. (2015). The impact of exam environments on student test scores in online courses. Journal of Criminal Justice Education. 26(3):273–82. https://doi.org/10.1080/10511253.2015.1012173.

Strauss, V. (2020). College Board says new online AP tests are going well - but students report big problems. The Washington Post. https://www.washingtonpost.com/education/2020/05/15/college-board-says-new-online-aptests-are-going-well-students-report-big-problems/.

Sun, L., Tang, Y., Zuo, W. (2020). Coronavirus pushes education online. Nature Material. 19(6):687. https://doi.org/10.1038/s41563-020-0678-8.

UNESCO. (2020). Managing high-stakes exams and assessments during the COVID-19 pandemic. https://unesdoc.unesco.org/ark:/48223/pf0000373247.

Way, W. D., Davis, L. L., Keng, L., Strain-Seymour, E. (2016). From standardization to personalization: The comparability of scores based on different testing conditions, modes and devices. Technology and Testing: Improving Educational and Psychological Measurement. 2:260–84.

Weiner, J., Hurtz, G. (2017). A comparative study of online remote proctored versus onsite proctored high-stakes exams. Journal of Applied Testing Technology. 18(1):13–20. http://jattjournal.com/index.php/atp/article/view/113061.

Wibowo, S., Grandhi, S., Chugh, R., Sawir, E. (2016). A pilot study of an electronic exam system at an Australian university. Journal of Educational Technology Systems. 45(1):5–33. https://doi.org/10.1177/0047239516646746 .

Wiley, A., Buckendahl, C. W. (2020). Your guess is as good as ours. Educational Measurement: Issues and Practice. 39(3):49–52. https://doi.org/10.1111/emip.12366