The Changing Landscape of Workplace-Based Assessment

The Changing Landscape of Workplace-Based Assessment

Authors

  • Vanessa C. Burch
     Honorary Professor, Department of Medicine, Faculty of Health Sciences, University of Cape Town, Rondebosch, Cape Town, 7700

Keywords:

Health Professions Education, Medical Education, Postgraduate Training, Workplace-based Assessment

Abstract

Health professions education has undergone radical changes over the past 100 years. This has necessitated a shift away from education programmes largely focused on testing knowledge and skills using predominantly written examinations. There has been a shift towards programmes which are intentionally designed with the end product in mind, a competent practitioner, who is able to provide safe, effective and efficient patient-centred health care. This major paradigm shift has placed an increased emphasis on what health care practitioners do in real clinical practice and the need for assessment processes which provide a holistic overview of trainees’ progress towards, and achievement of specified abilities required to undertake independent, unsupervised clinical practice. Such assessment processes, necessarily situated in the workplace, are collectively referred to as Workplace-Based Assessment (WBA). This paper provides an overview of the current state of WBA in health professions education, the challenges that need to be addressed and suggestions about the way forward. It is intended as an introductory text to the topic and a source of useful references to more detailed texts in the literature.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Downloads

Published

2019-05-30

How to Cite

Burch, V. C. (2019). The Changing Landscape of Workplace-Based Assessment. Journal of Applied Testing Technology, 20(S2), 37–59. Retrieved from http://www.jattjournal.net/index.php/atp/article/view/143675

Issue

Volume 20, Special Issue 2[Assessment in Medical Education], 2019

Section

Articles

References

Academy of Medical Royal Colleges (2016). Improving assessment: further guidance and recommendations. London: AoMRC. Available at http://www.aomrc.org.uk/ wp-content/uploads/2016/06/Improving_assessment_ Further_GR_0616-1.pdf.

Akl, E. A., Sackett, K. M., Pretorius, R., Bhoopathi, P. S. S., Mustafa, R., Schünemann, H. and Erdley, W. S. (2008).Educational games for health professionals. Cochrane Database of Systematic Reviews, (1). https://doi.org/10.1002/14651858.CD006411.pub2. PMid: 18254103.

Akl, E. A., Pretorius, R. W., Sackett, K., Erdley, W. S., Bhoopathi, P. S., Alfarah, Z. and Schünemann, H. J.

(2010). The effect of educational games on medical students’ learning outcomes: a systematic review: BEMEGuide No 14. Medical Teacher, 32(1), 16−27. https://doi.org/10.3109/01421590903473969. PMid: 20095770.

Archer, J., Norcini, J., Southgate, L., Heard, S. and Davies, H.(2008). mini-PAT (Peer Assessment Tool): A valid component of a national assessment programme in theUK? Advances in Health Sciences Education: Theory and Practice, 13(2), 181−92. https://doi.org/10.1007/s10459006-9033-3. PMid:17036157.

Baker, R. S. and Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3−17.

Baker, R. S. and Inventado, P. S. (2014). Educational Data Mining and Learning Analytics. In: Larusson J., White, B. (Eds.) Learning Analytics, Springer, New York, NY. p.61−75. PMCid: PMC4355053.

Basu, I., Parvizi, S. and Chin, K. (2013). The perception of online workâ€based assessments. The Clinical Teacher, 10(2), 73−77. https://doi.org/10.1111/j.1743498X.2012.00609.x. PMid: 23480106.

Batalden, P., Leach, D., Swing, S., Dreyfus, H. and Dreyfus, S.(2002). General competencies and accreditation in graduate medical education. Health Affairs, 21(5), 103−11.https://doi.org/10.1377/hlthaff.21.5.103. PMid: 12224871.

Beard, J. and Bussey, M. (2007). Workplace-based assessment. The Bulletin of the Royal College of Surgeons of England, 89(5), 158−60. https://doi.org/10.1308/147363507X196549.

Bindal, T., Wall, D. and Goodyear, H. M. (2011). Trainee doctors’ views on workplace-based assessments: are they just a tick box exercise? Medical Teacher, 33(11), 919−27.https://doi.org/10.3109/0142159X.2011.558140. PMid: 22022902.

Bindal, N., Goodyear, H., Bindal, T. and Wall, D. (2013).DOPS assessment: A study to evaluate the experience and opinions of trainees and assessors. Medical Teacher, 35(6), e1230−34. https://doi.org/10.3109/01421 59X.2012.746447. PMid: 23627359.

Black, P. and Wiliam, D. (1998). Assessment and classroom learning.Assessment in Education: principles, Policy & Practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102.

Bodle, J. F., Kaufmann, S. J., Bisson, D., Nathanson, B. and Binney, D. M. (2008). Value and face validity of objective structured assessment of technical skills (OSATS) forwork based assessment of surgical skills in obstetrics and gynaecology. Medical Teacher, 30(2), 212−16. https://doi.org/10.1080/01421590701881624. PMid: 18464149.

Bok, H. G., Teunissen, P. W., Favier, R. P., Rietbroek, N. J., Theyse, L. F., Brommer, H., Haarhuis, J. C., van Beukelen, P., van der Vleuten, C. P. and Jaarsma, D. A. (2013). Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Medical Education, 13(1), 123. https://doi.org/10.1186/1472-6920-13-123.PMid: 24020944, PMCid: PMC3851012.

Boud, D. (1990). Assessment and the promotion of academic values. Studies in Higher Education, 15(1), 101−11. https:// doi.org/10.1080/03075079012331377621.

Boulet, J. R., McKinley, D. W., Norcini, J. J. and Whelan, G. P.(2002). Assessing the comparability of standardized patient and physician evaluations of clinical skills. Advances in Health Sciences Education: Theory and Practice, 7(2), 85−97. https://doi.org/10.1023/A:1015750009235. PMid: 12075142.

Braddock, C. H., Fihn, S. D., Levinson, W., Jonsen, A. R.and Pearlman, R. A. (1997). How doctors and patients discuss routine clinical decisions: informed decision making in the outpatient setting. Journal of General Internal Medicine, 12(6), 339−45. https://doi. org/10.1007/s11606-006-5081-x, https://doi.org/10.1046/ j.1525-1497.1997.00057.x. PMid: 9192250, PMCid: PMC1497128.

Brinkman, W. B., Geraghty, S. R., Lanphear, B. P., Khoury, J. C., del Rey, J. A. G., DeWitt, T. G. and Britto, M. T. (2007). Effect of multisource feedback on resident communication skills and professionalism: a randomized controlled trial. Archives of Pediatrics and Adolescent Medicine, 161(1), 44-49. https://doi.org/10.1001/archpedi.161.1.44.PMid: 17199066.

Brown, S. (2005). Assessment for learning. Learning and Teaching in Higher Education, (1), 81−89.

Buckley, S., Coleman, J., Davison, I., Khan, K. S., Zamora, J., Malick, S., Morley, D., Pollard, D., Ashcroft, T., Popovic, C. and Sayers, J. (2009). The educational effects of portfolios on undergraduate student learning: A Best Evidence Medical Education (BEME) systematic review, BEME Guide No. 11. Medical Teacher, 31(4), 282−98. https://doi.org/10.1080/01421590902889897. PMid: 19404891.

Burford, B., Illing, J., Kergon, C., Morrow, G. and Livingston, M. (2010). User perceptions of multi-source feedback tools for junior doctors. Medical Education, 44(2), 165-76. https://doi.org/10.1111/j.1365-2923.2009.03565.x. PMid: 20059677.

Care E. (2018). Twenty-First Century Skills: From Theory to Action. In: Care E., Griffin P., Wilson M. (Eds.) Assessment and Teaching of 21st Century Skills. Educational Assessment in an Information Age. p. 3−17. Cham: Springer. https://doi.org/10.1007/978-3-319-65368-6_1.

Carraccio, C., Wolfsthal, S. D., Englander, R., Ferentz, K. and Martin, C. (2002). Shifting paradigms: from Flexner to competencies.Academic Medicine, 77(5), 361−67. https://doi. org/10.1097/00001888-200205000-00003. PMid: 12010689.

Carraccio, C. L. and Englander, R. (2013). From Flexner to competencies: reflections on a decade and the journey ahead. Academic Medicine, 88(8), 1067−73. https://doi.org/10.1097/ACM.0b013e318299396f. PMid: 23807096.

Castanelli, D. and Kitto, S. (2011). Perceptions, attitudes, and beliefs of staff anaesthetists related to multi-source feedback used for their performance appraisal. British Journal of Anaesthesia, 107(3), 372-377. https://doi.org/10.1093/ bja/aer152. PMid: 21676891.

Cheston, C. C., Flickinger, T. E. and Chisolm, M. S. (2013). Social media use in medical education: A systematic review. Academic Medicine, 88(6), 893−901. https://doi. org/10.1097/ACM.0b013e31828ffc23. PMid: 23619071.

Cho, S. P., Parry, D. and Wade, W. (2014). Lessons learnt from a pilot of assessment for learning. Clinical Medicine, 14(6), 577−84. https://doi.org/10.7861/clinmedicine.14-6-577. PMid: 25468839, PMCid: PMC4954126.

Choules, A. P. (2007). The use of e-learning in medical education: a review of the current situation. Postgraduate Medical Journal, 83(978), 212−16. https://doi. org/10.1136/pgmj.2006.054189. PMid:17403945, PMCid: PMC2600032.

Cleland, J. A., Knight, L. V., Rees, C. E., Tracey, S. and Bond, C. M. (2008). Is it me or is it them? Factors that influence the passing of underperforming students. Medical Education, 42(8), 800–809. https://doi.org/10.1111/ j.1365-2923.2008.03113.x. PMid: 18715477.

Cohen, S. N., Farrant, P. B. J. and Taibjee, S. M. (2009).Assessing the assessments: UK dermatology trainees’ views of the workplace assessment tools. British Journal of Dermatology, 161(1), 34–39. https://doi.org/10.1111/ j.1365-2133.2009.09097.x. PMid: 19416267.

Collins, J. (2010). Foundation for excellence: An evaluation of the Foundation Programme. London: Medical Education England.

Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J. and Montori, V. M. (2008). Internet-based learning in the health professions: a meta-analysis. Journal of the American Medical Association, 300(10), 1181−96.https://doi.org/10.1001/jama.300.10.1181, https://doi. org/10.1503/cmaj.071366. PMCid: PMC2292790.

Cook, D. A. (2009a). The failure of e-learning research to inform educational practice and what we can do about it. Medical Teacher, 31(2), 158−62. https://doi. org/10.1080/01421590802691393. PMid: 19330674.

Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G. and Pankratz, V. S. (2009b). Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74–79. https://doi.org/10.1007/s11606-008-0842-3. PMid: 19002533, PMCid: PMC2607488.

Cook, D. A. and Beckman, T. J. (2009c). Does scale length matter? A comparison of nine-versus five-point rating scales for the mini-CEX. Advances in Health Sciences Education: Theory and Practice, 14(5), 655−64. https://doi.org/10.1007/s10459-008-9147-x. PMid: 19034679.

Cook, D. A. and Triola, M. M. (2009d). Virtual patients: a critical literature review and proposed next steps. Medical Education, 43(4), 303−11. https://doi.org/10.1111/j.13652923.2008.03286.x. PMid: 19335571.

Cook, D. A., Hatala, R., Brydges, R., Zendejas, B., Szostek, J.H., Wang, A. T., Erwin, P. J. and Hamstra, S. J. (2011).Technology-enhanced simulation for health professions education: A systematic review and meta-analysis. Journal of the American Medical Association, 306(9), 978−88.https://doi.org/10.1001/jama.2011.1234. PMid: 21900138.

Cook, D. A., Brydges, R., Hamstra, S. J., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J. and Hatala, R. (2012). Comparative effectiveness of technology-enhanced simulation versus other instructional methods: A systematic review and meta-analysis. Simulation in Healthcare, 7(5), 308−20. https://doi.org/10.1097/SIH.0b013e3182614f95. PMid: 23032751.

Cook, D. A., Brydges, R., Zendejas, B., Hamstra, S. J. and Hatala, R. (2013a). Mastery learning for health professionals using technology-enhanced simulation: A systematic review and meta-analysis. Academic Medicine, 88(8), 1178−86.https://doi.org/10.1097/ACM.0b013e31829a365d. PMid: 23807104.

Cook, D. A., Brydges, R., Zendejas, B., Hamstra, S. J. and Hatala, R. (2013b). Technology-enhanced simulation to assess health professionals: A systematic review of validity evidence, research methods, and reporting quality. Academic Medicine, 88(6), 872−83. https://doi.org/10.1097/ ACM.0b013e31828ffdcf. PMid:23619073.

Cooper, A. (2012). What is analytics? Definition and essential characteristics. CETIS Analytics Series, 1(5), 1−10.

Crossley, J. and Jolly, B. (2012). Making sense of workâ€based assessment: Ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46(1), 28−37. https://doi.org/10.1111/j.13652923.2011.04166.x. PMid: 22150194.

Davies, H., Archer, J., Heard, S. and Southgate, L. (2005). Assessment tools for foundation programmes - A practical guide. British Medical Journal, 330(7500), s195−96. https://doi.org/10.1136/bmj.330.7500.s195a.

Davies, H., Archer, J., Southgate, L. and Norcini, J. (2009). Initial evaluation of the first year of the Foundation Assessment Programme. Medical Education 43(1), 74–81. https://doi. org/10.1111/j.1365-2923.2008.03249.x. PMid: 19141000.

Dean, B. J. F. and Duggleby, P. M. (2013). Foundation doctors’ experience of their training: a questionnaire study.JRSM Short Reports, 4(1), 1−7. https://doi.org/10.1258/ shorts.2012.012095. PMid: 23413407, PMCid:PMC3572662.

De Lima, A. A., Barrero, C., Baratta, S., Castillo Costa, Y., Bortman, G., Carabajales, J., Conde, D., Galli, A., Degrange, G. and van der Vleuten, C. (2007). Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training. Medical Teacher, 29(8), 785−90. https://doi.org/10.1080/01421590701352261. PMid: 17917984.

De Lima, A. A., Conde, D., Costabel, J., Corso, J. and Van der Vleuten, C. (2013). A laboratory study on the reliability estimations of the mini-CEX. Advances in Health Sciences Education: Theory and Practice, 18, 5−13. https:// doi.org/10.1007/s10459-011-9343-y. PMid: 22193944, PMCid: PMC3569586.

Dennis, A. A., Foy, M. J., Monrouxe, L. V. and Rees, C. E. (2018). Exploring trainer and trainee emotional talk in narratives about workplace-based feedback processes. Advances in Health Sciences Education: Theory and Practice, 23(1), 75-93. https://doi.org/10.1007/s10459-017-9775-0. PMid: 28456856, PMCid: PMC5801389.

Dijksterhuis, M. G. K., Schuwirth, L. W. T., Braat, D. D. M., Teunissen, P. W. and Scheele, F. (2013). A qualitative study on trainees’ and supervisors’ perceptions of assessment for learning in postgraduate medical education. Medical Teacher, 35(8), e1396–402. https://doi.org/10.3109/01421 59X.2012.756576. PMid: 23600668.

Donato, A. A., Pangaro, L., Smith, C., Rencic, J., Diaz, Y., Mensinger, J. and Holmboe, E. (2008). Evaluation of a novel assessment form for observing medical residents: A randomised, controlled trial. Medical Education, 42(12), 1234−42. https://doi.org/10.1111/j.13652923.2008.03230.x. PMid: 19120955.

Driessen, E., van der Vleuten, C., Schuwirth, L., van Tartwijk, J. and Vermunt, J. D. H. M. (2005). The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: A case study. Medical Education, 39(2), 214−20. https://doi.org/10.1111/j.13652929.2004.02059.x. PMid: 15679689.

Driessen, E., Van Tartwijk, J., van der Vleuten, C. and Wass, V. (2007a). Portfolios in medical education: why do they meet with mixed success? A systematic review. Medical Education, 41(12), 1224-1233. https://doi.org/10.1111/ j.1365-2923.2007.02944.x. PMid: 18045373.

Driessen, E. W., Muijtjens, A. M., van Tartwijk, J. and van der Vleuten, C. P. (2007b). Webâ€or paperâ€based portfolios: Is there a difference? Medical Education, 41(11), 1067−73.https://doi.org/10.1111/j.1365-2923.2007.02859.x. PMid: 17973767.

Dudek, N. L., Marks, M. B. and Regehr, G. (2005). Failure to fail: the perspective of clinical supervisors. Academic Medicine, 80(10 suppl.), S84−87. https://doi.org/10.1097/00001888200510001-00023.

Durning, S. J., Cation, L. J., Markert, R. J. and Pangaro, L.N. (2002). Assessing the reliability and validity of the mini—clinical evaluation exercise for internal medicine residency training. Academic Medicine, 77(9), 900-904.https://doi.org/10.1097/00001888-200209000-00020.PMid: 12228088.

Ellaway, R. and Masters, K. (2008). AMEE Guide 32: e-Learning in medical education Part 1: learning, teaching and assessment. Medical Teacher, 30(5), 455−73. https://doi.org/10.1080/01421590802108331. PMid: 18576185.

Ellaway, R. H., Pusic, M. V., Galbraith, R. M. and Cameron, T. (2014). Developing the role of big data and analytics in health professional education. Medical Teacher, 36(3), 216−22. https://doi.org/10.3109/0142159X.2014.874553.PMid: 24491226.

Englander, R., Frank, J. R., Carraccio, C., Sherbino, J., Ross, S., Snell, L. and ICBME Collaborators. (2017). Toward a shared language for competency-based medical education. Medical Teacher, 39(6), 582-587. https://doi.org/10. 1080/0142159X.2017.1315066. PMid: 28598739.

Ferenchick, G. S., Foreback, J., Towfiq, B., Kavanaugh, K., Solomon, D. and Mohmand, A. (2010). The implementation of a mobile problem-specific electronicCEX for assessing directly observed student–patient encounters. Medical Education online, 15(1), 4276.https://doi.org/10.3402/meo.v15i0.4276. PMid: 20174616, PMCid: PMC2823396.

Ferenchick, G. S., Solomon, D., Foreback, J., Towfiq, B., Kavanaugh, K., Warbasse, L., Addison, J., Chames, F., Dandan, A. and Mohmand, A. (2013). Mobile technology for the facilitation of direct observation and assessment of student performance. Teaching and Learning in Medicine, 25(4), 292−99. https://doi.org/10.1080/10401334.2013.82 7972. PMid: 24112197.

Finall, A. (2012). Trainers’ perceptions of the direct observation of practical skills assessment in histopathology training: a qualitative pilot study. Journal of Clinical Pathology, 65, 538−40. https://doi.org/10.1136/jclinpath-2012-200682.PMid: 22447913.

Fokkema, J. P., Teunissen, P. W., Westerman, M., van der Lee, N., van der Vleuten, C. P., Scherpbier, A. J., Dörr, P. J. and Scheele, F. (2013). Exploration of perceived effects of innovations in postgraduate medical education. Medical Education, 47(3), 271−81. https://doi.org/10.1111/ medu.12081. PMid: 23398013.

Frank, J. R. and Danoff, D. (2007). The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Medical Teacher, 29(7), 642−47. https://doi. org/10.1080/01421590701746983. PMid: 18236250.

Frank, J. R., Mungroo, R., Ahmad, Y., Wang, M., De Rossi, S. and Horsley, T. (2010a). Toward a definition of competencybased education in medicine: a systematic review of published definitions. Medical Teacher, 32(8), 631−37.https://doi.org/10.3109/0142159X.2010.500898. PMid: 20662573.

Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., Harris, P., Glasgow, N. J., Campbell, C., Dath, D. & Harden, R. M. (2010b). Competency-based medical education: Theory to practice. Medical Teacher, 32(8), 638−45. https://doi.org/10.3109/0142159X.2010.501190. PMid:20662574

Frank, J. R., Snell, L., Englander, R., Holmboe, E. S. and ICBME Collaborators. (2017). Implementing competency-based medical education: moving forward. Medical Teacher, 568573. https://doi.org/10.1080/0142159X.2017.1315069.PMid: 28598743.

Frehywot, S., Vovides, Y., Talib, Z., Mikhail, N., Ross, H., Wohltjen, H., Bedada, S., Korhumel, K., Koumare, A.K. and Scott, J. (2013). E-learning in medical education in resource constrained low-and middle-income countries.Human Resources for Health, 11(1), 4. https://doi.org/10.1186/1478-4491-11-4. PMid: 23379467, PMCid: PMC3584907.

Frenk, J., Chen, L., Bhutta, Z.A., Cohen, J., Crisp, N., Evans, T., Fineberg, H., Garcia, P., Ke, Y., Kelley, P. and Kistnasamy, B. (2010). Health professionals for a new century: transforming education to strengthen health systems in aninterdependent world. The Lancet, 376 (9756), 1923−58.https://doi.org/10.1016/S0140-6736(10)61854-5.

General Medical Council (2011). Learning and Assessment in The Clinical Environment: The Way Forward. London: GMC.

General Medical Council (2018). National Training Surveys 2018. London: GMC. Available at https://www.gmc-uk.org/-/media/documents/dc11391-nts-2018-initial-findingsreport_pdf-75268532.pdf.

General Medical Council (2013). Good Medical Practice.Available at: https://www.gmc-uk.org/-/media/documents/ good-medical-practice---english-1215_pdf-51527435.pdf.

Ginsburg, S., Eva, K. and Regehr, G. (2013). Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Academic Medicine, 88(10), 1539−44.https://doi.org/10.1097/ACM.0b013e3182a36c3d. PMid: 23969371.

Ginsburg, S., van der Vleuten, C., Eva, K. W. and Lingard, L.(2016). Hedging to save face: a linguistic analysis of written comments on in-training evaluation reports. Advances in Health Sciences Education: Theory and Practice, 21(1), 175−88. https://doi.org/10.1007/s10459-015-9622-0. PMid: 26184115.

Govaerts, M. J., van der Vleuten, C. P., Schuwirth, L. W. and Muijtjens, A. M. (2007). Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment. Advances in Health Sciences Education: Theory and Practice, 12 (2), 239-260. PMCid: PMC3557509.

Govaerts, M. & van der Vleuten, C. P. (2013). Validity in workbased assessment: Expanding our horizons. MedicalEducation, 47(12), 1164−74. https://doi.org/10.1111/ medu.12289 PMid: 24206150.

Govaerts, M. J. B., van de Wiel, M. W. J., Schuwirth, L. W. T., van der Vleuten, C. P. M. & Muijtjens, A. M. M. (2013).Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education: Theory and Practice, 18(3), 375−96.

Grieveson, B., Kirton, J. A., Palmer, N. O. A. and Balmer, M. C. (2011). Evaluation of workplace based assessment tools in dental foundation training. British Dental Journal, 211(4), E8. https://doi.org/10.1038/sj.bdj.2011.681. PMid: 21869769.

Griffin, P. and Care, E. (2015). The ATC21S Method. In: Griffin P., Care E. (Eds.) Assessment and Teaching of 21st Century Skills. Educational Assessment in an Information Age, Dordrecht: Springer. p. 3−33. https:// doi.org/10.1007/978-94-017-9395-7_1.

Guba, E. G. and Lincoln, Y. S. (1989). What is this constructivist paradigm anyway? In Guba, E. A. & Lincoln Y. S. (Eds.). Fourth Generation Evaluation, Newbury Park, California: Sage. p. 79−117. PMCid: PMC304045.

Hammoud, M. M., Dalrymple, J. L., Christner, J. G., Stewart, R. A., Fisher, J., Margo, K., Ali, I. I., Briscoe, G. W. and Pangaro, L.N. (2012). Medical student documentation in electronic health records: a collaborative statement from the Alliance for Clinical Education. Teaching and Learning in Medicine, 24(3), 257−66. https://doi.org/10.1080/10401 334.2012.692284. PMid: 22775791.

Harden, R. M. (1999). AMEE Guide No. 14: Outcome-based education: Part 1- An introduction to outcome-based education. Medical Teacher, 21(1), 7−14.

Harris, P., Bhanji, F., Topps, M., Ross, S., Lieberman, S., Frank, J.R., Snell, L., Sherbino, J. & ICBME Collaborators. (2017). Evolving concepts of assessment in a competency-based world. Medical Teacher, 39(6), p. 603−08. https://doi.org/1 0.1080/0142159X.2017.1315071. PMid: 28598736.

Hatala, R. and Norman, G. R. (1999). In-training evaluation during an internal medicine clerkship.Academic Medicine, 74(10), S118−20. https://doi.org/10.1097/00001888-199910000-00059.

Hattie, J. and Timperley, H. (2007). The power of feedback.Review of Educational Research, 77(1), 81−112.

https://doi.org/10.3102/003465430298487.

Hauer, K. E. (2000). Enhancing feedback to students using the mini-CEX (Clinical Evaluation Exercise). Academic Medicine, 75(5), 524. https://doi.org/10.1097/00001888200005000-00046. PMid:10824798.

Hauer, K. E., Holmboe, E. S. and Kogan, J. R. (2011). Twelve tips for implementing tools for direct observation of medical trainees’ clinical skills during patient encounters. Medical Teacher, 33(1), 27−33. https://doi.org/10.3109/01421 59X.2010.507710. PMid: 20874011.

Heeneman, S. and de Grave, W. (2017). Tensions in mentoring medical students toward self-directed and reflective learning in a longitudinal portfolio-based mentoring system - An activity theory analysis. Medical Teacher, 39(4), 368−76. https://doi.org/10.1080/0142159X.2017.1286308.PMid: 28379081.

Hodges, B. (2013). Assessment in the post-psychometric era: Learning to love the subjective and collective. Medical Teacher, 35(7), 564−68. https://doi.org/10.3109/01421 59X.2013.789134. PMid: 23631408.

Holmboe, E. S., Hawkins, R. E. and Huot, S. J. (2004). Effects of training in direct observation of medical residents’ clinical competence. Annals of Internal Medicine, 140(11), 874–81. https://doi.org/10.7326/0003-4819-140-11200406010-00008. PMid: 15172901.

Holmboe, E.E., Davis, H.M and Carraccio, C. (2017). Portfolios.In Holmboe, E. S., Hawkins, R. E. (Eds.). Practical Guide to the Evaluation of Clinical Competence. 2nd ed., Philadelphia: Elsevier Health Sciences. p. 270−87.

Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., Frank, J. R. and International CBME Collaborators. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676−82. https://doi.org/10.3 109/0142159X.2010.500704. PMid: 20662580.

Holmboe, E. S., Ward, D. S., Reznick, R. K., Katsufrakis, P. J., Leslie, K. M., Patel, V. L., Ray, D. D. and Nelson, E. A. (2011). Faculty development in assessment: The missing link in competency-based medical education. Academic Medicine, 86(4), 460−67. https://doi.org/10.1097/ ACM.0b013e31820cb2a7. PMid:21346509.

Holmboe, E.S. and Harden, R.M. (2017). Outcome-based education. In: Dent, J.mA., Harden, R. M., Hunt, D. (Eds.). APractical Guide for Medical Teachers, 5th ed., Edinburgh: Elsevier. p. 114−21.

Hrisos, S., Illing, J. C. and Burford, B. C. (2008). Portfolio learning for foundation doctors: Early feedback on its use in the clinical workplace. Medical Education, 42(2), 214– 23. https://doi.org/10.1111/j.1365-2923.2007.02960.x. PMid:18230095.

Isles, M. and Pugsley, L. (2010). Surgical procedure-based assessment: how did we get here and where are we going? The Bulletin of the Royal College of Surgeons of England, 92(7), 1−3. https://doi.org/10.1308/147363510X511625.

Jenkins, L., Mash, B. and Derese, A. (2013). The national portfolio of learning for postgraduate family medicine training in South Africa: experiences of registrars and supervisors in clinical practice. BMC Medical Education, 13(1), 149. https://doi.org/10.1186/1472-6920-13-149. PMid: 24207009, PMCid: PMC4226197.

Jones, R., Higgs, R., De Angelis, C. and Prideaux, D. (2001). Changing face of medical curricula. The Lancet, 357(9257), 699−703. https://doi.org/10.1016/S0140-6736(00)041349.

Jurd, S., de Beer, W., Aimer, M., Fletcher, S., Halley, E., Schapper, C. and Orkin, M. (2015). Introducing a competency based fellowship programme for psychiatry in Australia and New Zealand. Australasian Psychiatry, 23(6), 699−705. https:// doi.org/10.1177/1039856215600898. PMid: 26400448.

Jyothirmayi, R. (2012). Case-based discussion: assessment tool or teaching aid? Clinical Oncology, 24(10), 649−53. https:// doi.org/10.1016/j.clon.2012.07.008. PMid:22892380.

Kirton, J., Palmer, N., Grieveson, B. and Balmer, M. (2013). A national evaluation of workplace-based assessment tools (WPBAs) in foundation dental training: A UK study. Effective and useful but do they provide an equitable training experience? British Dental Journal, 214(6), 305–09.https://doi.org/10.1038/sj.bdj.2013.302. PMid:23518981.

Kogan, J. R., Holmboe, E. S. and Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. Journal of the American Medical Association, 302(12), 1316−26. https:// doi.org/10.1001/jama.2009.1365. PMid:19773567.

Kogan, J. R., Hess, B. J., Conforti, L. N. and Holmboe, E. S.(2010). What drives faculty ratings of residents’ clinical skills? The impact of faculty’s own clinical skills. Academic Medicine, 85(Suppl. 10), S25–28. https://doi.org/10.1097/ ACM.0b013e3181ed1aa3. PMid: 20881697.

Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W. and Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. MedicalEducation, 45(10), 1048−60. https://doi.org/10.1111/ j.1365-2923.2011.04025.x. PMid:21916943.

Kogan, J.R. and Holmboe, E. (2013). Realizing the promise and importance of performance-based assessment. Teaching and Learning in Medicine, 25(Suppl. 1), S68-S74. https:// doi.org/10.1080/10401334.2013.842912. PMid:24246110.

Kogan, J. R., Conforti, L. N., Iobst, W. F. and Holmboe, E. S.(2014). Reconceptualizing variable rater assessments as both an educational and clinical care problem. Academic Medicine, 89(5), 721−27. https://doi.org/10.1097/ ACM.0000000000000221. PMid: 24667513.

Lincoln, Y. S. and Guba, E. G. (1985). Establishing trustworthiness. In: Lincoln, Y. S., Guba, E. A. (Eds.). Naturalistic Inquiry. Beverly Hills, California: Sage. p. 289−332.

Lockyer, J., Violato, C. and Fidler, H. (2003). Likelihood of change: a study assessing surgeon use of multisource feedback data. Teaching and Learning in Medicine, 15(3), 168−74. https://doi.org/10.1207/S15328015TLM1503_04. PMid:12855387

Lockyer, J., Carraccio, C., Chan, M.K., Hart, D., Smee, S., Touchie, C., Holmboe, E.S., Frank, J.R. and ICBME Collaborators. (2017). Core principles of assessment in competencybased medical education. Medical Teacher, 39(6), 609−16. https://doi.org/10.1080/0142159X.2017.1315082. PMid: 28598746.

Lörwald, A. C., Lahner, F. M., Greif, R., Berendonk, C., Norcini, J. and Huwendiek, S. (2018). Factors influencing the educational impact of Mini-CEX and DOPS: a qualitative synthesis. Medical Teacher, 40(4), 414−20. https://doi.org/ 10.1080/0142159X.2017.1408901. PMid: 29188739.

Lurie, S. J., Mooney, C. J. and Lyness, J. M. (2009). Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review. Academic Medicine, 84(3), 301−09. https://doi.org/10.1097/ACM.0b013e3181971f08. PMid:19240434.

Maatsch, J.L., Huang, R., Downing, S. and Barker, B. (1983).Predictive validity of medical specialist examinations. Final report for Grant HS 02038-04, National Center of Health Services Research. Office of Medical Education Research and Development, Michigan State University, East Lansing, MI. PMid:6631930.

Macfadyen, L. P., Dawson, S., Pardo, A. and Gaševic, D. (2014).Embracing big data in complex educational systems: The learning analytics imperative and the policy challenge.Research and Practice in Assessment, 9, 17−28.

Makary, M. A. and Daniel, M. (2016). Medical error—the third leading cause of death in the US. British Medical Journal (Online), 353, i2139. https://doi.org/10.1136/bmj.i2139.

Malhotra, S., Hatala, R. and Courneya. C. A. (2008). Internal medicine residents’ perceptions of the mini-clinical evaluation exercise. Medical Teacher, 30(4), 414−19. https:// doi.org/10.1080/01421590801946962. PMid:18569664.Margolis,

M. J., Clauser, B. E., Cuddy, M. M., Ciccone, A., Mee, J., Harik, P. and Hawkins, R. E.. (2006). Use of the mini clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine, 81(10 Suppl.), S56–60. https://doi.org/10.1097/01.ACM.0000236514.53194.f4.PMid:17001137.

Marriott, J., Purdie, H., Crossley, J. and Beard, J. D. (2011).Evaluation of procedureâ€based assessment for assessing trainees’ skills in the operating theatre. British Journal of Surgery, 98(3), 450−57. https://doi.org/10.1002/bjs.7342.PMid:21254025.

Marshall, G.N. and Hays, R.D. (1994). The Patient Satisfaction Questionnaire Short Form (PSQ-18). Santa Monica, CA: RAND Corporation.

Martin, J. A., Regehr, G., Reznick, R., Macrae, H., Murnaghan, J., Hutchison, C. and Brown, M. (1997). Objective Structured Assessment Of Technical Skill (OSATS) for surgical residents. British Journal of Surgery, 84(2), 273−78. https:// doi.org/10.1046/j.1365-2168.1997.02502.x, https://doi. org/10.1002/bjs.1800840237. PMid:9052454.

Massie, J. and Ali, J. M. (2016). Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings. Advances in Health Sciences Education: Theory and Practice, 21(2), 455−73. https://doi.org/10.1007/s10459-015-9614-0. PMid: 26003590.

Masters, K. and Ellaway, R. (2008). e-Learning in medical education. AMEE Guide 32 Part 2: technology, management and design. Medical Teacher, 30(5), 474−89. https://doi. org/10.1080/01421590802108349. PMid: 18576186.

Masters, K., Ellaway, R. H., Topps, D., Archibald, D. and Hogue, R. J. (2016). Mobile technologies in medical education: AMEE Guide No. 105. Medical Teacher, 38(6), 537−49. https://doi.org/10.3109/0142159X.2016.1141190. PMid: 27010681.

McGaghie, W. C., Issenberg, S. B., Barsuk, J. H. and Wayne, D. B. (2014). A critical review of simulationâ€based mastery learning with translational outcomes. MedicalEducation, 48(4), 375−85. https://doi.org/10.1111/ medu.12391. PMid: 24606621.

McGill, D. A., van der Vleuten, C. P. M. and Clarke, M. J. (2013).A critical evaluation of the validity and the reliability of global competency constructs for supervisor assessment of junior medical trainees. Advances in Health Sciences Education: Theory and Practice, 18(4), 701−25. https://doi.org/10.1007/s10459-012-9410-z. PMid: 23053869.

McKavanagh, P., Smyth, A. and Carragher, A. (2012). Hospital consultants and workplace based assessments: how foundation doctors view these educational interactions?Postgraduate Medical Journal, 88(1037), 119–24. https:// doi.org/10.1136/postgradmedj-2011-130121. PMid: 22247316.

McKee, R. F. (2008). The Intercollegiate Surgical Curriculum Programme (ISCP). Surgery (Oxford), 26(10), 411−16.https://doi.org/10.1016/j.mpsur.2008.08.007.

Mehta, F., Brown, J. and Shaw, N. J. (2013). Do trainees value feedback in case-based discussion assessments? Medical Teacher, 35(5), e1166–72. https://doi.org/10.3109/01421 59X.2012.731100. PMid: 23137264.

Miller, A. and Archer, J. (2010). Impact of workplace-based assessment on doctors’ education and performance: A systematic review. British Medical Journal, 341, c5064.https://doi.org/10.1136/bmj.c5064. PMid: 20870696, PMCid: PMC2945627.

Miller, G. E. (1990). The assessment of clinical skills/competence/ performance. Academic Medicine, 65(Suppl. 9), S63−67. https://doi.org/10.1097/00001888-19900900000045.

Mislevy, R. J., Behrens, J. T., Dicerbo, K. E. and Levy, R. (2012). Design and discovery in educational assessment: Evidence-centered design, psychometrics, andeducational data mining. Journal of Educational Data Mining, 4(1), 11−48.

Mitka M. (2008). Many physician practices fall short on accurate blood pressure measurement. Journal of the American Medical Association, 299(24), 2842–44. https://doi. org/10.1001/jama.299.24.2842. PMid: 18577720.

Moonen-van Loon, J. M. W., Overeem, K., Donkers, H. H. L. M., Van der Vleuten, C. P. M. and Driessen, E. W. (2013). Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Advances in Health Sciences Education: Theory and Practice, 18(5), 1087−102. https://doi.org/10.1007/s10459-013-9450-z.PMid: 23494202.

Moores, A. and Parks, M. (2010). Twelve tips for introducing E-Portfolios with undergraduate students.

Medical Teacher, 32(1), 46−49. https://doi. org/10.3109/01421590903434151. PMid: 20095774.

Morris, A., Hewitt, J. and Roberts, C. M. (2006). Practical experience of using directly observed procedures, mini clinical evaluation examinations, and peer observation in preregistration house officer (FY1) trainees. Postgraduate

Medical Journal, 82(966), 285−88. https://doi. org/10.1136/pgmj.2005.040477. PMid: 16597818, PMCid: PMC2579636.

Morrison, K. and van der Werf, G. (2016). Large-scale data,“wicked problemsâ€, and “what works†for educational policy making. Educational Research and Evaluation, 22(5-6), 255−59. https://doi.org/10.1080/13803611.2016.1259789.

Morse, J. M. (2015). Critical analysis of strategies for determining rigor in qualitative inquiry. Qualitative

Health Research, 25(9), 1212−22. https://doi. org/10.1177/1049732315588501. PMid: 26184336.

Murphy, D. J., Bruce, D. A., Mercer, S. W. and Eva, K. W. (2009).The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Advances in Health Sciences Education: Theory and Practice, 14(2), 219−32. https://doi.org/10.1007/s10459-008-9104-8.PMid: 18306052.

Nair, B. R., Alexander, H. G., McGrath, B. P., Parvathy, M. S., Kilsby, E. C., Wenzel, J., Frank, I. B., Pachev, G. S. and Page, G. G. (2008). The mini clinical evaluation exercise (miniCEX) for assessing clinical performance of international medical graduates. Medical Journal of Australia, 189(3), 159−61. PMid: 18673104.

Norcini, J. J., Blank, L. L., Arnold, G. K. and Kimball, H.R. (1995). The mini-CEX (clinical evaluation exercise): a preliminary investigation. Annals of Internal Medicine, 123(10), 795−99. https://doi.org/10.7326/00034819-123-10-199511150-00008. PMid: 7574198.

Norcini J.J., Blank L.L., Arnold G.K., Kimball H.R. (1997).Examiner differences in the mini-CEX. In: Scherpbier, A. J. J. A, van der Vleuten, C. P. M, Rethans, J. J., van der Steeg, A. F. W. (Eds.), Advances in Medical Education. Dordrecht: Springer. p. 170−72. https://doi.org/10.1007/978-94-011-4886-3_49.

Norcini, J. J., Blank, L. L., Duffy, F. D. and Fortna, G. S.(2003). The mini-CEX: a method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476−81. https:// doi.org/10.7326/0003-4819-138-6-200303180-00012.PMid: 12639081.

Norcini, J. J. (2005). Current perspectives in assessment: the assessment of performance at work. Medical Education, 39(9), 880−89. https://doi.org/10.1111/j.13652929.2005.02182.x. PMid:16150027.

Norcini, J. and Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29, 855−71. https://doi.org/10.1080/01421590701775453.PMid: 18158655.

Norcini, J. (2014). Workplace assessment. In Swanick, T. (Ed.).Understanding Medical Education: Evidence, Theory and Practice (2nd ed.). London: John Wiley & Sons, Ltd. p.279-291.

Norcini, J., Anderson, M. B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., Hays, R., Palacios Mackay, M. F., Roberts, T.and Swanson, D. (2018). Consensus framework for good assessment. Medical Teacher. Published Ahead of Print. Available at https://www.tandfonline.com/doi/full/10.10 80/0142159X.2018.1500016, https://doi.org/10.1080/014 2159X.2018.1500016.

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A. and Cook, D. A. (2014). Standards for reporting qualitative research: a synthesis of recommendations. Academic Medicine, 89(9), 1245−51. https://doi.org/10.1097/ ACM.0000000000000388. PMid: 24979285.

O’Leary, D., Al-Taiar, H., Brown, N., Bajorek, T., Ghazirad, M.and Shaddel, F. (2016). Workplace assessment in crisis? The way forward. British Journal of Psychiatry Bulletin, 40(2), 61−63. https://doi.org/10.1192/pb.bp.114.050153.

Paauw, D. S., Wenrich, M. D., Curtis, J. R., Carline, J. D. and Ramsey, P. G. (1995). Ability of primary care physicians to recognize physical findings associated with HIV infection. Journal of the American Medical Association, 274(17), 1380–82. https://doi.org/10.1001/ jama.1995.03530170060033, https://doi.org/10.1001/ jama.274.17.1380. PMid: 7563564.

Papamitsiou, Z. and Economides, A. A. (2014). Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence. Journal of Educational Technology and Society, 17(4), 49−64.

Parry-Smith, W., Mahmud, A., Landau, A. and Hayes, K. (2014).Workplaceâ€based assessment: a new approach to existing tools. The Obstetrician and Gynaecologist, 16(4), 281−85.https://doi.org/10.1111/tog.12133.

Pelgrim, E. A., Kramer, A. W. M., Mokkink, H. G., Van den Elsen, L., Grol, R. P. T. M. and Van der Vleuten, C. P. M. (2011). In-training assessment using direct observation of single-patient encounters: a literature review. Advances in Health Sciences Education: Theory and Practice, 16(1), 131−42. https://doi.org/10.1007/s10459-010-9235-6. PMid: 20559868, PMCid: PMC3074070.

Pelgrim, E. A., Kramer, A. W., Mokkink, H. G. and van der Vleuten, C. P. (2012). The process of feedback in workplaceâ€based assessment: Organisation, delivery, continuity. Medical Education, 46(6), 604−12. https://doi.org/10.1111/j.1365-2923.2012.04266.x. PMid: 22626052.

Pereira, E. A. and Dean, B. J. (2009). British surgeons’ experiences of mandatory online workplace-based assessment.Journal of the Royal Society of Medicine, 102(7), 287−93. https://doi.org/10.1258/jrsm.2009.080398. PMid: 19605860, PMCid: PMC2711198.

Ponnamperuma, G. (2013). Workplace based assessment. In: Walsh, E. (Ed.). Oxford Textbook of Medical Education. 1st ed. Oxford: Oxford University Press. p. 537−48.https://doi.org/10.1093/med/9780199652679.003.0046.

Powell, H., DiMarco, A. and Saeed, S. (2014). Trainee and trainer opinion on increasing numbers of workplacebased assessments. Bulletin of the Royal College of Surgeonsof England, 96(5), 160–62. https://doi.org/10.1308/rcsbull.2014.96.5.160.

Ramani, S. and Krackov, S. K. (2012). Twelve tips for giving feedback effectively in the clinical environment. Medical Teacher, 34(10), 787−91. https://doi.org/10.3109/01421 59X.2012.684916. PMid: 22730899.

Ramsey, P. G., Wenrich, M. D., Carline, J. D., Inui, T. S., Larson, E. B. and LoGerfo. J. P. (1993). Use of peer ratings to evaluate physician performance. Journal of the American Medical Association, 269(13), 1655–60. https://doi.org/10.1001/jama.269.13.1655, https://doi.org/10.1001/ jama.1993.03500130069034. PMid: 8240483.

Rees, C. E., Knight, L. V. and Cleland, J. A. (2009). Medical educators' metaphoric talk about their assessment relationships with students: ‘you don’t want to sort of be the one who sticks the knife in them’. Assessment and Evaluation in Higher Education, 34(4), 455–67. https:// doi.org/10.1080/02602930802071098.

Rees, C. E., Cleland, J. A., Dennis, A., Kelly, N., Mattick, K. and Monrouxe, L. V. (2014). Supervised learning events in the Foundation Programme: A UK-widenarrative interview study. BMJ Open, 4(10), e005980.https://doi.org/10.1136/bmjopen-2014-005980PMid: 25324323, PMCid: PMC4202004.

Rethans, J. J., Norcini, J. J., Baronâ€Maldonado, M., Blackmore, D., Jolly, B. C., LaDuca, T., Lew, S., Page, G. G. and Southgate, L. H. (2002). The relationship between competence and performance: implications for assessing practice performance. Medical Education, 36(10), 901−09. https://doi. org/10.1046/j.1365-2923.2002.01316.x. PMid:12390456.

Ringsted, C., Østergaard, D. and Scherpbier, A. J. J. A. (2003).Embracing the new paradigm of assessment in residency training: an assessment programme for first-year residency training in anaesthesiology. Medical Teacher, 25(1), 54−62. https://doi.org/10.1080/0142159021000061431.

PMid: 14741860.

Roberts, C., Newble, D. I. and O’Rourke, A. (2002). Portfoliobased assessments in medical education: are they valid and reliable for summative purposes? Medical Education, 36(10), 899–900. https://doi.org/10.1046/j.13652923.2002.01288.x. PMid: 12390455.

Ruitz, J. G., Mintzer, M. J. and Leipzig, R. M. (2006). The impact of e-learning in medical education. Academic Medicine, 81(3), 207−12. https://doi.org/10.1097/00001888-20060300000002. PMid: 16501260.

Ryland, I., Brown, J., O’brien, M., Graham, D., Gillies, R., Chapman, T. and Shaw, N. (2006). The portfolio: how was it for you? Views of F2 doctors from the Mersey Deanery Foundation Pilot. Clinical Medicine, 6(4), 378−80. https:// doi.org/10.7861/clinmedicine.6-4-378. PMid: 16956144, PMCid: PMC4953521.

Sabey, A. and Harris, M. (2011). Training in hospitals: what do GP specialist trainees think of workplace-based assessments? Education for Primary Care, 22(2), 90−99.https://doi.org/10.1080/14739879.2011.11493974. PMid: 21439140.

Saedon, H., Salleh, S., Balakrishnan, A., Imray, C. H. and Saedon, M. (2012). The role of feedback in improving the effectiveness of workplace based assessments: A systematic review. BMC Medical Education, 12(1), 25. https://doi. org/10.1186/1472-6920-12-25. PMid: 22551353, PMCid: PMC3432628.

Sandars, J. and Haythornthwaite, C. (2007). New horizons for e-learning in medical education: ecological and Web 2.0 perspectives. Medical Teacher, 29(4), 307−10. https://doi. org/10.1080/01421590601176406. PMid: 17786742.

Sargeant, J. M., Mann, K. V., Ferrier, S. N., Langille, D. B., Muirhead, P. D., Hayes, V. M. and Sinclair, D. E. (2003). Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: a pilot study. Academic Medicine, 78(10), S42−44. https:// doi.org/10.1097/00001888-200310001-00014.

Sargeant, J., Mann, K. and Ferrier, S. (2005). Exploring family physicians’ reactions to multisource feedback: perceptions of credibility and usefulness. Medical Education, 39(5), 497−504. https://doi.org/10.1111/j.13652929.2005.02124.x. PMid: 15842684.

Scheele, F., Teunissen, P., Luijk, S.V., Heineman, E., Fluit, L., Mulder, H., Meininger, A., Wijnen-Meijer, M., Glas, G., Sluiter, H. & Hummel, T. (2008). Introducing competencybased postgraduate medical education in the Netherlands. Medical Teacher, 30(3), 248−53. https://doi. org/10.1080/01421590801993022. PMid: 18484450.

Schuwirth, L. W. and van der Vleuten, C. P. (2006). A plea for new psychometric models in educational assessment. Medical Education, 40(4), 296−300. https://doi. org/10.1111/j.1365-2929.2006.02405.x. PMid: 16573664.

Schuwirth, L. W. and van der Vleuten, C. P. (2011). Programmatic assessment: from assessment of learning to assessment for learning. Medical Teacher, 33(6), 478−85. https://doi.org/1 0.3109/0142159X.2011.565828. PMid: 21609177.

Schuwirth, L. and Ash, J. (2013). Assessing tomorrow’s learners: in competency-based education only a radically different holistic method of assessment will work. Six things we could forget. Medical Teacher, 35(7), 555−59. https://doi. org/10.3109/0142159X.2013.787140. PMid: 23641916.

Sebok-Syer, S. S., Klinger, D. A., Sherbino, J. and Chan, T. M. (2017). Mixed messages or miscommunication? Investigating the relationship between assessors’ workplace-based assessment scores and written comments. Academic Medicine, 92(12), 1774−79. https://doi. org/10.1097/ACM.0000000000001743. PMid: 28562452.

Shute, V.J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153−89. https://doi.org/10.3102/0034654307313795.

Simmons, M. (2013). How are workplace-based assessments viewed by UK psychiatry trainees? Psychiatria Danubina, 25(Suppl. 2), S182–84. PMid: 23995172.

Sin, K. and Muthu, L. (2015). Application of big data in education data mining and learning analytics – A literature review. ICTACT Journal on Soft Computing, 5(4). https:// doi.org/10.21917/ijsc.2015.0145.

Swanwick, T. and Chana, N. (2009). Workplace-based assessment.British Journal of Hospital Medicine, 70(5), 290−93.https://doi.org/10.12968/hmed.2009.70.5.42235. PMid: 19451876.

Swing, S. R. (2007). The ACGME outcome project: retrospective and prospective. Medical Teacher, 29(7), 648−54. https:// doi.org/10.1080/01421590701392903. PMid: 18236251.

Tailor, A., Dubrey, S. and Das, S. (2014). Opinions of the ePortfolio and workplace-based assessments: A survey of core medical trainees and their supervisors. Clinical Medicine, 14(4), 510–16. https://doi.org/10.7861/clinmedicine. 14-5-510. PMid: 25301912, PMCid: PMC4951960.

Ten Cate, O. (2005). Entrustability of professional activities and competencyâ€based training. MedicalEducation, 39(12), 1176−77. https://doi.org/10.1111/ j.1365-2929.2005.02341.x. PMid: 16313574.

Tochel, C., Haig, A., Hesketh, A., Cadzow, A., Beggs, K., Colthart, I. and Peacock, H. (2009). The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Medical Teacher, 31(4), 299−318. https://doi.org/10.1080/01421590902883056. PMid: 19404890.

Tsagkataki, M. and Choudhary, A. (2013). Mersey deanery ophthalmology trainees’ views of the objective assessment of surgical and technical skills (OSATS) workplace-based assessment tool. Perspectives on Medical Education, 2(1), 21–27. https://doi.org/10.1007/s40037-013-0041-8. PMid: 23670654, PMCid: PMC3576483.

Turnbull, J., MacFadyen, J., Van Barneveld, C. and Norman, G. (2000). Clinical work sampling. Journal of General Internal Medicine, 15(8), 556−61. https://doi.org/10.1046/ j.1525-1497.2000.06099.x. PMid: 10940147, PMCid: PMC1495580.UK Committee of Postgraduate Dental Deans and Directors.

(2012). Quality development of postgraduate dental training in the UK 2012. UK Committee of Postgraduate Dental Deans and Directors. Available at http://www.copdend.org/guidance/quality-development-of-postgraduatedental-training-in-the-uk-2012/.

Van der Vleuten, C. P. (1996). The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education: Theory and Practice, 1(1), 41−67. https://doi.org/10.1007/ BF00596229. PMid: 24178994.

Van der Vleuten, C. P. and Schuwirth, L. W. (2005). Assessing professional competence: from methods to programmes. Medical Education, 39(3), 309−17. https://doi. org/10.1111/j.1365-2929.2005.02094.x. PMid:15733167.

Van der Vleuten, C. P., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J. and van Tartwijk, J. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34(3), 205−14. https://doi. org/10.3109/0142159X.2012.652239. PMid: 22364452.

Van der Vleuten, C. and Verhoeven, B. (2013). Inâ€training assessment developments in postgraduate education in Europe. ANZ Journal of Surgery, 83(6), 454−59. https:// doi.org/10.1111/ans.12190. PMid: 23627525.

Van der Schaaf, M., Donkers, J., Slof, B., Moonen-van Loon, J., van Tartwijk, J., Driessen, E., Badii, A., Serban, O. and Ten Cate, O. (2017). Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics. Educational Technology Research and Development, 65(2), 359−80. https://doi.org/10.1007/ s11423-016-9496-8.

Van Tartwijk, J. and Driessen, E. W. (2009). Portfolios for assessment and learning: AMEE Guide no.

Medical Teacher, 31(9), 790−801. https://doi. org/10.1080/01421590903139201. PMid: 19811183.

Veloski, J., Boex, J. R., Grasberger, M. J., Evans, A. and Wolfson, D. B. (2006). Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Medical Teacher, 28(2), 117−28. https://doi.org/10.1080/01421590600622665. PMid: 16707292.

Violato, C., Lockyer, J. M. and Fidler, H. (2008). Changes in performance: A 5-year longitudinal study of participants in a multi-source feedback programme. Medical Education, 42(10), 1007−13. https://doi.org/10.1111/j.13652923.2008.03127.x. PMid: 18823520.

Vukanovic-Criley, J. M., Criley, S., Warde, C. M., Boker, J. R., Guevara-Matheus, L., Churchill, W. H., Nelson, W. P. and Criley, J. M. (2006). Competency in cardiac examination skills in medical students, trainees, physicians, and faculty: A multicenter study. Archives of Internal Medicine, 166(6), 610−16. https://doi.org/10.1001/archinte.166.6.610. PMid:16567598.

Watling, C., Kenyon, C. F., Schulz, V., Goldszmidt, M. A., Zibrowski, E. and Lingard, L. (2010). An exploration of faculty perspectives on the in-training evaluation of residents. Academic Medicine, 85(7), 1157–62. https://doi. org/10.1097/ACM.0b013e3181e19722. PMid: 20592512.

Watling, C., Driessen, E., van der Vleuten, C. P., Vanstone, M. and Lingard, L. (2012). Understanding responses to feedback: The potential and limitations of regulatory focus theory. Medical Education, 46(6), 593−603. https://doi. org/10.1111/j.1365-2923.2012.04209.x. PMid: 22626051.

Weller, J. M., Jolly, B., Misur, M. P., Merry, A. F., Jones, A., Crossley, J. M., Pedersen, K., Smith. K. (2009a). Miniclinical evaluation exercise in anaesthesia training. British Journal of Anaesthesia, 102(5), 633−41. https://doi. org/10.1093/bja/aep055. PMid: 19336537.

Weller, J. M., Jones, A., Merry, A.F., Jolly, B. and Saunders, D. (2009b). Investigation of trainee and specialist reactions to the mini-clinical evaluation exercise in anaesthesia: implications for implementation. British Journal of Anaesthesia, 103(4), 524−30. https://doi.org/10.1093/bja/ aep211. PMid: 19687033.

Whitehouse, A., Hassell, A., Bullock, A., Wood, L. and Wall, D. (2007). 360 degree assessment (multisource feedback) of UK trainee doctors: field testing of team assessment of behaviours (TAB). Medical Teacher, 29(2-3), 171−76. https://doi.org/10.1080/01421590701302951. PMid: 17701629.

Wilkinson, J. R., Crossley, J. G. M., Wragg, A., Mills, P., Cowan, G. and Wade, W. (2008). Implementing workplace-based assessment across the medical specialties in the United Kingdom. Medical Education, 42(4), 364–73. https://doi. org/10.1111/j.1365-2923.2008.03010.x. PMid: 18338989.

Wilson M. and Scalise K. (2015) Assessment of Learning in Digital Networks. In: Griffin P., Care, E. (Eds.) Assessment and Teaching of 21st Century Skills. Educational Assessment in an Information

Age. Springer, Dordrecht. p. 57−81. https://doi.org/10.1007/978-94-017-9395-7_3.

Wragg, A., Wade, W., Fuller, G., Cowan, G. and Mills, P. (2003). Assessing the performance of specialist registrars. Clinical Medicine, 3(2), 131−34. https://doi.org/10.7861/clinmedicine.

-2-131. PMid: 12737369, PMCid: PMC4952733.

Yeates, P., O’Neill, P., Mann, K. and Eva, K. (2013). Seeing the same thing differently. Advances in Health Sciences Education: Theory and Practice, 18(3), 325−41. https:// doi.org/10.1007/s10459-012-9372-1. PMid: 22581567.

Zendejas, B., Brydges, R., Wang, A. T. and Cook, D. A. (2013). Patient outcomes in simulation-based medical education: a systematic review. Journal of General Internal Medicine, 28(8), 1078−89. https://doi.org/10.1007/ s11606-012-2264-5. PMid: 23595919, PMCid: PMC3710391.