A Medical Record Peer-Review System to Evaluate Residents’ Clinical Competence: Criterion Validity Analysis

A Medical Record Peer-Review System to Evaluate Residents' Clinical Competence: Criterion Validity Analysis

JUNICHI KAMEOKA,^1,2 MAKOTO KIKUKAWA,³ DAIKI KOBAYASHI,⁴ TOMOYA OKUBO,⁵ SEIICHI ISHII² and YUTAKA KAGAYA²

¹Division of Hematology and Rheumatology, Faculty of Medicine, Tohoku Medical and Pharmaceutical University, Sendai, Miyagi, Japan
²Office of Medical Education, Tohoku University Graduate School of Medicine, Sendai, Miyagi, Japan
³Department of Medical Education, Faculty of Medical Sciences, Kyushu University, Fukuoka, Fukuoka, Japan
⁴Division of General Internal Medicine, Department of Internal Medicine, St. Luke's International Hospital, Tokyo, Japan
⁵Research Division, The National Center for University Entrance Examinations, Tokyo, Japan

In contrast to input evaluation (education delivered at school) and output evaluation (students' capability at graduation), the methods of outcome evaluation (performance after graduation) of medical education have not been sufficiently established. To establish a method to measure the quality of patient care and conduct outcome evaluation, we have been developing a peer review system of medical records. Here, we undertook a pilot study to evaluate the criterion validity of our system by using “evaluation by program directors (supervisors in the hospitals)” as a criterion standard. We selected 13 senior residents from three teaching hospitals. Five reviewers (general internists working in other hospitals) visited the hospitals independently and evaluated five patients' records for each resident based on the previously established sheet comprising 15 items. Independently, program directors of the senior residents evaluated their clinical performance using an evaluation sheet comprising ten items. Pearson's analysis revealed statistically significant correlation coefficients in three pairs of assessments including clinical reasoning (r = 0.5848, P = 0.0358). Bootstrap analysis revealed statistically significant correlation coefficients in additional 5 pairs including history taking (r = 0.509, 95% confidence interval: 0.034-0.847). In contrast, the correlation coefficients were low in some items: r = 0.132 (−0.393-0.639) for physical examination and r = 0.089 (−0.847-0.472) for attitude toward patients. To the best of our knowledge, this is the first study, albeit a pilot one, that investigates the criterion validity of medical record evaluations conducted by comparing the assessments of medical records with those by program directors.

keywords —— criterion validity; medical records; outcome evaluation; peer review system; physical examination

===============================

Tohoku J. Exp. Med., 2019, 248, 253-260

Correspondence: Junichi Kameoka, Division of Hematology and Rheumatology, Faculty of Medicine, Tohoku Medical and Pharmaceutical University, 1-15-1 Fukumuro, Miyagino-ku, Sendai, Miyagi 983-8536, Japan.

e-mail: j-kame@tohoku-mpu.ac.jp