Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study.

Abstract

Background: The COVID-19 pandemic has led to the popularization of information and communication technology in medical education. This study aimed to compare the objective structured clinical examination (OSCE) and the virtual objective structured clinical examination (vOSCE), based on expert opinion, as tools for assessing the competencies needed under medical curriculum guidelines in Brazil.

Methods: In this multicenter study, the suitability levels of the OSCE and vOSCE for assessing the competencies needed under the Brazilian National Curriculum Guidelines (DCNs) were compared. The DCNs encompass five groups of domains, namely, health education; general; health management; public health needs; and individual health needs. Sixteen teaching experts served as judges. They responded to a questionnaire with the provided Likert scores indicating the ability of the OSCE and vOSCE to assess each type of competency. Virtual meetings were held via the Delphi method and a focus group-based approach to enable the experts to reach a consensus.

Results: A total of 200 items included in the guidelines (100 for the OSCE and 100 for the vOSCE) were evaluated across a total of 3,200 responses. Analysis via the Likert scale revealed a high proportion of positive evaluations for the use of the vOSCE, with kappa values ​​ranging from 0.4 to 0.72. The values ​​of the absolute agreement (positive view) between examiners and equivalence with the kappa statistic for the adoption of the vOSCE ranged from 0.38 to 0.72. Via the use of virtual meetings (Delphi and focus groups), consensus was reached regarding the capacity of the OSCE and vOSCE to evaluate 97% and 92% of these items, respectively.

Conclusion: The vOSCE can be employed to assess relevant competencies. However, it faces limitations regarding 8% (8/100) of the items, and some items (3%) cannot be assessed via either method. The difference in results between the two assessment methodologies (OSCE and vOSCE) is 5%. It is necessary to invest in the creation of instruments that can be used to apply the vOSCE and to conduct cost‒benefit analysis of its broader application in health education institutions. We conclude that the vOSCE is an effective tool for assessing most competencies of medical professionals and students required under Brazilian guidelines.