Adaptation of internet-based multiple mini-interviews in a limited-resource medical school during the coronavirus disease 2019 pandemic
Article information
Abstract
The coronavirus disease 2019 pandemic has affected many aspects of life, including medical education. The selection of students using multiple mini-interviews (MMI) should be adapted promptly as internet-based MMI (iMMI) in the current setting. The current transition from MMI to iMMI is supported by experience in conducting conventional MMI in previous years; hence, the station and assessor preparation could be completed without significant issues. The greater challenges to the use of iMMI are due to technical issues, including poor internet connections and adaptations necessary for assessing candidates’ nonverbal expressions. This paper aims to describe the experience of conducting iMMI in the current pandemic situation, with further highlights on contextual challenges in a limited-resource setting.
Introduction
The coronavirus disease 2019 (COVID-19) pandemic, which began in February 2020, has necessitated adaptations in numerous aspects of life around the world, including in medical education. Medical schools should formulate and enact strategies to move learning and assessment fully online, defer other face-to-face and hands-on activities to later dates, prevent undergraduate medical students from conducting their clinical rotations (due to safety reasons), enrich the curriculum with COVID-19 subjects, and attempt to support medical and healthcare professionals working on the front line [1,2]. Amid uncertainties and challenges, especially in assuring that medical students achieve clinical competencies, medical schools must ensure that the strategies being implemented still facilitate high-quality learning and consider all stakeholders, including current and future students.
Multiple mini-interviews (MMI) have been used extensively around the world for conducting admission to various majors and levels of study [3-5]. Participants (candidates) are required to complete various activities at each station, varying from completing a certain task, conducting a role play based on a given scenario, making decisions, interacting with other participants or the interviewer, or answering questions from the interviewer as in a standard interview session [3,4,6]. In general, the reported reliability of MMI has ranged from moderate to high, with a Cronbach’s α of 0.69–0.98 and a G coefficient reaching 0.81 on generalizability analysis [4]. In the Faculty of Medicine, Universitas Indonesia, this method has been applied as part of the undergraduate international-class medical student admissions process since 2013. This method has been an integral part of the process, along with the standard admission written examination, which allows assessors to evaluate applicants’ non-cognitive attributes.
The similar concept of conducting MMI through online videoconferencing platforms was conducted at the University of Sydney in 2011, as studied by Tiller et al. [7]. No significant difference in scores equivalence was found between conventional MMI and internet-based MMI (iMMI) [7]. The reliability coefficient demonstrated by generalizability analysis also showed satisfactory results [7]. Regarding satisfaction with iMMI, both interviewers and candidates rated it positively, showing acceptability among participants. This method is also considered cost-friendly, giving institutions significant budget savings. Based on these results, iMMI is considered an important innovation that can be readily applied by other institutions [7].
In this article, we aim to share our experience in preparing and conducting our version of iMMI as part of the medical student admissions process at the Faculty of Medicine, Universitas Indonesia, during the COVID-19 pandemic. This alternative was indeed practical and necessary, especially in this pandemic situation, which requires that people minimize social interaction—conducting conventional in-person interviews is not possible. We also aim to describe challenges in conducting our version of iMMI—given limited resources—especially regarding internet bandwidth in Indonesia, compared with other countries in Asia [8].
Methods
1. Preparation
Preparations for this iMMI were made over approximately 1 week, from the decision to adopt iMMI to the day of the interviews. The preparations focused primarily on revising scenarios, preparing technical implementations, selecting and informing interviewers, and informing candidates about the change of plan.
1) Notifications
Approximately 2 days before the interview dates, notifications were sent via email and online messaging platforms to interviewers and candidates. Candidates were also contacted directly through online messaging platforms and invited to join a group for the coordination process. They were told to prepare a desktop or laptop computer with the online video-conferencing software installed and were not allowed to use phones or tablets during the iMMI. The camera angle needed to be adjusted so that the upper half of candidates could be seen on-camera during the interview. Candidates were also required to have a stable internet connection for the interview. During the application process, candidates were asked to send a photo ID with their application; this photo ID was also used to identify the candidate during the briefing and interview process. No significant problems were raised by the candidates.
Interviewers were contacted individually through online messaging platforms to obtain consent and to ensure their understanding of the iMMI process. We used an online videoconferencing platform that was quite commonly used throughout our institution during the pandemic period. To prevent confusion to interviewers and candidates, technical control of the application was largely handled by the IT team; participants only needed to join in the room and follow the instructions displayed on-screen.
2) Scenarios
Our iMMI stations consisted of four scenario stations, one writing station, and one traditional semi-structured interview station, with a total of 20 prompts—similar to our previous MMI. The stations assessed effective communication, critical thinking, ethical decision making, altruism, writing skills, and motivation. The scenarios were modified to the current COVID-19 pandemic situation, considering that face-to-face activity is not possible in this setting. Due to limited human resources, we also had to modify the effective communication station to accommodate the lack of actors. An example of the modifications made to the effective communication scenario can be seen in Table 1.
3) Stations
Each station was set up using the break-out room feature on an online videoconferencing platform, in which each break-out room consisted of the interviewer (except in the writing station), the candidate, and a co-host displaying the scenario and countdown timer for the station. The co-host station was manned by the information and technology (IT) department, which also provided technical support as needed. Each station had an interview duration of 7 minutes, with an additional 1–2 minutes for station transition, during which the candidates were moved to another break-out room/station and the interviewers scored the candidates. All interview processes were recorded; the participation of the interviewers and candidates was considered consent for the recording. The writing station was not attended by an interviewer. To ensure that the short essay written was the candidate’s own work, in addition to recording the candidate at the writing station, we developed a form for the candidates to write their short essay, with a countdown timer in which the form closes after 7 minutes. The short essays were then assessed by faculty, who also comprised the iMMI committee.
4) Scoring
The most compatible online assessment system available to our institution at the time was our objective structured clinical examination (OSCE) scoring system, in which the candidates are scored from 0 to 3 instead of 1 to 4. We modified the scoring system to the OSCE system (0: unsatisfactory, 1: satisfactory, 2: good, 3: excellent). The global rating score was also adapted to the OSCE system, which consists of “did not pass” (previously “no”), “borderline” (previously “not sure”), “pass”, and “superior” (previously “yes”).
5) Simulation
To avoid technical problems, a simulation was conducted prior to the iMMI, using three stations. The simulation was held using the online videoconferencing platform that would be used during the actual interview (Zoom). Three faculty acted as the interviewers, three alumni acted as the candidates, and the IT department served as the co-host displaying the scenarios. The simulation was conducted in real-time, in which the candidates tried to answer the scenarios and complete the form prepared for the writing station. The candidates were also moved from one station/break-out room to the next. Potential pitfalls were identified.
Results
1. Implementation
1) Briefing
Only the event coordinator and several IT department staff were present in the faculty on the interview day. With social distancing and pandemic safety protocols in place, the event coordinator and the IT department staff accessed the interview from the faculty’s computer lab, which normally can accommodate 120 computers. Thirty computers were utilized for the co-hosts at each station, displaying the scenario and countdown timer for the station. Interviewers and interviewees accessed the meeting remotely using their personal computers.
One hour before the iMMI, two briefing sessions were conducted simultaneously for both the interviewers and candidates. The briefings were held in two break-out rooms on Zoom. The briefing explained the iMMI mechanism, stations, and timing, with an emphasis on the expectations for the candidates and the scenarios for the interviewers. During the briefing, interviewers were informed of their assigned stations and were given the password to access the stations’ scenario. They were also briefed on how to access and complete the online assessment system.
All of the interviewers had experience as an MMI interviewer; thus, they were familiar with the scenarios and prompts. Training for student assessment and the OSCE was a prerequisite for teaching staff to participate in MMI interviewer training. In our previous conventional MMI process, after the briefing and before the interview, interviewers at the same station would convene to discuss the scenario and reach an agreement on scoring the prompts. Due to time constraints, this process could not be carried out. Interviewers and candidates were directly transferred to different breakout rooms on Zoom by the IT department.
2) iMMI
Fifty candidates and 36 interviewers were involved in the iMMI, divided into two sessions, with the first and second sessions consisting of six circuits and four circuits, respectively. Each circuit consisted of six candidates and five interviewers. During the interview process, aside from Zoom, the interviewers, the MMI committee, and the IT team also coordinated through a separate online messaging platform (WhatsApp) to address any technical problems that might arise. Personnel from the IT department manned a computer lab in which the computers served as the co-host in each station. The timers on the co-host computer proved highly useful in helping the interviewer and the candidate adhere to the allocated time for each station, as there were technical difficulties in moving the candidates to the next break-out room. The differences between iMMI and MMI are summarized in Table 2.
2. Challenges
The preparation stage posed only minimal difficulties, as there was sufficient time to plan and coordinate the changes to be applied to the iMMI. In terms of cost, the iMMI required less budget allocation compared to MMI due to significant cost reduction for transportation and consumption. The implementation stage, however, posed myriad problems, such as inadequate human resources, poor internet connectivity, and the faculty’s limited technological expertise.
The briefing session for the interviewers did not begin on-time, as some interviewers experienced problems in joining Zoom. Two interviewers also were suddenly unable to participate in this interview; thus, last-minute substitutions needed to be made. Due to these disruptions, the interview session was delayed for approximately 1 hour.
The most significant problem during the interview was poor internet connectivity, both for the interviewers and the candidates. No disconnection problems occurred with the interviewers, but two candidates were disconnected completely from the beginning of the interview for more than 15 minutes, causing them to miss two stations. The assessments for these two candidates were annulled and make-up sessions were conducted on a different day with a different set of scenarios.
There were also issues regarding station change. Several times, the station change took more than 2 minutes because the IT department had to move 20–30 candidates at once to the next designated break-out room. This issue delayed the entire interview process—each session lasted for approximately 90 minutes when they should have lasted no longer than 1 hour. Although the interviews lasted longer than anticipated, these delays did not significantly disrupt the interview process.
Discussion
This paper aims to elaborate the preparation and implementation of iMMI, which was conducted in response to the needs of student selection during the COVID-19 pandemic in a medical school in Indonesia. The school has been conducting face-to-face MMI for the past 6 years. iMMI was conducted in different settings, both as the original student selection method [7] and as an adaptation during the pandemic [9]. Our report supports the feasibility of iMMI implementation, despite challenges with internet connectivity and familiarity with the use of videoconferencing technology for selection purposes. The cost efficiency of iMMI also supported the feasibility, although further technical preparations and the internet bandwidth might pose additional cost.
We learned that more detailed technical preparation and a pilot study for such a high-stakes selection assessment should be conducted. This should involve more technical training for the assessors and the IT support team to prevent delays due to unanticipated technical glitches. Overall, because Indonesia has systemic challenges in internet connectivity, the decision to conduct iMMI in our setting was subject to numerous considerations, including a contingency plan to overcome technical problems. The recommended internet bandwidth for the online videoconferencing service that we used (Zoom) is 1.5 Mbps (up/down). This bandwidth is well within the capability of Indonesia’s internet connection, although stability differs widely. In the future, the recommended bandwidth information should be sent to all candidates and interviewers, along with guidelines on how to check the internet bandwidth using a speed test, in order to minimize technical problems during the interview. We realized that, despite the technical challenges, iMMI was the best possible way to conduct MMI for student selection while still ensuring implementation of COVID-19 transmission-prevention measures [9].
The station blueprint for iMMI was the same as for the “conventional” MMI that we had conducted before the COVID-19 pandemic. This ensured that the MMIs had comparable content. Several adjustments were made for iMMI in our current setting—the interviewer also became an actor performing roleplay at the communication skills station, no interviewer/supervisor was present at the writing station, and the use of Zoom for technical meetings and the iMMI itself was optimized. The adjustments implemented did not jeopardize the validity of the iMMI stations. The total number of stations and the duration of each station were within an acceptable range of best practices for MMI [4]. Some stations also utilized the COVID-19 pandemic as contextual material, which made the candidates’ experience at the iMMI stations more relevant to the current global challenges. The assessors previously conducted conventional MMI; therefore, they were well-trained in assessing MMI. However, as mentioned previously, it became apparent that the transition to iMMI required further planning to prepare the assessors to use the iMMI system [10]. In addition to the assessors’ readiness, the iMMI requires adequate preparedness by the support and IT staff, who play significant roles in ensuring the “smoothness” of iMMI implementation. While conventional MMI also requires such technical support, the support of IT staff is more crucial for iMMI. The IT staff must be able to navigate the online platform being used and to manage hardware, network, and software issues simultaneously.
To assure the reliability of assessments in similar stations across iMMI circuits, the assessors were allowed to discuss the relevant scenario, questions, and scoring to reach a similar understanding, which is key to ensuring the reliability of the MMI [4]. One challenge during iMMI was preparing the assessors and candidates to complete the virtual interview at each station. While assessors seemed able to assess the candidates’ critical thinking, reasoning, judgments, and verbal expressions, it was quite difficult to observe non-verbal expressions and the candidates’ comfort in a new environment, which was a reported challenge in conducting iMMI [9]. Nevertheless, given evidence that MMI has good fairness and does not favor candidates with certain cultural, gender, or socio-economic backgrounds [4,6], we expect that issues regarding limited observation during iMMI did not compromise fairness.
iMMI in the current setting was necessitated by the COVID-19 pandemic, which obliges social distancing and other preventive measures; conventional MMI could not be completed. The use of iMMI was feasible and supported by adequate validity and reliability. In the future, faculty should decide whether to use iMMI or the conventional MMI format. The ultimate aim of conducting MMI as a means of selecting the best candidates should be thoroughly considered. Evidently, iMMI and conventional MMI reportedly produce comparable results and save resources [7,10]. In addition, the limitations of iMMI in assessing non-verbal expressions and stress tolerance in a new environment require further adaptations [9]. The adaptation level of assessors and candidates in using virtual interviews should be assessed; this must be a consideration in using iMMI. In a limited-resource setting, especially regarding internet access and quality, technical issues due to network problems may reduce iMMI’s advantages. Therefore, whether or not iMMI becomes common practice after the pandemic depends on multiple factors. It is more important that the MMI format being used is valid, reliable, fair, and feasible.
We are fully aware that this report has several limitations. First, despite the validity of iMMI, as evidenced by the brief evaluation and feedback received from the assessors, candidates, and organizers during separate debriefings following iMMI, this report is not supported by a comprehensive satisfaction evaluation from all stakeholders. Further evaluation of iMMI should be conducted to inform future implementation in the current setting. Second, since our focus is on elaborating the preparation and implementation of iMMI as a shift from conventional MMI, we have not reported more detailed results from the selection process, which may be necessary to strengthen the validity and reliability of iMMI. Details on preparation and implementation should illuminate the face validity and feasibility of iMMI in a limited-resource setting such as that in our study. Finally, future studies on iMMI can focus on the assessment of non-verbal expression and adaptation in new environment which might require further development of iMMI stations and the use of more sophisticated technology.
In conclusion, we have described the experience of conducting iMMI in a limited-resource setting. All aspects of the preparation process—the scenario, stations, scoring system, simulation before interview day, briefing for candidates and assessors, and all technical support— should be detailed in advance to anticipate and mitigate risks during the iMMI. Because internet connections can pose a great challenge in the current setting, a contingency plan for internet connectivity problems should be discussed in advance and enacted promptly. Despite the challenges and limitations, the Faculty of Medicine at Universitas Indonesia has succeeded in conducting its first iMMI while preserving its validity and reliability. Future evaluation is necessary to reassess the format for future MMI.
Acknowledgements
We would like to extend our appreciation and gratitude to the scenario writers, interviewers, and supporting and IT staff, as well as candidates who participated in the iMMI. We would also like to thank the Dean of the Faculty of Medicine at Universitas Indonesia, 2018–2022, who provided all necessary resources to conduct iMMI during this challenging time.
Notes
Funding
No financial support was received for this study.
Conflicts of interest
No potential conflict of interest relevant to this article was reported.
Author contributions
Sophie Yolanda led the manuscript writing and prepared the iMMI details. Wismandari Wisnu led the iMMI process and contributed in the manuscript writing. James Wiguna Wahjudi contributed in the data collection and manuscript writing. Ardi Findyartini designed the concept of the manuscript and contributed in the data analysis and manuscript writing. All authors provided final approval of the version to be published.