Introduction
The coronavirus disease 2019 (COVID-19) pandemic, which began in February 2020, has necessitated adaptations in numerous aspects of life around the world, including in medical education. Medical schools should formulate and enact strategies to move learning and assessment fully online, defer other face-to-face and hands-on activities to later dates, prevent undergraduate medical students from conducting their clinical rotations (due to safety reasons), enrich the curriculum with COVID-19 subjects, and attempt to support medical and healthcare professionals working on the front line [
1,
2]. Amid uncertainties and challenges, especially in assuring that medical students achieve clinical competencies, medical schools must ensure that the strategies being implemented still facilitate high-quality learning and consider all stakeholders, including current and future students.
Multiple mini-interviews (MMI) have been used extensively around the world for conducting admission to various majors and levels of study [
3-
5]. Participants (candidates) are required to complete various activities at each station, varying from completing a certain task, conducting a role play based on a given scenario, making decisions, interacting with other participants or the interviewer, or answering questions from the interviewer as in a standard interview session [
3,
4,
6]. In general, the reported reliability of MMI has ranged from moderate to high, with a Cronbach’s α of 0.69–0.98 and a G coefficient reaching 0.81 on generalizability analysis [
4]. In the Faculty of Medicine, Universitas Indonesia, this method has been applied as part of the undergraduate international-class medical student admissions process since 2013. This method has been an integral part of the process, along with the standard admission written examination, which allows assessors to evaluate applicants’ non-cognitive attributes.
The similar concept of conducting MMI through online videoconferencing platforms was conducted at the University of Sydney in 2011, as studied by Tiller et al. [
7]. No significant difference in scores equivalence was found between conventional MMI and internet-based MMI (iMMI) [
7]. The reliability coefficient demonstrated by generalizability analysis also showed satisfactory results [
7]. Regarding satisfaction with iMMI, both interviewers and candidates rated it positively, showing acceptability among participants. This method is also considered cost-friendly, giving institutions significant budget savings. Based on these results, iMMI is considered an important innovation that can be readily applied by other institutions [
7].
In this article, we aim to share our experience in preparing and conducting our version of iMMI as part of the medical student admissions process at the Faculty of Medicine, Universitas Indonesia, during the COVID-19 pandemic. This alternative was indeed practical and necessary, especially in this pandemic situation, which requires that people minimize social interaction—conducting conventional in-person interviews is not possible. We also aim to describe challenges in conducting our version of iMMI—given limited resources—especially regarding internet bandwidth in Indonesia, compared with other countries in Asia [
8].
Results
1. Implementation
1) Briefing
Only the event coordinator and several IT department staff were present in the faculty on the interview day. With social distancing and pandemic safety protocols in place, the event coordinator and the IT department staff accessed the interview from the faculty’s computer lab, which normally can accommodate 120 computers. Thirty computers were utilized for the co-hosts at each station, displaying the scenario and countdown timer for the station. Interviewers and interviewees accessed the meeting remotely using their personal computers.
One hour before the iMMI, two briefing sessions were conducted simultaneously for both the interviewers and candidates. The briefings were held in two break-out rooms on Zoom. The briefing explained the iMMI mechanism, stations, and timing, with an emphasis on the expectations for the candidates and the scenarios for the interviewers. During the briefing, interviewers were informed of their assigned stations and were given the password to access the stations’ scenario. They were also briefed on how to access and complete the online assessment system.
All of the interviewers had experience as an MMI interviewer; thus, they were familiar with the scenarios and prompts. Training for student assessment and the OSCE was a prerequisite for teaching staff to participate in MMI interviewer training. In our previous conventional MMI process, after the briefing and before the interview, interviewers at the same station would convene to discuss the scenario and reach an agreement on scoring the prompts. Due to time constraints, this process could not be carried out. Interviewers and candidates were directly transferred to different breakout rooms on Zoom by the IT department.
2) iMMI
Fifty candidates and 36 interviewers were involved in the iMMI, divided into two sessions, with the first and second sessions consisting of six circuits and four circuits, respectively. Each circuit consisted of six candidates and five interviewers. During the interview process, aside from Zoom, the interviewers, the MMI committee, and the IT team also coordinated through a separate online messaging platform (WhatsApp) to address any technical problems that might arise. Personnel from the IT department manned a computer lab in which the computers served as the co-host in each station. The timers on the co-host computer proved highly useful in helping the interviewer and the candidate adhere to the allocated time for each station, as there were technical difficulties in moving the candidates to the next break-out room. The differences between iMMI and MMI are summarized in
Table 2.
2. Challenges
The preparation stage posed only minimal difficulties, as there was sufficient time to plan and coordinate the changes to be applied to the iMMI. In terms of cost, the iMMI required less budget allocation compared to MMI due to significant cost reduction for transportation and consumption. The implementation stage, however, posed myriad problems, such as inadequate human resources, poor internet connectivity, and the faculty’s limited technological expertise.
The briefing session for the interviewers did not begin on-time, as some interviewers experienced problems in joining Zoom. Two interviewers also were suddenly unable to participate in this interview; thus, last-minute substitutions needed to be made. Due to these disruptions, the interview session was delayed for approximately 1 hour.
The most significant problem during the interview was poor internet connectivity, both for the interviewers and the candidates. No disconnection problems occurred with the interviewers, but two candidates were disconnected completely from the beginning of the interview for more than 15 minutes, causing them to miss two stations. The assessments for these two candidates were annulled and make-up sessions were conducted on a different day with a different set of scenarios.
There were also issues regarding station change. Several times, the station change took more than 2 minutes because the IT department had to move 20–30 candidates at once to the next designated break-out room. This issue delayed the entire interview process—each session lasted for approximately 90 minutes when they should have lasted no longer than 1 hour. Although the interviews lasted longer than anticipated, these delays did not significantly disrupt the interview process.
Discussion
This paper aims to elaborate the preparation and implementation of iMMI, which was conducted in response to the needs of student selection during the COVID-19 pandemic in a medical school in Indonesia. The school has been conducting face-to-face MMI for the past 6 years. iMMI was conducted in different settings, both as the original student selection method [
7] and as an adaptation during the pandemic [
9]. Our report supports the feasibility of iMMI implementation, despite challenges with internet connectivity and familiarity with the use of videoconferencing technology for selection purposes. The cost efficiency of iMMI also supported the feasibility, although further technical preparations and the internet bandwidth might pose additional cost.
We learned that more detailed technical preparation and a pilot study for such a high-stakes selection assessment should be conducted. This should involve more technical training for the assessors and the IT support team to prevent delays due to unanticipated technical glitches. Overall, because Indonesia has systemic challenges in internet connectivity, the decision to conduct iMMI in our setting was subject to numerous considerations, including a contingency plan to overcome technical problems. The recommended internet bandwidth for the online videoconferencing service that we used (Zoom) is 1.5 Mbps (up/down). This bandwidth is well within the capability of Indonesia’s internet connection, although stability differs widely. In the future, the recommended bandwidth information should be sent to all candidates and interviewers, along with guidelines on how to check the internet bandwidth using a speed test, in order to minimize technical problems during the interview. We realized that, despite the technical challenges, iMMI was the best possible way to conduct MMI for student selection while still ensuring implementation of COVID-19 transmission-prevention measures [
9].
The station blueprint for iMMI was the same as for the “conventional” MMI that we had conducted before the COVID-19 pandemic. This ensured that the MMIs had comparable content. Several adjustments were made for iMMI in our current setting—the interviewer also became an actor performing roleplay at the communication skills station, no interviewer/supervisor was present at the writing station, and the use of Zoom for technical meetings and the iMMI itself was optimized. The adjustments implemented did not jeopardize the validity of the iMMI stations. The total number of stations and the duration of each station were within an acceptable range of best practices for MMI [
4]. Some stations also utilized the COVID-19 pandemic as contextual material, which made the candidates’ experience at the iMMI stations more relevant to the current global challenges. The assessors previously conducted conventional MMI; therefore, they were well-trained in assessing MMI. However, as mentioned previously, it became apparent that the transition to iMMI required further planning to prepare the assessors to use the iMMI system [
10]. In addition to the assessors’ readiness, the iMMI requires adequate preparedness by the support and IT staff, who play significant roles in ensuring the “smoothness” of iMMI implementation. While conventional MMI also requires such technical support, the support of IT staff is more crucial for iMMI. The IT staff must be able to navigate the online platform being used and to manage hardware, network, and software issues simultaneously.
To assure the reliability of assessments in similar stations across iMMI circuits, the assessors were allowed to discuss the relevant scenario, questions, and scoring to reach a similar understanding, which is key to ensuring the reliability of the MMI [
4]. One challenge during iMMI was preparing the assessors and candidates to complete the virtual interview at each station. While assessors seemed able to assess the candidates’ critical thinking, reasoning, judgments, and verbal expressions, it was quite difficult to observe non-verbal expressions and the candidates’ comfort in a new environment, which was a reported challenge in conducting iMMI [
9]. Nevertheless, given evidence that MMI has good fairness and does not favor candidates with certain cultural, gender, or socio-economic backgrounds [
4,
6], we expect that issues regarding limited observation during iMMI did not compromise fairness.
iMMI in the current setting was necessitated by the COVID-19 pandemic, which obliges social distancing and other preventive measures; conventional MMI could not be completed. The use of iMMI was feasible and supported by adequate validity and reliability. In the future, faculty should decide whether to use iMMI or the conventional MMI format. The ultimate aim of conducting MMI as a means of selecting the best candidates should be thoroughly considered. Evidently, iMMI and conventional MMI reportedly produce comparable results and save resources [
7,
10]. In addition, the limitations of iMMI in assessing non-verbal expressions and stress tolerance in a new environment require further adaptations [
9]. The adaptation level of assessors and candidates in using virtual interviews should be assessed; this must be a consideration in using iMMI. In a limited-resource setting, especially regarding internet access and quality, technical issues due to network problems may reduce iMMI’s advantages. Therefore, whether or not iMMI becomes common practice after the pandemic depends on multiple factors. It is more important that the MMI format being used is valid, reliable, fair, and feasible.
We are fully aware that this report has several limitations. First, despite the validity of iMMI, as evidenced by the brief evaluation and feedback received from the assessors, candidates, and organizers during separate debriefings following iMMI, this report is not supported by a comprehensive satisfaction evaluation from all stakeholders. Further evaluation of iMMI should be conducted to inform future implementation in the current setting. Second, since our focus is on elaborating the preparation and implementation of iMMI as a shift from conventional MMI, we have not reported more detailed results from the selection process, which may be necessary to strengthen the validity and reliability of iMMI. Details on preparation and implementation should illuminate the face validity and feasibility of iMMI in a limited-resource setting such as that in our study. Finally, future studies on iMMI can focus on the assessment of non-verbal expression and adaptation in new environment which might require further development of iMMI stations and the use of more sophisticated technology.
In conclusion, we have described the experience of conducting iMMI in a limited-resource setting. All aspects of the preparation process—the scenario, stations, scoring system, simulation before interview day, briefing for candidates and assessors, and all technical support— should be detailed in advance to anticipate and mitigate risks during the iMMI. Because internet connections can pose a great challenge in the current setting, a contingency plan for internet connectivity problems should be discussed in advance and enacted promptly. Despite the challenges and limitations, the Faculty of Medicine at Universitas Indonesia has succeeded in conducting its first iMMI while preserving its validity and reliability. Future evaluation is necessary to reassess the format for future MMI.