Assessing webinar outcomes for health professionals: a perspective from Indonesia during coronavirus disease 2019 pandemic
Article information
Abstract
Purpose
With the ongoing coronavirus disease 2019 (COVID-19) pandemic, health institutions and hospitals are increasingly relying on e-learning for continuing education. However, in many countries there is still limited data on the effectiveness of online learning particularly in the healthcare field. This study aims to evaluate whether webinar as a form of online educational intervention is satisfactory and effective for the continuing education of health professionals in Indonesia.
Methods
We collected participants’ demographic information including health profession, place of work, work unit, and year of graduation. There were six independent webinars included in this study. Webinar outcomes included satisfaction and learning scores. Regarding satisfaction, participants were told to complete a satisfaction survey and asked whether they would recommend the webinar to their colleagues. Regarding learning, information on their mean pre-test and post-test scores was collected.
Results
A total of 3,607 health professionals were enrolled, with the highest participation in webinars about emergency cases and COVID-19 management. The response towards satisfaction was overwhelmingly positive. In all six webinars, post-test scores were statistically significantly higher than pre-test scores. Recently graduated physicians scored higher in learning than senior physicians, while place of work and work unit did not significantly affect the scores.
Conclusion
The use of webinar for health professionals training in Indonesia was well-received amid the ongoing pandemic. In the future, health institutions and teaching hospitals should optimize the implementation of webinar training as it is associated with low cost, high flexibility, and less time commuting.
Introduction
In the era of the Fourth Industrial Revolution, educators from all sectors are required to keep up with the latest information and communication technology. Today’s health professions education, for example, cannot escape the influence of technology on its content and delivery. Medical schools, teaching hospitals, and other health institutions are seeing an increase in the use of e-learning in educating the next generation of health professionals. E-learning, also known as online learning, is referred to as the use of the Internet or other information technologies to acquire knowledge and relevant skills [1]. It can be either asynchronous or synchronous, but the main advantage that e-learning offers is flexibility for participants to adjust their own pace of learning [1]. Although it was historically viewed as a replacement for traditional face-to-face teaching, e-learning is now seen as an adjuvant to transcend the learning process and make it more effective [2]. With the ongoing coronavirus disease (COVID-19) pandemic, online learning has attracted an unprecedented level of fame and has been incorporated by many medical schools around the world in their latest curriculum.
By utilizing technology such as videoconferencing platforms, medical educators spend less time commuting to classrooms and more time tending to their work in hospitals and clinics [3]. One of the forms of e-learning commonly used in today’s continuing education for health professionals is webinar. Webinar is a form of online educational intervention supported by audiovisual software that connects participants and tutors [4]. Its synchronous nature allows people in different locations to easily exchange questions, ideas, and feedback. Previous studies by Gegenfurtner et al. [4,5] concluded that webinars are more effective than both asynchronous learning and no intervention, and they are equivalent to or slightly more effective than traditional face-to-face teaching.
Although the learning efficacy and satisfaction of webinars in other academic fields are well-documented, these measures are still understudied for webinars related to continuing education for health professionals [4-6]. Moreover, healthcare and medical education vary between countries. They are affected by country-specific factors, namely economy, technological advances, political situation, social values, and public view of the educational system [7]. To our knowledge, there has not been any study that specifically addresses webinar outcomes for health professions in Indonesia. Hence, the aim of this study is to evaluate the experience of health professionals in Indonesia after participating in webinar, in terms of their satisfaction and learning outcomes. We hope the result of this study will be able to help health institutions, especially in Indonesia, create more effective webinars for health professions education.
Methods
This is a cross-sectional study that used three set of multi-item questionnaire to evaluate the satisfaction and learning output of participants following six separate webinars conducted by Universitas Indonesia Hospital via Zoom videoconferencing platform and Youtube Live from the hospital’s channel. The webinar series, known as i-CORPS (Interactive Course for General Practitioners), was part of Universitas Indonesia Hospital’s program for continuing education for health professionals. The hospital is part of a higher education institute that also manages a leading medical school in Indonesia. The reason why the hospital conducted this webinar series was due to pandemic constraints on medical education, where there is a need to re-invent continuing medical education (CME) in a way that does not involve face-to-face teaching. Moreover, we acknowledged that many hospitals in Indonesia are facing staff shortage due to the pandemic, and this has led to a significant increase in workload for health professionals. However, this crisis must not discourage health professionals from participating in CME activity as it is part of their professional obligation. Ethical clearance was given by an Institutional Review Board which is the Ethical Committee of Universitas Indonesia Hospital (005/SKPE/KKO/2020/00). Data from participants were safeguarded from potential privacy breaches and reserved only for this research.
1. Webinar background
There were six webinars included in the i-CORPS webinar series. Each webinar was delivered in 2 consecutive days with four to five subtopics and lasted for 1 hour each. Arranged in order from the first to sixth webinar, the themes were as follow: emergency cases in daily practice, disorders of the special senses and nervous system, clinical approach towards COVID-19, medicolegal aspect of medicine and non-clinical medicine, metabolic disorders and nutrition, and emergency cases of the vital organs. Albeit having different themes, all six webinars essentially had the same intended goal which was to provide high-quality and comprehensive CME for health workers in Indonesia in spite of the ongoing pandemic. Ultimately, it was hoped that health workers in Indonesia could still gain positive experience and significant learning improvements after participating in the webinar, since CME in Indonesia before the pandemic was heavily reliant on face-to-face teaching only. Registration was done separately for each webinar through the Universitas Indonesia Hospital official website (https://rs.ui.ac.id/webinar/ ). Therefore, the webinars were independent of one another in terms of scoring and data collection. Each subject could register as participant in as many webinars as they liked.
2. Sample
Subjects enrolled were physicians, nurses, pharmacists, other healthcare workers, and medical students and had to be based in Indonesia. Only those who gave consent to the webinar terms and conditions, followed all the procedures, and completed all the questionnaires were included as the study subjects. We collected relevant personal information about the subjects’ place of work or institution, health profession, year of graduation, and work unit. Following the end of each webinar, subjects were required to complete a satisfaction questionnaire and a post-test. Post-test could be submitted multiple times (up to 30 attempts) to achieve a score of 60 or more, in order to pass the test. The passing score (≥60 out of 100) was set the same across different groups as all participants were given exactly the same material by the same speaker in a particular webinar. Moreover, it was also adjusted with the passing score for the Indonesia National Board Medical Examination (>65) as well as the number of questions in the pre-test and post-test (10 items).
3. Survey data collection
In this study, a satisfaction survey was used to assess satisfaction outcome and the combination of pre-test and post-test scores were used to evaluate learning outcomes following webinar intervention. The scores were also assessed and compared with regards to the participants’ demographic profile such as their health profession, place of work, year of graduation, and work unit. For the satisfaction survey, a 10-point Likert scale (1, very unsatisfied; 10, very satisfied) was used. The satisfaction survey evaluated participants’ satisfaction towards four aspects of the webinar: registration procedure (one item), speaker (4–5 items; one for each webinar subtopic), educational material (4–5 items; one for each webinar subtopic), and videoconferencing platform (one item). An overall mean satisfaction score for each webinar was then calculated by adding up satisfaction scores from all four aspects and dividing the total by the number of survey items. Validity and reliability of the satisfaction survey were confirmed by calculating Cronbach’s α (0.953) and item validation correlation value (0.69–0.82). Also, a single yes/no question asked whether participants would recommend this type of webinar to their colleagues in the future. For pre-test and post-test, their questions were identical in the form of four-options multiple-choice questions. Each correct answer was given a score of 10 while wrong answer was given a score of 0. Questions varied between the six webinars and were developed properly and independently by all speakers using existing literature as reference depending on the assigned topic. Each speaker is a licensed physician and certified teacher in the Universitas Indonesia teaching hospital. The test questions covered various aspects of particular diseases according to each webinar theme, including etiology, pathophysiology, clinical signs and symptoms, diagnosis, treatment, and prevention.
4. Data analysis
Data were analyzed using IBM SPSS ver. 20.0 software (IBM Corp., Armonk, USA). In the descriptive analysis, categorical variables that describe the participants and webinar characteristics were displayed in frequency (N) and proportion (%). For statistical analysis purposes, the profession variable was regrouped into physician and non-physician (medical student, nurse, pharmacist, other healthcare professional) while the work unit variable was regrouped into patient service (emergency, outpatient, inpatient) and non-patient service (managerial and others). Reaction and learning outputs were computed as means and standard deviations, unless otherwise stated. Descriptive statistics were performed to describe the distribution of satisfaction scores reported by participants across all six webinars. Paired-samples t-test and independent-samples t-test were used to test the differences in webinar average pre-test and post-test scores and participants’ post-test scores according to demographic variables, respectively. The cut-off threshold selected for testing significance was p<0.05.
Results
A total of 3,607 health professionals from all around Indonesia participated in the webinar series (Table 1). Most of the subjects were physicians (93.6%) and worked mainly in Java Island (58.1%). Almost half of them (41.6%) were working in a managerial and related position or other unspecified work units. The majority of the subjects (78.9%) were recent graduates, completing their formal education within 2015–2020.
Table 2 shows the frequency and proportion of participants in each webinar. Of all subjects, 45.6% participated in only one webinar, 18% participated in two, 11.3% participated in three, 9% participated in four, 8.2% participated in five, and 7.9% participated in all six.
Table 3 shows the distribution of overall satisfaction mean score and response about webinar recommendation among physicians and non-physicians across all six webinars. Moreover, participants were also asked whether they would recommend this kind of webinar to their colleagues in the future. The responses were overwhelmingly positive, with more than 98% of all physicians and non-physicians saying “yes”.
Table 4 shows the average pre-test and post-test score for each webinar as well as the post-test score in different demographic subgroups according to the profession. In general, the scores statistically significantly increased from pre-test to post-test in all six webinars for both profession groups. Place of work and work unit did not have any statistically significant effect on any post-test score, with few exceptions. In contrast, the year of graduation played a much bigger role in determining the learning outcome for physicians. In all six webinars, the average post-test scores of recently graduated physicians were higher than those who graduated in 2014 or prior. The differences were all statistically significant except in webinar 3.
Discussion
To our knowledge, this is the first and largest study that explores how webinar affects the satisfaction and learning experience of Indonesian health professionals. In this study, satisfaction is defined as the extent to which participants feel satisfied and find the webinar engaging, whereas learning is referred to as the degree to which participants obtain the intended knowledge from the webinar. Since all methods and data collection were performed fully online, we were able to obtain an overview of webinar outcomes from thousands of health professionals spread across Indonesia as well as make meaningful comparisons between them.
Previously, several systematic reviews already evaluated the evidence regarding the benefits of online learning for continuing education for health professionals. Some studies found that e-learning intervention was able to provide a satisfactory learning experience for health professionals and attained comparable learning outcomes to face-to-face teaching [8-10]. Its strengths lie in its ability to offer a convenient mode of learning for busy professionals, enhance collaboration by bringing together participants from different geographical areas, and provide educational material at a low cost [11]. However, some other studies adopted a more ambivalent or even disapproving attitude towards e-learning, in which they reported that e-learning had negligible effect on health professionals’ knowledge and that it was difficult to directly compare e-learning with traditional face-to-face teaching [12-14]. Since conclusive evidence is still lacking, further investigation is warranted.
As the webinar series was mainly marketed to physicians particularly general practitioners, it is not surprising that they constituted 93.6% of all subjects and most of them (78.9%) were recent graduates. The fact that Universitas Indonesia teaching hospital was located in Depok, which is a part of satellite Jakarta metropolitan area, may explain why more subjects were from Java Island than from outside Java. We initially expected that most of them worked in the patient service units as the themes of the webinar were intended to supplement daily clinical practice, but surprisingly almost half of all subjects (41.6%) worked in a managerial and related position. This position includes, but is not limited to, hospital board of directors, academician, researcher, and member of health organizations (e.g., Ministry of Health). This proportion, however, may be overestimated as we did not account for other unspecified work units.
Bodolica and Spraggon [15] mentioned the persistent gap between the macro and micro level of healthcare governance as the main cause of healthcare inefficiencies. Macro governance refers to the role of managers and health organization leaders in managing resources to boost performance and efficiency, whereas micro governance relates to patient-physician relationship. Hospital managers often overlook the quality of patient care, but this pandemic situation might have potentially urged them to seek more active involvement and refresh their clinical knowledge. Hence, high participation of macro-level stakeholders in online educational interventions such as webinar suggests that this method may help bridge the gap within healthcare governance [16,17]. Webinars hold great appeal to all levels of stakeholders as they require low cost, minimum effort, and less time commuting. Currently, India is striving towards electronic health governance (eHealth Governance) which includes rapid development of e-learning programs and interventions to facilitate health professionals training across the country. To do this, the Indian Ministries of Health, Communication and Information Technology, and Family Welfare collaborate to connect medical colleges to hospitals and one another through a telemedicine network [18]. The framework for eHealth Governance in Indonesia is also worth exploring considering the unequal health service distribution across its thousands of islands. However, this needs strong political commitment and willingness to invest in e-learning resources.
Webinars about emergency cases (webinar 1 and 6) and COVID-19 management (webinar 3) had the highest attendance (60.2%), and most participants were from patient service work units worked in the emergency department. This trend may be associated with the current pandemic and state of emergency medicine (EM) in Indonesia. A recent review by Yusvirazi et al. [19] mentioned that in 2018 there were less than 100 EM-trained physicians in Indonesia, so most emergency units are handled by general practitioners who are mostly fresh graduates from medical school. Meanwhile, the number of COVID-19 cases also remains high in the country, so all health professionals are obliged to keep up with the most recent COVID-19-related knowledge. In addition, since the pandemic has worsened staff shortage in health facilities [20], senior workers may feel the need to re-educate themselves outside their area of expertise.
The mean overall satisfaction score reported by all physicians and non-physicians ranged from 8.70 (lowest) to 9.00 (highest). These numbers concluded that participants found the webinar series highly satisfying (Table 3). Our result supports the findings from researchers in Korea University College of Medicine. Due to the pandemic, its medical education department has changed the format for all seminars to be online. In addition to a surge of increase in participation, they found that participants were able to engage better with webinars through the “chat” feature. Although certain issues such as Internet connectivity and unfamiliarity with the webinar platform may cause negative experience, it is evident from the participants’ responses that the benefits outweigh the limitations [21]. A meta-analysis on three different learning methods found that webinar offers higher enjoyment than asynchronous online teaching but lower enjoyment than face-to-face teaching [22]. Nevertheless, the differences are unlikely to be statistically significant. It is too early to tell, however, whether this observed effect is likely to be maintained when face-to-face teaching becomes available again in the future [21].
Regarding knowledge, the significant increase from pre-test to post-test score in all six webinars reflects the intervention’s success. Physicians and non-physicians responded differently in terms of which webinar helped them learn the most. For example, the largest increase in learning output for physicians was shown in webinar 6, where the difference between average post-test score and pre-test score reached 14.1 points. Meanwhile, the biggest increase for non-physicians was shown in webinar 1, where the difference reached 19.2 points. Nevertheless, the findings above suggest that both physicians and non-physicians across all six webinars were able to significantly improve their clinical knowledge following webinar intervention. This may indicate that the webinar speakers were able to successfully adjust the difficulty level of the educational materials, allowing participants to grasp the knowledge easily despite coming from different backgrounds.
The pre-test and post-test questions were designed to be at general practitioner level knowledge, so it may be obvious that recent graduates achieved high post-test scores as they could still easily recall the knowledge from their time in medical schools. The main concern is why senior physicians were not able to score as highly as their junior counterparts. Like in other countries, physicians in Indonesia are obliged to participate in CME for re-registration purposes and lifelong learning [23]. Very limited study has been conducted to evaluate the current condition of CME in Indonesia, but studies in Pakistan and Australia found that physicians in Western countries are much more likely to attend CME than physicians in developing countries. Among those who attend, physicians prefer face-to-face activities to online-based activities. A high number of senior physicians have positive attitude towards CME but do not participate. Reasons vary, but the most common ones are lack of information about the benefits of CME, lack of time, and clash with duty hours [24,25].
The success of i-CORPS webinar series can hopefully serve as a reference for the implementation of future webinars or other online educational interventions. We believe that four main aspects contributed to the webinar success: (1) the webinar series covered a diverse range of medical topics and diseases commonly encountered by health professionals in daily practice; (2) rather than attempting to discuss all topics in one exhausting session, we separated the topics into six webinars that were conducted consecutively every week for 2 days each. This allowed the participants enough time to absorb as much knowledge as possible in a consistent manner; (3) all webinars were recorded, and the recordings were uploaded to a central server where participants can access it again any time they want. This combination of synchronous and asynchronous methods offers flexibility and convenience; and (4) using Zoom videoconferencing platform, the webinar was designed to be highly interactive by utilizing the “chat” and “polling” features. This enabled a two-way communication between participants and speakers that almost resembled face-to-face interaction.
This study is not without limitations. Most of the webinar participants were physicians, so the observed effect of webinar on satisfaction and learning outcomes may not be as pronounced in other health professionals as in physicians. Lastly, since the webinar required good Internet connection, potential participants from rural areas might not be included in this study. Further investigation is also needed to evaluate more accurately the satisfaction and effectiveness of webinar as an online educational intervention for other health professionals. Future research is also needed to study how this webinar series can ultimately impact the participants’ daily practice.
In conclusion, webinar as a form of online educational intervention is a reliable alternative for health professionals training during this difficult time. Considering its positive association with satisfaction and learning, webinar should be integrated with traditional, face-to-face teaching rather than being discarded in the future. However, several limitations of webinar must be addressed, including connection problems, lack of interest in online CME, server overload, and other technical problems. Hospitals and other health facilities are encouraged to organize more webinars and other online learning activities to boost their staff performance without interfering too much with their duty hours.
Acknowledgements
We would like to extend our appreciation and gratitude to all speakers of our webinar as well as all supporting IT staff.
Notes
Funding
No financial support was received for this study.
Conflicts of interest
No potential conflict of interest relevant to this article was reported.
Author contributions
ECY analyzed the data collected, performed literature search, and drafted the manuscript. ANW, DYF, RAW, and DP helped coordinate with the webinar IT staff, assisted in data analysis, reviewed the manuscript, and suggested changes and additions. All authors provided final approval of this version of manuscript to be published.