The application of augmented reality for improving clinical skills: a scoping review

Article information

Korean J Med Educ. 2024;36(1):65-79

Publication date (electronic) : 2024 February 28

doi : https://doi.org/10.3946/kjme.2024.285

Mahmud Mahmud^,¹

, Dwi Cahyani Ratna Sari ²

, Djayanti Sari ¹

, Nur Arfian ²

, Muhammad Ary Zucha ³

¹Department of Anesthesiology & Intensive Care Therapy, Sardjito General Hospital, Faculty of Medicine, Public Health and Nursing Universitas Gadjah Mada, Yogyakarta, Indonesia

²Department of Anatomy, Faculty of Medicine, Public Health and Nursing Universitas Gadjah Mada, Yogyakarta, Indonesia

³Department of Obstetrics and Gynecology, Sardjito General Hospital, Faculty of Medicine, Public Health and Nursing Universitas Gadjah Mada, Yogyakarta, Indonesia

Corresponding Author: Mahmud Mahmud (https://orcid.org/0000-0002-7224-2637) Department of Anesthesiology & Intensive Care Therapy, Sardjito General Hospital, Faculty of Medicine, Public Health and Nursing Universitas Gadjah Mada, Jl Kesehatan No.1 Sekip, Yogyakarta, Indonesia Tel: +62.274.514495 Fax: +62.274.514495 email: mahmudanestesi@ugm.ac.id

Received 2023 July 26; Revised 2023 October 18; Accepted 2023 December 11.

Abstract

Augmented reality technology had developed rapidly in recent years and had been applied in many fields, including medical education. Augmented reality had potential to improve students’ knowledge and skills in medical education. This scoping review primarily aims to further elaborate the current studies on the implementation of augmented reality in advancing clinical skills. This study was conducted by utilizing electronic databases such as PubMed, Embase, and Web of Science in June 2022 for articles focusing on the use of augmented reality for improving clinical skills. The Rayyan website was used to screen the articles that met the inclusion criteria, which was the application of augmented reality as a learning method in medical education. Total of 37 articles met the inclusion criteria. These publications suggested that using augmented reality could improve clinical skills. The most researched topics explored were laparoscopic surgery skills and ophthalmology were the most studied topic. The research methods applied in the articles fall into two main categories: randomized control trial (RCT) (29.3%) and non-RCT (70.3%). Augmented reality has the potential to be integrated in medical education, particularly to boost clinical studies. Due to limited databases, however, any further studies on the implementation of augmented reality as a method to enhance skills in medical education need to be conducted.

Keywords: Augmented reality; Clinical competence; Skills; Medical education; Mobile application

Introduction

Medical education consists of various methods, primarily on theoretical learning in classrooms and clinical skills in hospitals, where students gain clinical experiences. These learning methods mostly employ the traditional method. Traditional method involves having students come to class multiple times a week. The instructor of the class will often concentrate on instructing, assigning homework, and completing assessments like quizzes, examinations, or projects. The most popular method of instruction is the use of aural and visual aids or media, such as films, demonstrations, or case studies [1].

Rapid innovations and advancement in medical education technology, such as simulation technologies and open-access medical education have occurred recently. The application of augmented reality (AR) as one of simulation technologies in medical education is currently gaining popularity because of its potential to improve the learning qualities [2]. AR can improve the quality of educational settings by providing a life-like experience and interaction of the physical world through the addition of computer-generated sensory input such as sound, video, graphics, or GPS data [2,3]. Previous studies have demonstrated the implementation of AR in multiple settings including medication management and planning, learning tools in medical education and training, simulating objects or organs in surgery for better visualization, improving techniques in rehabilitation for improved outcomes, and many more. As a result, AR system applications will continue to advance and have potential use to facilitate the process of education in healthcare and medical education [2].

As a result of the use of digital strategies, modern healthcare professional education has undergone a considerable paradigm shift which is now represented in many curricula. Digital learning methods such as using software, videos, gaming, online classes, augmented and virtual reality has proven to have many advantages to improve students’ knowledge and skills. Digital methods are not limited by time or space, and they are flexible, allowing students to learn and practice repeatedly anywhere and anytime. Meanwhile, the disadvantages of this strategy involve higher costs for preparation and platform maintenance, the dependence on the device such as computer or smartphone and Internet connection and the lack of instructors’ presence [1]. The advantages and disadvantages of digital and traditional methods are detailed in Table 1.

Table 1.

The Advantages and Disadvantages of Digital and Traditional Learning Method [1]

Students’ self-confidence is the key factor that influences their success in demonstrating the acquisition of skills. Students want to feel safe during their practices, both for themselves and the patients. Digital learning, for example, AR could be an alternative to improve knowledge and skills in medical students and staffs by providing simulation for trainings [3]. This scoping review mainly purposes to describe the existing research on the use of AR for improving clinical skills.

Methods

1. Information sources

This study conducted by having a scoping review in which the researchers select relevant articles through medical journals and databases such as PubMed, Embase, and Web of Science. A scoping review examines an exploratory research issue and aims to identify important ideas (components), forms of evidence, and research gaps through knowledge synthesis. The titles and abstracts of the papers were examined and analyzed for relevance to the use of AR for enhancing clinical skills using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

2. Eligibility and inclusion criteria

This research involved all articles with the population or participants who were provided intervention using AR technology to compare the technology with the existing traditional methods to improve their clinical skills. This study used randomized control trial and experimental studies which compared two methods of study using AR and the traditional methods.

1) Exclusion criteria

Non-English language articles; discussion, book chapters, proceedings, and dissertations/thesis were excluded.

2) Databases

From the inception of the research to June 2022, the researchers had searched three databases; PubMed, Embase, and Web of Science.

3) Search strategy

This research employed a combination of Medical Subject Headings (MeSH) and free-text search terms for AR and skills as described in Table 2.

Table 2.

Search Strategy

This study used Rayyan Intelligent Systematic Review (https://www.rayyan.ai/), a free online application that assisted systematic review methodology and meta-analysis projects, to filter the articles that met the inclusion criteria. This application allows user to search the existing citation records such as title, abstract, and author.

3. Data extraction and analysis

The extracted data obtained from various literatures were analyzed using both qualitative and descriptive method. The researchers retrieved descriptive data concerning year of publication, type of artificial intelligence used in the study, number of subjects, outcome of study, and parameter/assessment tool as well as took note on the results of the studies. In the analysis stage, all data were meticulously analyzed using Braun and Clarke’s qualitative thematic analysis, a widely used method in scoping reviews. In addition, this review applied a comprehensive thematic representation of certain research areas, and artificial intelligence application to improve skills of medical students were carefully selected. Familiarization of data set included type of study, type of AR, and outcome/parameter of assessment. The authors discussed theme review, data, and collation and excluded articles with had little engagement with the key issue (AR and skill improvement) and those that mentioned the bias issues.

Results

1. Synthesis of results

The initial search yielded a total of 4,258 entries. After the process of filtering, eliminating duplicates, screening, and excluding them based on titles and abstracts, 85 publications were reviewed in the study. A total of 37 articles were then selected for the full-review study. The PRISMA flow diagram of the study selection is presented in Fig. 1. This research focused on several countries: the United States (n=10), Canada (n=4), the United Kingdom (n=4), Germany (n=4), Switzerland (n=4), Netherlands (n=3), Serbia (n=2), Austria (n=2), Japan (n=2), Portugal (n=1), and Thailand (n=1). All study’s characteristics included in this study were described in Table 3 [4-40].

Fig. 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses Diagram of Study Selection

Table 3.

Summary of the Articles for Scoping Review

2. Randomized control trial articles

The previous studies used randomized control trials (RCTs) to investigate the application of AR. Eleven publications (29,7%) of 37 publications reviewed in this study employed RCT as the research method. These publications originated from a variety of medical science fields, the majority of which dealt with surgery. Ten articles investigated skill enhancement after adopting AR as a novel learning technique. By giving participants a new method of mentoring or teaching, AR system was applied to improve skills in cricothyroidotomy [4], transthoracic echocardiogram [5], and laparoscopic suturing task [6], Robotic system, simulator, and instructions using AR were also used in urethrovesical anastomosis [7], ophthalmoscopy and ophthalmic microsurgery [8-10], local anesthesia of inferior alveolar nerve block [11], dynamic hip screw guidewire [12], and single extracorporeal membrane oxygenation (ECMO) cannulation [13]. One study used AR with flashcards to improve emergency ultrasonography knowledge [14]. Ten articles found that the AR intervention group outperformed the traditional approach control group.

3. Non-randomized control trial articles

There were 26 articles (70,3%) of 37 articles examined in this study employed non RCT study methods. Besides, 11 articles concentrated on skill improvement, and all of the studies focused on improvement. AR is essential to help medical practitioners do laparoscopic suturing, knot-tying training, and better visualization for digital rectal examination. AR has been used in tele-mentoring to assist students in achieving better skills in port placement and abdominal incision. The implementation of AR to provide learning assisting is considered advantageous for acetabular cup orientation in total hip arthroplasty. The growing popularity of AR in the medical field aids numerous departments in developing their skills and simplifying their work. The non-RCT articled reviewed stated that Immersive Touch simulator can increase success rate of ventriculostomy catheter placement [15]. OsiriX assists surgeons in navigating patients’ anatomy [16]. The improvement of clinical skills is possible using intelligent tutoring system for micromanipulation in ophthalmic surgery [17]. Despite requiring improvement, ODG (Osterhout Design Group) R-7 Smartglasses (https://www.osterhoutgroup.com/products-r7HL/) is used in temporal bone surgery experience [18], Meanwhile, other 15 articles had investigated the application of AR in skill improvement, with an emphasis on the system’s usability or feasibility.

4. Augmented reality in laparoscopic surgery skills

Minimally invasive surgery technique such as using laparoscopy for surgery has become widespread in various areas of surgery. This technique has numerous benefits including a lower risk of infection, less surgical stress and complications, and a shorter hospital stay and recovery time. The application of AR is able to facilitate the process of learning and training of laparoscopic surgery skills. The majority of articles in this review investigated the utilization of AR to enhance laparoscopic surgical skills. One study examined how instructional video scheduling has affected learning curves for laparoscopic skill training. The participants first received a course on the basic skills on virtual reality simulator before divided into two groups of instructional videos either preoperatively or intraoperatively. Both groups improved their laparoscopic skills, with no significant differences [19]. Another study looked on the use of optical see-through AR to view an interactive virtual pointer on the laparoscopy. The participants would be separated into two groups: HoloPointer and standard. The results revealed that the HoloPointer group outperformed the regular group. The HoloPointer group also exhibited a reduction in mistakes and increased movement economy [20]. The ProMIS Augmented Reality laparoscopic simulator was investigated to improve laparoscopic suturing abilities and found a substantial difference in the participants’ performance curves and metrics [21,22]. Nineteen participants completed a 6-week training program utilizing a virtual reality simulator (LapSim) and were evaluated using an AR simulator (ProMIS) for cholecystectomy performance, demonstrating improvement in surgical skills [23]. The use of AR telementoring for laparoscopic surgery intracorporeal suturing task shown faster skill development and fewer failures than standard learning methods [6].

5. Augmented reality in ophthalmology

In order to establish a diagnosis, indirect ophthalmoscopy was a necessary examination in ophthalmology. Training for these skills requires trial and error, and numerous approaches, including fundus dummies, have been devised. Augmented reality ophthalmoscopy (ARO) was created to assist students in learning and practicing binocular indirect ophthalmoscopy. The ARO group improved their ophthalmoscopy skills faster than the traditional group [9]. EyeSI, which was comparable to ARO, an AR binocular indirect ophthalmoscopy simulator developed and investigated in Canada to be contrasted with traditional teaching methods, was developed and studied. The EyeSI group outperformed the typical group and improved significantly [8]. Students are trained for skill improvement in microsurgical operations using AR. One study looked into the use of AR to improve motor abilities and intelligent tutoring systems to improve training efficiency. The results demonstrated that adopting AR as a training tool improved skill. Students’ performance improved with both intelligent tutoring system and natural sequential steps or classic training [10,17].

Discussion

1. Principal findings

AR is a novel technology that has a variety of applications that are constantly being developed. AR is a technology that combines two-dimensional or three-dimensional virtual objects into a three-dimensional real environment, then projects these virtual objects in real time. One of the elements contributing to its growing popularity is the ability to observe and interact with digital objects without having to turn away from the real world to view the monitor displaying the appropriate medical imaging [41]. The difference between AR and virtual reality are AR using real-world setting and enhances both virtual and real world, users can control their presence in real world and AR can be accessed with smartphone while virtual reality is completely virtual, controlled by the system and required headset device. This scoping analysis was carried out to describe the existing research on the use of AR in improving medical practitioners’ clinical abilities. This scoping review discovered 37 papers that met the criteria. More than 80% of the publications in this review stated that using AR could improve clinical skills. AR is able to outperform the conventional methods of learning by providing more improvement of students’ knowledge and skills.

A comparison of the effects of mobile AR learning and textbook learning on medical students revealed that the mobile AR group improved their knowledge more than the control group [24]. These findings were also seen in the publications we examined for this investigation. They discovered that AR improved and increased the effectiveness of learning.

Because of the disparities in what medical students learned in school and in clinical settings, it was discovered that surgery and interventional procedures received more attention in the development of new digital learning tools. Students’ knowledge and abilities will not improve until they practice them. Chowriappa et al. [7] used an RCT to compare the use of AR, namely hand-on surgical training, and traditional approaches in skills training for robot-assisted urethrovesical anastomosis. The results demonstrated that the AR training increased students’ skills [7]. Other studies focusing on skill improvement in surgery and the interventional procedure category were computed tomography (CT)-guided intervention [25], port placement and abdomen incision [26], basic and laparoscopic skills and suturing [16,22,27], cholecystectomy [20,23], total hip arthroplasty [28], percutaneous renal access and nephrolithotomy [29,30], percutaneous rhizotomy [31], ECMO cannulation, and transthoracic echocardiogram [5,13]. It is found that AR is used to improve language learning, mechanical skills, and spatial abilities in addition to clinical skills.

Each AR application or device has its unique user interface and benefits that may or may not be applicable in all circumstances. The standard approaches evaluated still increase clinical skill learning, but the AR groups demonstrated greater improvement. Previous research has demonstrated the advantages of using AR in medical education, such as minimizing radiation exposure. Park et al. [42] investigated how the use of AR could increase procedural efficiency in terms of process time, puncture, and radiation exposure for CT-guided intervention. Park et al. [42] discovered that utilizing AR in CT-guided intervention reduced the mean CT dose index from 28.7 to 16.9 mGy.

AR also shows its effectiveness in improving surgical skills through telementoring, which offers experts in faraway locations real-time guidance on surgical operations. This can be a potential to use AR in the future as a tool for supervision and skill development. For surgical novices, expert supervision is crucial, and AR telementoring may provide more flexibility and efficiency than traditional “direct” supervision, which in certain cases might be demanding and ineffective. Also, considering that objective performance indicators may be monitored utilizing AR platforms in a simulated or, ideally, real-life operational environment to assess technical ability, the function of AR as an evaluation tool is worth taking into account [43,44].

AR applications are most successful in motivating users for training and knowledge transmission. It serves as a tool to provide interactive experiences and makes users feel comfortable with decisions and supported while doing skills or activities. However, technological concern such as latencies, eyesight, perception, a lack of ergonomic gear, location and accuracy failures, illumination occlusions, and other technical issues remain as the disadvantages of this digital method of learning and needed to be solved [45]. As this technology develops, further research will be required to assess various modalities that AR may be applied to. To be effective, future AR devices would also need to be able to interface and link up with other medical equipment, such emergency medical record systems, ultrasound machines, and vital monitors. It will be necessary to create software standards for this [2].

2. Strength and limitations

This review may provide insight into the potential application of AR as a learning tool in medical education and training. It also could inform the application of AR across various areas or departments in medical fields. The majority of articles on the development of AR technology highlight industrialized countries and significant medical education centers. AR is still considered a novel and expensive technology in medical sectors. Another limitation is that the included studies only discussed small-scale AR implementations and used a variety of AR platforms. For further research on the use of AR to improve students' skills, the researchers advise using and applying AR by using an application (mobile or PC) or online that provides AR simulation, if access and expense are limited.

Conclusion

AR has the potential to be integrated in medical education, particularly to boost clinical studies. Due to limited databases, however, any further studies on the implementation of AR as a method to enhance skills in medical education need to be conducted.

Acknowledgements

None.

Notes

Funding

None.

Conflicts of interest

No potential conflict of interestrelevant to this article was reported.

Author contributions

Conception or design of the work: MM, DCR, DS, NA; data collection: MM; data analysis: MM, DCR, DS, NA; data interpretation: MM, DCR, DS, NA, MAZ; drafting the article: MM; critical revision of the article: MM, DCR, DS, NA, MAZ; and final approval of the version to be published: DCR, DS, NA.

References

1. Tran B. Educational experiences with traditional, online and hybrid learning environments. J Educ Soc Policy 2016;3(3):60–63.

2. Munzer BW, Khan MM, Shipman B, Mahajan P. Augmented reality in emergency medicine: a scoping review. J Med Internet Res 2019;21(4)e12368.

3. Kurt Y, Öztürk H. The effect of mobile augmented reality application developed for injections on the knowledge and skill levels of nursing students: an experimental controlled study. Nurse Educ Today 2021;103:104955.

4. Andersen DS, Cabrera ME, Rojas-Muñoz EJ, et al. Augmented reality future step visualization for robust surgical telementoring. Simul Healthc 2019;14(1):59–66.

5. Cecilio-Fernandes D, Cnossen F, Coster J, Jaarsma AD, Tio RA. The effects of expert and augmented feedback on learning a complex medical skill. Percept Mot Skills 2020;127(4):766–784.

6. Vera AM, Russo M, Mohsin A, Tsuda S. Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology. Surg Endosc 2014;28(12):3467–3472.

7. Chowriappa A, Raza SJ, Fazili A, et al. Augmentedreality-based skills training for robot-assisted urethrovesical anastomosis: a multi-institutional randomised controlled trial. BJU Int 2015;115(2):336–345.

8. Rai AS, Rai AS, Mavrikakis E, Lam WC. Teaching binocular indirect ophthalmoscopy to novice residents using an augmented reality simulator. Can J Ophthalmol 2017;52(5):430–434.

9. Leitritz MA, Ziemssen F, Suesskind D, et al. Critical evaluation of the usability of augmented reality ophthalmoscopy for the training of inexperienced examiners. Retina 2014;34(4):785–791.

10. Menozzi M, Ropelat S, Köfler J, Huang YY. Development of ophthalmic microsurgery training in augmented reality. Klin Monbl Augenheilkd 2020;237(4):388–391.

11. Mladenovic R, Pereira LA, Mladenovic K, Videnovic N, Bukumiric Z, Mladenovic J. Effectiveness of augmented reality mobile simulator in teaching local anesthesia of inferior alveolar nerve block. J Dent Educ 2019;83(4):423–428.

12. Sugand K, Wescott RA, Carrington R, Hart A, van Duren BH. Training and transfer effect of FluoroSim, an augmented reality fluoroscopic simulator for dynamic hip screw guidewire insertion: a single-blinded randomized controlled trial. J Bone Joint Surg Am 2019;101(17)e88.

13. Wolf J, Wolfer V, Halbe M, Maisano F, Lohmeyer Q, Meboldt M. Comparing the effectiveness of augmented reality-based and conventional instructions during single ECMO cannulation training. Int J Comput Assist Radiol Surg 2021;16(7):1171–1180.

14. Ienghong K, Kotruchin P, Tangpaisarn T, Apiratwarakul K. Practical emergency ultrasound flashcards with augmented reality in teaching point-of-care ultrasound in ER. Open Access Maced J Med Sci 2021;9(E):39–42.

15. Yudkowsky R, Luciano C, Banerjee P, et al. Practice on an augmented reality/haptic simulator and library of virtual brains improves residents’ ability to perform a ventriculostomy. Simul Healthc 2013;8(1):25–31.

16. Volonté F, Pugin F, Bucher P, Sugimoto M, Ratib O, Morel P. Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion. J Hepatobiliary Pancreat Sci 2011;18(4):506–509.

17. Ropelato S, Menozzi M, Michel D, Siegrist M. Augmented reality microsurgery: a tool for training micromanipulations in ophthalmic surgery using augmented reality. Simul Healthc 2020;15(2):122–127.

18. Yong M, Pauwels J, Kozak FK, Chadha NK. Application of augmented reality to surgical practice: a pilot study using the ODG R7 Smartglasses. Clin Otolaryngol 2020;45(1):130–134.

19. Broekema TH, Talsma AK, Wevers KP, Pierie JE. Laparoscopy instructional videos: the effect of preoperative compared with intraoperative use on learning curves. J Surg Educ 2017;74(1):91–99.

20. Heinrich F, Huettl F, Schmidt G, et al. HoloPointer: a virtual augmented reality pointer for laparoscopic surgery training. Int J Comput Assist Radiol Surg 2021;16(1):161–168.

21. Botden SM, de Hingh IH, Jakimowicz JJ. Suturing training in augmented reality: gaining proficiency in suturing skills faster. Surg Endosc 2009;23(9):2131–2137.

22. Oostema JA, Abdel MP, Gould JC. Time-efficient laparoscopic skills assessment using an augmented-reality simulator. Surg Endosc 2008;22(12):2621–2624.

23. Nomura T, Mamada Y, Nakamura Y, et al. Laparoscopic skill improvement after virtual reality simulator training in medical students as assessed by augmented reality simulator. Asian J Endosc Surg 2015;8(4):408–412.

24. Albrecht UV, Folta-Schoofs K, Behrends M, von Jan U. Effects of mobile augmented reality learning compared to textbook learning on medical students: randomized controlled pilot study. J Med Internet Res 2013;15(8)e182.

25. Amiras D, Hurkxkens TJ, Figueroa D, et al. Augmented reality simulator for CT-guided interventions. Eur Radiol 2021;31(12):8897–8902.

26. Andersen D, Popescu V, Cabrera ME, et al. Medical telementoring using an augmented reality transparent display. Surgery 2016;159(6):1646–1653.

27. Nagayo Y, Saito T, Oyama H. A novel suture training system for open surgery replicating procedures performed by experts using augmented reality. J Med Syst 2021;45(5):60.

28. Logishetty K, Western L, Morgan R, Iranpour F, Cobb JP, Auvinet E. Can an augmented reality headset improve accuracy of acetabular cup orientation in simulated THA?: a randomized trial. Clin Orthop Relat Res 2019;477(5):1190–1199.

29. Mu Y, Hocking D, Wang ZT, Garvin GJ, Eagleson R, Peters TM. Augmented reality simulator for ultrasound-guided percutaneous renal access. Int J Comput Assist Radiol Surg 2020;15(5):749–757.

30. Müller M, Rassweiler MC, Klein J, et al. Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J Comput Assist Radiol Surg 2013;8(4):663–675.

31. Shakur SF, Luciano CJ, Kania P, et al. Usefulness of a virtual reality percutaneous trigeminal rhizotomy simulator in neurosurgical training. Neurosurgery 2015;11 Suppl 3:420–425.

32. Glossop N, Wang Z, Wedlake C, Moore J, Peters T. Augmented reality laser projection device for surgery. Stud Health Technol Inform 2004;98:104–110.

33. Kollmann C, Santner L. A prototype of a handheld augmented reality application to get familiar with ultrasound equipment knobology. Med Ultrason 2021;23(4):438–442.

34. Kratzke IM, Goss RS, Razzaque S, et al. Navigation improves tumor ablation performance: results from a novel liver tumor simulator study. Am Surg 2023;89(5):1693–1700.

35. Mendes HC, Costa CI, da Silva NA, Leite FP, Esteves A, Lopes DS. PIÑATA: pinpoint insertion of intravenous needles via augmented reality training assistance. Comput Med Imaging Graph 2020;82:101731.

36. Mladenovic R, AlQahtani S, Mladenovic K, Bukumiric Z, Zafar S. Effectiveness of technology-enhanced teaching methods of undergraduate dental skills for local anaesthesia administration during COVID-19 era: students’ perception. BMC Oral Health 2022;22(1):40.

37. Muangpoon T, Haghighi Osgouei R, Escobar-Castillejos D, Kontovounisios C, Bello F. Augmented reality system for digital rectal examination training and assessment: system validation. J Med Internet Res 2020;22(8)e18637.

38. Rochlen LR, Levine R, Tait AR. First-person point-of-view-augmented reality for central line insertion training: a usability and feasibility study. Simul Healthc 2017;12(1):57–62.

39. Wagner A, Millesi W, Watzinger F, et al. Clinical experience with interactive teleconsultation and teleassistance in craniomaxillofacial surgical procedures. J Oral Maxillofac Surg 1999;57(12):1413–1418.

40. Zhao S, Xiao X, Wang Q, et al. An intelligent augmented reality training framework for neonatal endotracheal intubation. Int Symp Mix Augment Real 2020;2020:672–681.

41. Barcali E, Iadanza E, Manetti L, Francia P, Nardi C, Bocchi L. Augmented reality in surgery: a scoping review. Appl Sci 2022;12(14):6890.

42. Park BJ, Hunt SJ, Nadolski GJ, Gade TP. Augmented reality improves procedural efficiency and reduces radiation dose for CT-guided lesion targeting: a phantom study using HoloLens 2. Sci Rep 2020;10(1):18620.

43. Kovoor JG, Gupta AK, Gladman MA. Validity and effectiveness of augmented reality in surgical education: a systematic review. Surgery 2021;170(1):88–98.

44. El-Sabawi B, Magee W 3rd. The evolution of surgical telementoring: current applications and future directions. Ann Transl Med 2016;4(20):391.

45. Villagran-Vizcarra DC, Luviano-Cruz D, Pérez-Domínguez LA, Méndez-González LC, Garcia-Luna F. Applications analyses, challenges and development of augmented reality in education, industry, marketing, medicine, and entertainment. Appl Sci 2023;13(5):2766.

Article information Continued

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Fig. 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses Diagram of Study Selection

Table 1.

The Advantages and Disadvantages of Digital and Traditional Learning Method [1]

Method	Advantages	Disadvantages
Digital (mLearning)	- Unlimited time and place	- Requires higher material preparation and platform maintenance costs
	- Flexible so students can move at their own learning pace	- Lack of presence of instructors or tutors
	- Students can study, repeat material and practice many times outside class hours	- Depends on computer or smartphone, internet connection, learning system
		- It can give rise to distrust or cheating by students by asking other people to replace their presence during class or exams
Traditional	- Direct and face-to-face interaction between students and teachers	- Implementation depends on the presence of students and teachers at a specific place and time
	- Students and teachers can immediately get feedback from the implementation of learning
	- Teachers and students have a better connection	- The speed of learning or the material delivered is determined by the teacher (some students may feel the material is delivered too quickly, but there are also students who feel the material is delivered too slowly)
	- Teachers can assess students who actively participate and understand the material or students who are not interested or who do not understand the material

Table 2.

Search Strategy

	Search strategy
Embase	‘medical student’/exp OR ‘resident’/exp OR ‘consultation’/exp OR ‘postgraduate student’/exp OR ‘medical student’:ab,ti OR ‘fellow’:ab,ti OR ‘residen’:ab,ti OR ‘trainee’:ab,ti OR ‘consultan’:ab,ti OR ‘postgraduate’:ab,ti AND ‘augmented reality’/exp OR ‘augmented realit’:ab,ti OR image:ab,ti AND ‘skill’/exp OR skill:ab,ti OR proficienc:ab,ti
Web of Science	medical student OR medical students OR fellow OR fellowship OR fellowships OR resident OR residents OR trainee OR trainees OR consultant OR consultants OR postgraduate OR postgraduates AND augmented reality OR image OR augmented realities OR images AND skill OR skills OR proficiency OR proficiencies

Search strategy

Embase

‘medical student’/exp OR ‘resident’/exp OR ‘consultation’/exp OR ‘postgraduate student’/exp OR ‘medical student*’:ab,ti OR ‘fellow*’:ab,ti OR ‘residen*’:ab,ti OR ‘trainee*’:ab,ti OR ‘consultan*’:ab,ti OR ‘postgraduate*’:ab,ti AND ‘augmented reality’/exp OR ‘augmented realit*’:ab,ti OR image*:ab,ti AND ‘skill’/exp OR skill*:ab,ti OR proficienc*:ab,ti

Web of Science

medical student OR medical students OR fellow OR fellowship OR fellowships OR resident OR residents OR trainee OR trainees OR consultant OR consultants OR postgraduate OR postgraduates AND augmented reality OR image OR augmented realities OR images AND skill OR skills OR proficiency OR proficiencies

Table 3.

Summary of the Articles for Scoping Review

Author (year)	Procedure	Study design	No. of subjects	Intervention	Control	Outcome	Assessment tool/parameter	Results
Albrecht et al. [24] (2013)	Legal medicine	Non-RCT	6	mARble app on mobile device	NA	Usability of system	AttrakDiff2 questionnaire based on Hassenzahl’s paradigm of experience design	Preliminary evaluation from Hassenzahl’s AttrakDiff2 questionnaire showed an average value of 0.381 (CI, ±0.492) for pragmatic quality.
Amiras et al. [25] (2021)	CT-guided intervention	Non-RCT	16	CT-guided intervention with AR	NA	Feasibility of system	Feedback and questionnaire	From 16 participants, 11 felt the accuracy and realism was adequate for training and 12 felt more confident about their CT biopsy skills after training session.
Andersen et al. [26] (2016)	Port placement + abdominal incision	Non-RCT	20	STAR	Conventional system	Skill improvement	Placement error, number of focus shift, time to task completion	STAR group had less placement error (45% and 68%) and fewer focus shifts (86% and 44%), but they were more slowly (19% for each task). rather than the conventional group. This means that STAR system gave less placement error and fewer focus shifts in the procedure but giving more time to complete the task.
Andersen et al. [4] (2019)	Surgical telementoring of cricothyroidotomy	RCT	20	Telementoring using a visualization of future steps–STAR	Conventional telestrator-based telementoring (Telestrator)	Skill improvement	Idle time ratio, recall error, and task performance	Trainees in the future step visualization group had 48% smaller idle ratio, 26% less recall error, and 10% higher task performance score.
Botden et al. [21] (2009)	Laparoscopic suturing/knot tying	Non-RCT	18	Performing suturing knots on suturing model on AR ProMIS ver. 2.0	NA	Skill improvement	Time spent during knot tying and quality of knot	The result showed a statistically significant difference between the baseline knot and the knot at top of performance curve. Significant difference also found between both of assessment scores.
Broekema et al. [19] (2016)	Laparoscopic skill	Non-RCT	21	Preoperative AR instructional video	Interactive instructional video	Skill improvement	MAP consist of left and right pathlength, insertion, angular area, volume, and time	Both groups showed significant improvements from first task to fourth. After the fourth repetition, when the performance of the two groups was compared, no significant difference in improvement were found.
Cecilio-Fernandes et al. [5] (2020)	Transthoracic echocardiogram	RCT	36	AR visual feedback (HS) (n=16), combination expert and augmented feedback (EF+HS) (n=10)	EF (n=10)	Skill improvement	Time of obtaining images, image quality	On the first test EF group were faster at obtaining images. On the retention test, there were no difference in speed of obtaining images for all group, but EF+HS group had higher score on image quality than the other groups.
Chowriappa et al. [7] (2015)	Robot-assisted UVA	RCT	52	HoST AR simulator	Conventional method	Skill improvement	UVA evaluation score (needle driving, needle positioning and suture placement); GEARS score	HoST group had significant difference of evaluation score in needle driving, needle positioning and suture placement, this group also had higher GEARS score.
Glossop et al. [32] (2004)	Projecting preoperative plans	Non-RCT	NA	The XarTrax AR system	Conventional robotic telesurgery system	Evaluation of system	Accuracy of system on device	The system and device were recommended for carrying out precise preoperative planning and for teleconsultation applications, which allow for the effective transmission of scheduled or real-time consultations to a local caregiver with less training.
Heinrich et al. [20] (2021)	Laparoscopic surgery (cholecystectomy)	Non-RCT	10	AR interactive virtual pointer	Conventional method	Applicability and feasibility	Economy of movement, error rates, user performance	The use of interactive virtual pointer for training showed a significant improvement in economy of movement (p=0.047) and error rates (p=0.047), the user performance also got good scores (total z-score; p=0.031).
Ienghong et al. [14] (2021)	Emergency ultrasound knowledge	RCT	46	Practical ultrasound flashcards with AR and traditional POCUS learning (n=23)	Traditional POCUS learning only (n=23)	Knowledge improvement	Pretest and post-test using multiple-choice question, questionnaire	Participants in flashcard group had better performance on POCUS knowledge post-test than control group. Participants were satisfied with the flashcards.
Kollmann et al. [33] (2021)	Learning knobology of ultrasound equipment	Non-RCT	Students	Detection of the equipment using AR app	NA	Usability of application	Questionnaire	The app had the ability to detect specific ultrasound equipment in real environment and can be used as preparation of practical course or orientation within the lab.
Kratzke et al. [34] (2022)	Tumor ablation performance	Non-RCT	15	AR Emprint SX navigation	Standard ultrasound guidance	Evaluation of ablation (efficacy and efficiency)	Percentage under-ablation and over-ablation, time of trial, number of attempts of antenna position	281 trials were done with the results volume decreased of under-ablations and over-ablations, less task completion time and fewer antenna placement attempts with Emprint SX. In novice group, all metrics were improved by using Emprint SX navigation.
Leitritz et al. [9] (2014)	Binocular indirect ophthalmoscopy	RCT	37	ARO (n=19)	Conventional ophthalmosco py (n=18)	Skill improvement	Ophthalmoscopy training score	Median of training score for conventional group was 1.2 and showed significant difference to ARO group.
Logishetty et al. [28] (2019)	Acetabular cup orientation in total hip arthroplasty	Non-RCT	24	AR guidance	One-on-one instruction from surgeon	Skill improvement	Accuracy of orienting cup, errors, feedback/ questionnaire of users’ experience	AR-guidance group had smaller mean errors in orientation than control group. Both groups showed skill improvement. Eleven of 12 participants said would use the AR-guidance method as training tool and 10 of 12 for procedure-specific training.
Mendes et al. [35] (2020)	Insertion of intravenous needles	Non-RCT	18	Needle insertion with PINATA	Conventional method	Skill improvement	Task completion time and number of needle insertion errors; semi-structured interviews	Correlation was found between task completion time in both methods. The differences of time and errors were found between expert and novice participants. Fourteen participants considered that PINATA can outperform the conventional system of needle insertion.
Menozzi et al. [10] (2020)	Ophthalmic microsurgery training	RCT	23	AR ITS	Natural sequence method	Skill improvement	Motor skill performance	Participants underwent training in 5 days. The result showed both training method gave improvement of motor skill. There was no significance difference in both groups, but for the first 3 days of training, the ITS group showed steeper training progress than natural sequence group.
Mladenovic et al. [11] (2019)	Local anesthesia of IANB	RCT	41	Mobile AR simulator (n=22)	Control group (n=19)	Skill improvement	Time for performing anesthesia, anesthetics success, post-clinical questionnaire	The experimental group had shorter period of time (50.0±14.3 seconds vs. 68.4±25.5 seconds) in performing IANB and higher success rate than the control group. AR group also had 90.9% anesthesia success rate, while control group was 73.7%.
Mladenovic et al. [36] (2022)	Local anesthesia administration	Non-RCT	19	E-learning training using AR dental simulator (simulated-based games)	NA	Effectiveness of simulation-based games	Validated post-training survey	All participants felt comfortable, and over 90% believed that the games could facilitate learning process and had benefit.
Mu et al. [29] (2020)	Ultrasound-guided PCA	Non-RCT	30	AR simulator for ultrasound-guided PCA	NA	Effectiveness of simulator	Face validity, content validity, performance score	The average face validity score of 4.39 and content validity score of 4.53. There was a a significant difference (p<0.05) in performances between experts and novices. Significant performance improvement was found in novice participants.
Muangpoon et al. [37] (2020)	Digital rectal examination	Non-RCT	19	AR model	Real model	Movement in examining finger is accurately aligned the virtual and real model	Questionnaire of system usefulness	Movement in examining finger were accurately aligned the virtual and real model with a frame rate of 60 fps and mean error of 3.9 mm. Participants also found that the movement was realistic (mean±SD, 3.9±1.2).
Müller et al. [30] (2013)	Percutaneous nephrolithotomy	Non-RCT	53	Mobile AR	NA	Feasibility of system	Radiation exposure, puncture time	The trainee had best performance in puncturing time by using AR, while experts did their best with fluoroscopy. Mobile AR lowered radiation exposure by a factor of 3 for trainee and 1.8 for experts. Mean visualization accuracy was 2.5 mm.
Nagayo et al. [27] (2021)	Suture training	Non-RCT	2	AR motion capture system of surgical instruments and a 3D replication system of captured procedures on the surgical field	NA	Usability of system	Tracked time of AR marker, procedure time; performance and usability of replication system	ANOVA demonstrated a significant difference among the three AR markers for both experts. Tukey-Kramer test showed significant differences (p<0.05) for all pairs of AR markers in both experts. The 3D replication system can show the entire 3D models of surgical instruments withing FOV and users can see both models and actual instruments then imitate the replicated procedure.
Nomura et al. [23] (2015)	Laparoscopic skills (cholecystectomy)	Non-RCT	19	Virtual reality simulator (LapSim)	NA	Skill improvement	Pretest and post-test using ProMIS AR	Performance score was better after LapSim training in execution time of the task (p<0.001), left and right instrument path length (p=0.001), left and right instrument economy of movement (p<0.001) than before training.
Oostema et al. [22] (2008)	Laparoscopic skills	Non-RCT	46	ProMIS AR simulator	NA	Feasibility of system	Time, path length smoothness of the procedure	The result showed a significant correlation between experience and performance in time, path length and smoothness for task 2 to 4. The strongest correlation was found for the knot-tying task (R2=0.60 for time and 0.59 smoothness).
Rai et al. [8] (2017)	Binocular indirect ophthalmoscopy	RCT	28	EyeSI AR BIO (n=13)	Conventional BIO (n=15)	Skill improvement	Raw score, total time elapsed and performance score of 3 tasks	The control group outperformed the AR simulator group in three metrics of evaluation.
Rochlen et al. [38] (2017)	Central line insertion	Non-RCT	40	First-person POVAR	NA	Usability and feasibility of first POV AR	Open ended questions and free-text responses	Participants reported that the AR system was realistic (77.5%) and the ability to view the internal anatomy was helpful (92.5%). The success rate of placing the needle was similar between experienced and non-experienced participants.
Ropelato et al. [17] (2020)	Micromanipulation performance in ophthalmic surgery	Non-RCT	50	ITS using AR system	Fixed task sequence	Skill improvement	Performance score of three repetition (before first training, before second training, after second training	A significant improvement in the first session of micromanipulation performance for both groups. For the second session, only ITS group had further improvement.
Shakur et al. [31] (2015)	Percutaneous trigeminal rhizotomy	Non-RCT	92	AR simulator for percutaneous trigeminal rhizotomy	NA	Usefulness of simulator	Post graduate year, number of fluoroscopy shots, distance from the ideal entry point, distance from ideal target (lower score is better)	The mean distance from the entry point (9.4 mm vs. 12.6 mm, p=0.01), the distance from the target (12.0 mm vs. 15.2 mm, p=0.16), and final score (31.1 vs. 37.7, p=0.02) of senior residents was lower than the junior residents. The mean number of fluoroscopy shots was similar between two groups. Increased postgraduate year level had significant result with decreased distance from ideal entry point, shorter distance from target, and better final score.
Sugand et al. [12] (2019)	Dynamic hip screw guidewire insertion	RCT	45	FluoroSim AR simulator	Conventional method	Skill improvement	Tip-apex distance, predicted cut-out rate, total procedural time, total number of radiographs, total number of guidewires retries	The training cohorts had significant improvement in all metrics. Significant difference of both groups was found in procedural time (25%), number of radiograph (57%), and number of guidewires retries (100%). After the 1-week washout period, there was also an improvement in the baseline of the learning curve, which suggests skill retention.
Vera et al. [6] (2014)	Fundamental of laparoscopic surgery suturing task	RCT	19	ART	Traditional mentoring	Skill improvement	Time and errors of the performance	ART group gained faster (b=-0.567, R2=0.92) skill improvement than the control group (b=-0.453, R2=0.74), even after 10 repetition or an hour of practice (mean 167.4 seconds vs. 242.4 seconds, p=0.014). ART group also had fewer errors (8 vs. 13).
Volonté et al. [16] (2011)	Laparoscopic and robotic surgery	Non-RCT	Patients who underwent laparoscopic operations	AR OsiriX	NA	Making surgical interventions easier, faster, and safer	User performance	OsiriX gave benefits to surgeon such as navigations through patients’ anatomy, high lightning regions, and marked pathologies.
Wagner et al. [39] (1999)	Craniomaxillofacial surgical procedure	Non-RCT	27	AR tele-assisted surgery method	NA	Feasibility of system	SPS	Tele-assisted surgery was applied successfully. There were found technical problems in six cases but did not interfere the system performance.
Wolf et al. [13] (2021)	Single ECMO cannulation training	RCT	21	AR instructions	Conventional instructions	Skill improvement	Training times, detailed error protocol, standardized UEQ	AR instructions was associated with slightly higher training times and lower error counts. Training times and error count did not have significant correlation.
Yong et al. [18] (2019)	Temporal bone surgery	Non-RCT	7	ODG R-7 AR Smartglasses	Conventional method	Rating of experience and user-friendliness, ability to perform/teach tasks, quality of communication	Visual analog score was used to obtain mean, SD and range. Open-ended questions were used for collecting the strength and weakness data	Both residents and instructors enjoyed being able to connect by voice, text message, or sending and receiving editable still images. Technical issues were still found and need further improvement.
Yudkowsky et al. [15] (2013)	Ventriculostomy catheter placement	Non-RCT	16	Immersive touch AR system	NA	Success rate and improvement of the procedure	Pre-test and post-test, performance score, questionnaire	The result was the success rate of simulator cannulation was increased and actual procedure outcomes showed improvement. Residents also reported that the simulations were realistic and helpful in improving their skills.
Zhao et al. [40] (2020)	Neonatal endotracheal intubation	Non-RCT	45	Capturing motions and visualization using AR HMD	NA	Accuracy of learning model		The classification accuracy of machine learning model in this study was 84.6%.

RCT: Randomized control trial, NA: Not applicable, CI: Confidence interval, CT: Computed tomography, AR: Augmented reality, STAR: System for telementoring with AR, MAP: Motion analysis parameter, HS: Help screen, EF: Expert feedback, UVA: Urethrovesical anastomosis, HoST: Hands-on surgical training, GEARS score: Global Evaluative Assessment of Robotic Skills score, POCUS: Point-of-care ultrasound, ARO: AR ophthalmoscopy, PINATA: Pinpoint insertion of intravenous needles via AR training assistance, ITS: Intelligent tutoring system, IANB: Inferior alveolar nerve block, PCA: Percutaneous renal access, SD: Standard deviation, 3D: Three-dimensional, ANOVA: Analysis of variance, FOV: Field of view, POV: Point-of-view, ART: AR telementoring, SPS: System performance scale, ECMO: Extracorporeal membrane oxygenation, UEQ: User experience questionnaire, HMD: Head-mounted display.