Non-face to face training during the COVID-19 pandemic era

The coronavirus disease 2019 (COVID-19) pandemic has made it difficult to provide medical training across borders. In particular, it has become virtually impossible to see high-tech medical equipment from other countries and observe surgery to learn. However, educational methods involving the “metaverse” are also being introduced in the medical field to meet the growing demand for non-face-to-face education, as medical staff from around the world who visited Korea to gain expertise in medical technology and medical students who need to practice have had fewer opportunities to work with patients directly due to COVID-19. Efficient surgical training is also challenging using available video-conferencing systems such as Zoom. In order to resolve these limitations, new digital methods using metaverse technology are emerging in the Korean medical community (Supplement 1). In this editorial, I would like to introduce a training course in the metaverse held in Korea and describe expectations about how this technology will be used in the medical field in the future.

Example of training in lung cancer surgery through the metaverse

On May 29, 2021, the 6th Outreach program using an extended reality (XR) technology platform was held under the supervision of the Asian Thoracic Surgery Education Group at the 29th Online Conference of the Asian Heart and Thoracic Surgery Society (Supplement 1). More than 200 thoracic surgeons from Asian countries attended the Outreach program and received training. Surgeons from Manchester University Hospital in the United Kingdom and National University Hospital in Singapore also accessed the virtual environment to visit the system and had active discussions.

The participants in the program wore a head-mounted display (HMD) in their respective laboratories or experienced a real place in a virtual environment with their laptops. The 360° environment was implemented with HMDs and laptops through recent platform upgrades. After participants set up their individual avatars as in a game, they entered a virtual classroom where they viewed lectures on lung cancer surgery techniques and trends in virtual and mixed technology (Fig. 1). They also continued discussions in real-time while observing the surgical process in a virtual environment. Lung cancer surgery education was conducted with high-resolution virtual reality cameras broadcasting all surgical scenes in 360°. The virtual space was decorated to resemble a conference venue.

Surgery was broadcast in the smart operating room of Seoul National University (SNU) Bundang Hospital. Participants indicated that it seemed like they were observing the procedure in the actual operating room because they could see the surgeon, the surgical nurse, and the environment in the operating room as desired through the 360°-8K-3D camera built in the operating room (Figs. 2, 3). The platform is characterized by a virtual environment and high-quality voice conversation through 3-dimensional (3D) XR immersive sound technology. Another advantage is that smooth real-time voice support and various screens such as the actual environment can be implemented. The surgical scene is more visible than when the observer enters the physical operating room and watches the procedure. Participants can also see the surgeon’s monitor and how the operating room nurses hand over surgical tools, how the forceps move, and even the sweat while they suture the wound. As participants move the mouse, they can see every corner of the operating room from the angle that the observer wants. If they wear a 3D headset, the visible scene changes every time they turn their heads, and the feeling of being in the operating room becomes more palpable than when a headset is not used.

How could this training be accomplished?

Accomplishing this training was possible because SNU Bundang Hospital’s smart operating room and metaverse environment are combined. The term “metaverse” combines “meta,” which means “virtual and transcendent,” and “universe,” denoting the world [1]. The metaverse is a reproduction of reality in a virtual space. This concept can be intuitively understood by thinking of virtual spaces using avatars such as Roblox [2] and Fortnite [3]. SNU Bundang Hospital created a smart operating room in 2019 that contains 6 lenses in a virtual reality camera that shoots 360° in all directions, a high-resolution camera, and fluorescent imaging equipment that can visualize lymph nodes in 1 place in the operating room. Advanced imaging equipment has become critical with the increasing frequency of surgery to minimize laparotomy and leverage the power of endoscopy. Surgeons wear 3D glasses and observe high-resolution 3D images from laparoscopic cameras.

Using this technology, an observer can watch the process more extensively and precisely than when a surgeon opens the stomach and performs surgery. The operating room camera or lighting has a voice recognition function. A surgeon can operate with both hands and turn off the camera or light directly with a voice command. Medical staff may have a remote conversation about the biopsy results during surgery. For instance, if a surgeon removes tissue during surgery and sends it to a pathologist, it will be possible for the pathologist to display microscopic images on the large screen of the operating room after the biopsy. While watching the video, the surgeon would listen to the pathologic findings in real-time and share opinions. A control room where an observer can look into the smart operating room transparently is attached right next to it, allowing them to transmit videos or adjust cameras there.

Importance of a smart operating room

After the COVID-19 pandemic, the importance of smart operating rooms has increased. Surgeons from the United States and Europe previously came to SNU Bundang Hospital to observe state-of-the-art surgery, but these visits have become impossible since COVID-19. However, through 360° video, surgeons located anywhere in the world can see the procedures as vividly as if they were in the operating room. There is no need to come to Korea. Dr. Sang-Hoon Ahn, a professor of surgery at SNU Bundang Hospital, said that “Video education using the metaverse is more effective than any other educational method available so far.” An observer can only see the abdominal cavity through a typical video broadcast, but the 360° video shows the entire situation in the operating room. For example, in Professor Ahn’s training on the use of articulated forceps for laparoscopic surgery, it is difficult to know precisely how to use the forceps only by looking at images of them moving in the abdominal cavity. However, through the 360° video, if Professor Ahn changes the wrist’s angle of movement, the observers can see how the forceps move inside the body. They can learn more accurately than from direct observation in the operating room [4]. Furthermore, the number of people who can physically enter an operating room is limited. In addition, in the case of endoscopic surgery, the monitor angle is tailored to the doctor who is operating, so even if someone enters the operating room, they often cannot see the surgical scene correctly. However, videos from the smart operating room do not have this drawback, and they have therefore been used for surgeons, nurses, and medical student education. Dr. So Hyun Kang, an SNU Bundang Hospital fellow, said that this educational system benefits medical residents: “Residents and fellows are always busy. I cannot run to the operating room to observe while working far away. It is only seen in medical dramas that medical doctors look down at the surgical scene from above.” Even if residents cannot watch a surgical broadcast in real-time, they can study by watching videos recorded from various angles later.

Further application of the metaverse through tailored education and training

This demonstration at SNU Bundang Hospital is an example of using the metaverse in clinical education [4]. Among the various types of metaverse applications, this is an example of XR implemented by mixing augmented reality (AR) and virtual reality (VR). XR refers to all VR technologies ranging from VR to mixed reality and AR [5]. With the prolonged COVID-19 situation, the importance of medical education using smart operating rooms and the metaverse is growing. A technical leap is expected, such as higher camera resolution and the ability to deliver haptic input reflecting the surgeon’s tactile sensations. Thus, the metaverse is expected to be actively used in medical education or education for residents and students in the future. The use of the metaverse through tailored education and training in the COVID-19 era will be accelerated even in the post-COVID-19 period. Once these cutting-edge technologies are introduced into the medical field, they will contribute to the health of Koreans and people around the world.

Article information

Authors’ contributions

All the work was done by Huilyung Koo.

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

None.

Data availability

None.

Acknowledgments

Supplementary materials

Fig. 1.

Avatars of participants in the virtual conference of the Asian Heart and Thoracic Surgery Society in 2021 (use of the photo was permitted by Seoul National University Bundang Hospital).

Fig. 2.

Surgical scene in the smart operating room, Seoul National University Bundang Hospital (use of the photo was permitted by Seoul National University Bundang Hospital).

Fig. 3.

Avatars watching a surgical scene in the smart operating room, Seoul National University Bundang Hospital (use of the photo was permitted by Seoul National University Bundang Hospital).

References

• 1. Kye B, Han N, Kim E, Park Y, Jo S. Educational applications of metaverse: possibilities and limitations. J Educ Eval Health Prof 2021;18:32. https://doi.org/10.3352/jeehp.2021.18.32 ArticlePubMedPMC
• 2. Roblox Corp. Roblox [Internet]. San Mateo (CA): Roblox Corp.; c2021 [cited 2021 Dec 29]. Available from: https://www.roblox.com
• 3. Epic Games Inc. Fortnite [Internet]. Cary (NC): Epic Games Inc.; 2021 [cited 2021 Dec 29]. Available from: https://www.epicgames.com/fortnite/en-US/home
• 4. Koo H. Shall we go to the match against COVID-19?: my daily life on “Metaverse”. JTBC [Internet]. 2021 Jul 4 [cited 2021 Dec 29]. Available from: https://news.jtbc.joins.com/article/article.aspx?news_id=NB12014875
• 5. Wikipedia. Extended reality [Internet]. San Francisco (CA): Wikimedia Foundation Inc.; 2021 [cited 2021 Dec 28]. Available from: https://en.wikipedia.org/wiki/Extended_reality

Figure & Data

References

Citations

Citations to this article as recorded by  

• Development of a Metaverse Online Learning System for Undergraduate Nursing Students
  Hyeongju Ryu, Hyeongsuk Lee, Hye Jin Yoo
  Nurse Educator.2024; 49(2): E74.     CrossRef
• Evaluation of Group Genetic Counseling Sessions via a Metaverse-based Application
  Boeun Yoo, Arang Kim, Hye Sung Moon, Min-Kyung So, Tae-Dong Jeong, Kyoung Eun Lee, Byung-In Moon, Jungwon Huh
  Annals of Laboratory Medicine.2024; 44(1): 82.     CrossRef
• Metaverse and oral and maxillofacial radiology: Where do they meet?
  Thaísa Pinheiro Silva, Maria Fernanda Andrade-Bortoletto, Deborah Queiroz Freitas, Christiano Oliveira-Santos, Wilton Mitsunari Takeshita
  European Journal of Radiology.2024; 170: 111210.     CrossRef
• Opportunities and challenges for metaverse application in oral cancer
  Snehasish Tripathy, Ankita Mathur, Kabir Suman Dash, Priyanka Kharat, Vini Mehta
  Oral Oncology Reports.2024; 9: 100181.     CrossRef
• Current Trends and Beyond Conventional Approaches: Advancements in Breast Cancer Surgery through Three-Dimensional Imaging, Virtual Reality, Augmented Reality, and the Emerging Metaverse
  Weronika Magdalena Żydowicz, Jaroslaw Skokowski, Luigi Marano, Karol Polom
  Journal of Clinical Medicine.2024; 13(3): 915.     CrossRef
• A bibliometric analysis of metaverse technologies in healthcare services
  Seckin Damar, Gulsah Hancerliogullari Koksalmis
  Service Business.2024;[Epub]     CrossRef
• Metaverse and Anesthesia
  Valentina Bellini, Massimo Maffezzoni, Elena Bignami
  Anesthesia & Analgesia.2024; 138(3): 491.     CrossRef
• Metaverse and Telementoring: From Surgery to Workshop
  Michele Ammendola, Francesca Vescio, Mohanad Al Ansari, Jozel Hila, Laura Rizzo, Roberto Romano, Francesco Marchegiani, Nicola de’Angelis, Tullio Piardi, Davide Cavaliere, Adam E. Frampton, Tamara M. H. Gall, Maria Luposella, Riccardo Memeo, Giuseppe Nava
  Surgical Innovation.2024; 31(2): 212.     CrossRef
• A Bibliometric Analysis of Technology in Digital Health: Exploring Health Metaverse and Visualizing Emerging Healthcare Management Trends
  Hoang-Sy Nguyen, Miroslav Voznak
  IEEE Access.2024; 12: 23887.     CrossRef
• Advancing Education Through Extended Reality and Internet of Everything Enabled Metaverses: Applications, Challenges, and Open Issues
  Senthil Kumar Jagatheesaperumal, Kashif Ahmad, Ala Al-Fuqaha, Junaid Qadir
  IEEE Transactions on Learning Technologies.2024; 17: 1120.     CrossRef
• Unveiling metaverse sentiments using machine learning approaches
  Thamaraiselvan Natarajan, P. Pragha, Krantiraditya Dhalmahapatra, Deepak Ramanan Veera Raghavan
  Kybernetes.2024;[Epub]     CrossRef
• Assessing the effectiveness and student perceptions of synchronous online flipped learning supported by a metaverse-based platform in medical English education: A mixed-methods study
  Emin İbi̇li̇, Melek Ölmez, Aysel Burcu İbi̇li̇, Fırat Bi̇lal, Abdullah Ci̇han, Nurullah Okumuş
  Education and Information Technologies.2024;[Epub]     CrossRef
• DİJİTAL DÖNÜŞÜM ÇAĞINDA SAĞLIK SEKTÖRÜNDE METAVERSE: SWOT ANALİZİ
  Mehmet Naci Efe
  ASSAM Uluslararası Hakemli Dergi.2024; (24): 50.     CrossRef
• CardioVerse: The cardiovascular medicine in the era of Metaverse
  Ioannis Skalidis, Olivier Muller, Stephane Fournier
  Trends in Cardiovascular Medicine.2023; 33(8): 471.     CrossRef
• Who really needs a Metaverse in anatomy education? A review with preliminary survey results
  Joe Iwanaga, Edward C. Muo, Yoko Tabira, Koichi Watanabe, Susan J. Tubbs, Anthony V. D'Antoni, Mathangi Rajaram‐Gilkes, Marios Loukas, Mohammed K. Khalil, R. Shane Tubbs
  Clinical Anatomy.2023; 36(1): 77.     CrossRef
• Virtual hospitals and digital doctors: how far are we from the CardioVerse?
  Ioannis Skalidis, Stephane Fournier, Emmanouil Skalidis, Niccolo Maurizi
  European Heart Journal.2023; 44(1): 7.     CrossRef
• Facing the challenges of metaverse: a systematic literature review from Social Sciences and Marketing and Communication
  Verónica Crespo-Pereira, Eva Sánchez-Amboage, Matías Membiela-Pollán
  El Profesional de la información.2023;[Epub]     CrossRef
• Metaverse in Healthcare Integrated with Explainable AI and Blockchain: Enabling Immersiveness, Ensuring Trust, and Providing Patient Data Security
  Sikandar Ali, Abdullah, Tagne Poupi Theodore Armand, Ali Athar, Ali Hussain, Maisam Ali, Muhammad Yaseen, Moon-Il Joo, Hee-Cheol Kim
  Sensors.2023; 23(2): 565.     CrossRef
• Tumor Lung Visualization and Localization through Virtual Reality and Thermal Feedback Interface
  Samir Benbelkacem, Nadia Zenati-Henda, Nabil Zerrouki, Adel Oulefki, Sos Agaian, Mostefa Masmoudi, Ahmed Bentaleb, Alex Liew
  Diagnostics.2023; 13(3): 567.     CrossRef
• How do Plastic Surgeons use the Metaverse: A Systematic Review
  Pengfei Sun, Shanbaga Zhao, Yang Yang, Chao Liu, Bo Pan
  Journal of Craniofacial Surgery.2023; 34(2): 548.     CrossRef
• Metaverse for Healthcare: A Survey on Potential Applications, Challenges and Future Directions
  Rajeswari Chengoden, Nancy Victor, Thien Huynh-The, Gokul Yenduri, Rutvij H. Jhaveri, Mamoun Alazab, Sweta Bhattacharya, Pawan Hegde, Praveen Kumar Reddy Maddikunta, Thippa Reddy Gadekallu
  IEEE Access.2023; 11: 12765.     CrossRef
• The rising trend of Metaverse in education: challenges, opportunities, and ethical considerations
  Sanaa Kaddoura, Fatima Al Husseiny
  PeerJ Computer Science.2023; 9: e1252.     CrossRef
• Metaverse applied to musculoskeletal pathology: Orthoverse and Rehabverse
  Juan M. Román-Belmonte, E. Carlos Rodríguez-Merchán, Hortensia De la Corte-Rodríguez
  Postgraduate Medicine.2023; 135(5): 440.     CrossRef
• Beyond Your Sight Using Metaverse Immersive Vision With Technology Behaviour Model
  Poh Soon JosephNg, Xiaoxue Gong, Narinderjit Singh, Toong Hai Sam, Hua Liu, Koo Yuen Phan
  Journal of Cases on Information Technology.2023; 25(1): 1.     CrossRef
• Metaverse and translation studies: analysis of machine translation
  Rubén González Vallejo
  Metaverse Basic and Applied Research.2023;[Epub]     CrossRef
• Using the metaverse in training: lessons from real cases
  Omaima Hajjami, Sunyoung Park
  European Journal of Training and Development.2023;[Epub]     CrossRef
• What can the metaverse do for urology?
  Carlo Gandi, Luigi Cosenza, Marco Campetella, Filippo Marino, Mauro Ragonese, Riccardo Bientinesi, Angelo Totaro, Marco Racioppi, Emilio Sacco
  Urologia Journal.2023; 90(3): 454.     CrossRef
• Metaverse: Freezing the time
  Pengxin Qian, Dawei Yang, Chunxue Bai
  Clinical eHealth.2023; 6: 29.     CrossRef
• Metaverse Chronicles: A Bibliometric Analysis of Its Evolving Landscape
  Walton Wider, Leilei Jiang, Jiaming Lin, Muhammad Ashraf Fauzi, Jingjing Li, Choon Kit Chan
  International Journal of Human–Computer Interaction.2023; : 1.     CrossRef
• Merging virtual and physical experiences: extended realities in cardiovascular medicine
  Tsung-Ying Tsai, Yoshinobu Onuma, Adriana Złahoda-Huzior, Shigetaka Kageyama, Dariusz Dudek, Qingdi Wang, Ruth P Lim, Scot Garg, Eric K W Poon, John Puskas, Fabio Ramponi, Christian Jung, Faisal Sharif, Arif A Khokhar, Patrick W Serruys
  European Heart Journal.2023; 44(35): 3311.     CrossRef
• Pediatric Urology Metaverse
  Marcello Della Corte, Erica Clemente, Enrico Checcucci, Daniele Amparore, Elisa Cerchia, Berenice Tulelli, Cristian Fiori, Francesco Porpiglia, Simona Gerocarni Nappo
  Surgeries.2023; 4(3): 325.     CrossRef
• Collaborative strategies for deploying AI-based physician decision support systems: challenges and deployment approaches
  Mirja Mittermaier, Marium Raza, Joseph C. Kvedar
  npj Digital Medicine.2023;[Epub]     CrossRef
• Metaverse-related perceptions and sentiments on Twitter: evidence from text mining and network analysis
  Uğur Gündüz, Sadettin Demirel
  Electronic Commerce Research.2023;[Epub]     CrossRef
• Teaching Building Information Modeling in the Metaverse—An Approach Based on Quantitative and Qualitative Evaluation of the Students Perspective
  Niels Bartels, Kristina Hahne
  Buildings.2023; 13(9): 2198.     CrossRef
• Toward Trustworthy Metaverse: Advancements and Challenges
  Jamin Rahman Jim, Md. Tanzib Hosain, M. F. Mridha, Md. Mohsin Kabir, Jungpil Shin
  IEEE Access.2023; 11: 118318.     CrossRef
• METAVERSE AND HEALTHCARE SECTOR
  Ziya ÇEÇEN, Oğuzhan YÜKSEL
  International Journal of Engineering and Innovative Research.2023; 5(3): 280.     CrossRef
• Implications of immersive technologies in healthcare sector and its built environment
  Eunsil Yang
  Frontiers in Medical Technology.2023;[Epub]     CrossRef
• Phenomenological toolkit of the metaverse for medical informatics’ adaptive learning
  Inna Kryvenko, Kyrylo Chalyy
  Educación Médica.2023; 24(5): 100854.     CrossRef
• Metaverse in Medical Education
  Agus Rizal Ardy Hariandy Hamid, Ferdiansyah Sultan Ayasasmita Rusdhy, Prasandhya Astagiri Yusuf
  Medical Journal of Indonesia.2023; 32(2): 67.     CrossRef
• Not just for play: Why the metaverse should be leveraged for co-creation
  Yuna Kim
  Business Horizons.2023;[Epub]     CrossRef
• The Multidomain Metaverse Cancer Care Digital Platform: Development and Usability Study
  Sunghak Kim, Timothy Jung, Dae Kyung Sohn, Yoon Chae, Young Ae Kim, Seung Hyun Kang, Yujin Park, Yoon Jung Chang
  JMIR Serious Games.2023; 11: e46242.     CrossRef
• The surgical metaverse
  Kabir Matwala, Taner Shakir, Chetan Bhan, Manish Chand
  Cirugía Española (English Edition).2023;[Epub]     CrossRef
• Application and challenges of a metaverse in medicine
  Yingshu Wang, Congcong Li, Lai Qu, Hongfei Cai, Yingying Ge
  Frontiers in Robotics and AI.2023;[Epub]     CrossRef
• MedMetaverse: Medical Care of Chronic Disease Patients and Managing Data Using Artificial Intelligence, Blockchain, and Wearable Devices State-of-the-Art Methodology
  Dileep Kumar Murala, Sandeep Kumar Panda, Sujata Priyambada Dash
  IEEE Access.2023; 11: 138954.     CrossRef
• The Role of the Metaverse in Transforming Healthcare
  Srinivasan Ramamurthy, Sarah Al Yammahi, Alya Abdul Rahim
  Research Journal of Pharmacy and Technology.2023; : 5506.     CrossRef
• Application of computer-based testing in the Korean Medical Licensing Examination, the emergence of the metaverse in medical education, journal metrics and statistics, and appreciation to reviewers and volunteers
  Sun Huh
  Journal of Educational Evaluation for Health Professions.2022; 19: 2.     CrossRef
• The Metaverse in Cardiovascular Medicine: Applications, Challenges, and the Role of Non-Fungible Tokens
  Ioannis Skalidis, Olivier Muller, Stephane Fournier
  Canadian Journal of Cardiology.2022; 38(9): 1467.     CrossRef
• Metaverse phenomenon and its impact on health: A scoping review
  Ali Garavand, Nasim Aslani
  Informatics in Medicine Unlocked.2022; 32: 101029.     CrossRef
• Provision of surgical pre-operative patient counseling services through the Metaverse technology
  Anusha Anwer, Yumna Jamil, Maham Bilal
  International Journal of Surgery.2022; 104: 106792.     CrossRef
• Evolution of Medical Conferences for Congenital Heart Disease Imagers in the Era of COVID-19: From Onsite to Virtual Meetings
  I-Chen Tsai, Hyun Woo Goo, Haifa Abdul Latiff, Seon Young Goo, Sang Joon Park
  Cardiovascular Imaging Asia.2022; 6(3): 82.     CrossRef
• Gastroenterology in the Metaverse: The dawn of a new era?
  Chi Zhang, Shuyan Feng, Ruonan He, Yi Fang, Shuo Zhang
  Frontiers in Medicine.2022;[Epub]     CrossRef
• The Metaverse: A New Challenge for the Healthcare System: A Scoping Review
  Luca Petrigna, Giuseppe Musumeci
  Journal of Functional Morphology and Kinesiology.2022; 7(3): 63.     CrossRef
• The metaverse in education: Definition, framework, features, potential applications, challenges, and future research topics
  Xinli Zhang, Yuchen Chen, Lailin Hu, Youmei Wang
  Frontiers in Psychology.2022;[Epub]     CrossRef
• Novel pathway regarding good cosmetics brands by NFT in the metaverse world
  Jinkyung Lee, Ki Han Kwon
  Journal of Cosmetic Dermatology.2022; 21(12): 6584.     CrossRef
• Advances in Metaverse Investigation: Streams of Research and Future Agenda
  Mariapina Trunfio, Simona Rossi
  Virtual Worlds.2022; 1(2): 103.     CrossRef
• Dynamics of Metaverse and Medicine: A Review Article
  Mrudul A Kawarase, Ashish Anjankar
  Cureus.2022;[Epub]     CrossRef
• What the Literature on Medicine, Nursing, Public Health, Midwifery, and Dentistry Reveals: An Overview of the Rapidly Approaching Metaverse
  Muhammet DAMAR
  Journal of Metaverse.2022; 2(2): 62.     CrossRef
• Interactive and innovative technologies for smart education
  Babatomiwa Omonayajo, Fadi Al-Turjman, Nadire Cavus
  Computer Science and Information Systems.2022; 19(3): 1549.     CrossRef