303
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
ADVANCES IN AUGMENTED REALITY (AR) FOR MEDICAL
SIMULATION AND TRAINING
Vladimir Ivanov
Herzen State Pedagogical University of Russia
Saint-Petersburg, (Russia).
E-mail: voliva@rambler.ru ORCID: https://orcid.org/0000-0001-8194-2718
Alexander Klygach
Herzen State Pedagogical University of Russia
Saint-Petersburg, (Russia).
E-mail: voolf00@yandex.ru ORCID: https://orcid.org/0000-0002-2984-0201
Sam Shterenberg
Pavlov First Saint-Petersburg State Medical University
Saint-Petersburg, (Russia).
E-mail: sam.d.s@mail.ru ORCID: https://orcid.org/0000-0002-6428-8328
Sergey Strelkov
Herzen State Pedagogical University of Russia,
Saint-Petersburg, (Russia).
E-mail: sergin3d2d@gmail.com ORCID: https://orcid.org/0000-0002-4830-5407
Jason Levy
University of Hawaii,
Honolulu, (USA).
E-mail: jlevy@hawaii.edu ORCID: https://orcid.org/0000-0002-9978-5412
Recepción:
26/02/2020
Aceptación:
17/04/2020
Publicación:
30/04/2020
Citación sugerida Suggested citation
Ivanov, V., Klygach, A., Shterenberg, S., Strelkov, S., y Levy, J. (2020). Advances in augmented reality
(AR) for medical simulation and training. 3C Tecnología. Glosas de innovación aplicadas a la pyme. Edición
Especial, Abril 2020, 303-312. http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
304
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
ABSTRACT
Digital technologies are transforming the eld of medical training, simulation and mode-
ling. Advances in the eld of virtual Augmented Reality (AR) and virtual simulation are
described in detail, particularly as they relate to medical education and training. An over-
view of key medical simulation tools is provided in order provide foundational knowledge
about this rapidly growing eld. A timely and valuable original Augmented Realty system
is put forward. The key components of this original system for medical training and simu-
lation include the following three dimensions: advances in open surgery, realistic visualiza-
tions and innovative haptic was used. Each component of this Augmented Reality system is
described in detail. First, the open surgery module emphasized appendectomies (the most
common surgical procedures used in our model). Second, three dierent approaches for
creating realistic and accurate 3D medical models were put forth. Third, haptic feedback
involved the use of an enhanced Novint Falcon system in which a custom grip provides
additional degrees of freedom. Finally, advances in game simulation, modeling and role
playing are discussed for the eld of emergency medicine.
KEYWORDS
Surgical Simulator, Virtual Reality, Real-time Rendering, 3D Visualization, Haptic Feed-
back, Open Surgery, Laparoscopy, Emergency Medicine, Simulations.
305
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
1. INTRODUCTION
Medical simulations, modeling and visualizations have undergone a rapid shift the begin-
ning of 20th century due to a number of reasons (Kron et al., 2010). First, modern approa-
ches for less invasive surgery have redened surgical procedures such as endoscopy and ro-
botics surgery. Second, the dramatic rise of computing power has provided an opportunity
to implement complex simulations in real-time. Finally, more accurate algorithms for rigid
and soft body simulations, realistic 3d visualizations, haptic controllers, and virtual reality
have allowed medical simulation to be used for digital gaming rather than simply physical
modeling.
Specic approaches and technologies for medical simulations for medical simulations have
grown by leaps and bounds. For example, innovative research has occurred dealing with
the generation of textures of irregular objects from models and photo sequences (Chen et
al., 2003). The role of medical simulations has rapidly expanded throughout the healthcare
eld (Kunkler, 2006). This paper involves a case study of medical simulations for lap.
2. MODERN MEDICAL SIMULATION MARKET
2.1. MARKET OVERVIEW
According to Prescient & Strategic Intelligence data, the global surgical simulation market was
valued at $254.7 million in 2017 with a growing trend. The value of this eld is forecasted to
increase to twice its value in 2023 (Figure 1). Another notable trend is that augmented reality
(AR) and virtual reality (VR) are being used to enhance the quality and eciency of medical
training. Thus, it is expected that this market will continue to grow, and digital technologies
will continue to have a major impact on the medical simulation eld.
Figure 1. Worldwide surgical simulation market by offering (2013-2023).
306
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
2.2. MEDICAL SIMULATORS
VirtaMed is a company primarily focused on simulator development for orthopedics,
genecology and urology. The company develops surgical simulators which are designed
on a single exible plat-form with the ability to expand and add additional procedures. All
simulators are combined with an anatomical model to provide the optimal tactile feedback
and real-world manipulations. In addition, for better eciency each virtual procedure
allows for guided training: specic colored hints and ghost tools show trainees how to
perform dierent tasks (Figure 2).
Figure 2. ArthroS Ankle by VirtaMed AG.
NeuroVR is a platform for neurological training that enables neurosurgeons to practice
skills with the help of virtual reality (Figure 3). Such a system does not depend on real life
models but uses haptic controllers for VR manipulations. The range of allowed exercises
are derived from actual patient images, which provides more realistic and accurate images
of surgical procedures. The system also captures objective metrics and measures the
prociency of procedures in order to track educational progress.
Figure 3. NeuroVR system with stereoscopic microscopic view.
SurgicalScience is a company which develops various simulation products, mostly for
laparoscopy and endoscopy. The LapSim product is designed to improve psychomotor
307
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
skills using virtual reality with haptic feedback (Figure 4). It features dierent modules for
laparoscopic exercises that arrange from navigation to suturing. This system also has a
portable version known as “LapSim essence”.
Figure 4. LapSim with in-house developed haptic system.
3. AN ORIGINAL APPROACH TO THE DEVELOPMENT OF
MEDICAL SIMULATION
3.1. KEY ASPECTS
Most surgical simulators that are currently available on the market have weaknesses ranging
from simplied 3D visualizations to a limited ability to perform open surgery. Most existing
modeling tools are also limited because they have been designed primarily for specic sur-
gical approaches, like endoscopy. For this reasons we develop an open surgery simulator for
carried out an appendectomy with realistic visualizations using haptic feedback (Figure 5).
Figure 5. Surgery simulator based on open appendectomy with haptic feedback.
308
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
3.2. REALISTIC VISUALIZATION
Despite advanced real-time rendering solutions that are currently available, it is still dicult
to produce realistic images in a surgical simulator due to a software limitations. For example
it is dicult to incorporate graphics solutions onto a complex modeling engine (that captures
the physics of the system). To overcome this challenge the developed system is based on
a modern game engine which allows for the use of physically based shader models and
enhances it with a customized physics engine to work with soft tissue (Figure 6).
Figure 6. The stage, where the mesoappendix is dissected.
3.3. CREATING MODELS BASED ON PATIENT’S DATA AND ANATOMICAL
ATLASES
More accurate modeling results may be obtained by a number of approaches. One method
involves taking photos from an actual surgical procedure and extracting textures from these
images (Figure 7). It is also possible to build models based on a sequence of photos using
photogrammetry solutions.
Figure 7. Photo from surgery procedure (left), appendix model with textures (center), overall view in surgical
simulator (right).
309
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
Another method involves building models based on MRI or CT data. For instance, a heart
model with specic pathologies can be recreated with the help of MRI and contouring data.
This is achieved through multiple stages (Figure 8). At rst, we built model of ventricles
from countering data and then projected a master heart model onto them and nally added
textures based on real-life heart images.
Figure 8. MRI with contouring data (left), reconstructed heart 3d model (center), heart 3d model with textures
and internal structures (right).
In complex cases, where photography or tomography is not enough to build a full model,
anatomical atlases are used: knowledge and data from multiple atlases like Gray’s anatomy
and 3d atlas of human body were combined to reveal the position of the larynx. This
helped accurately showcase the location of the larynx in comparison other anatomical
structures like skull, and muscles (Figure 9).
Figure 9. Larynx image from Gray’s anatomy atlas (left), 3d model of larynx (center), neck section with larynx
(right).
3.4. HAPTIC FEEDBACK
In order to achieve fully realistic visualization our system is supplemented with haptic feed-
back. This helps to sense virtual 3D objects and allows the development of proper psycho-
motor skills for surgeons. The Novint Falcon Haptic device was used as the foundation to
implement this haptic feedback technology. While this tool is designed primary for games,
310
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
it also provides accurate haptic feed-back with three degrees of freedom (DOF). This de-
vice is also the cheapest haptic system on the market, thereby reducing the overall cost of
simulator signicantly.
In order to use this device as a tool for surgical simulations the system was redesigned in
several ways. For example a custom designed grip was developed with extra three DOF to
allow for the creation of a tilting surgical tool (Figure 10). This tool is based on absolute hall
encoders and transmits data as separate stream through a digital-to-analog converter. In
addition, it has a slot for swapping dierent surgical instruments.
Figure 10. Custom designed grip to provide extra 3-degree of freedom.
3.5. LAPROSCOPIC SURGICAL SIMULATORS
There are a number of laparoscopic surgical simulators available on the market. Laparosco-
pic training simulators have been used eectively to enhance laparoscopic surgery training
scenarios. For example, Sauerland, Jaschinski and Neugebauer (2010) examined the use of
laparoscopic versus open surgery for suspected appendicitis. Our proposed Laparoscopic
Surgery Simulator can demonstrate virtually all major abdominal surgical procedures and
assist surgeons in learning a range of surgical methodologies. Our open surgery simulator
constitutes a timely, complex and important niche simulator tool. The aforementioned so-
lutions for creating 3d models and haptic feedback can help to achieve more realistic lapa-
roscopic surgery visualization and modeling in a cost-ecient way.
4. GAMES, ROLE PLAYING AND SIMULATIONS FOR EMERGENCY
PHYSICIANS AND THE COVID-19 PANDEMIC
There is a need for better-trained emergency physicians in the COVID-19 era. Emergency
physicians and other educational, research, and practitioners in the health eld must in-
311
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
creasingly have simulation experience to make crucial decisions within a highly politicized,
volatile and pressurized context. The education of future generations of emergency physi-
cians and emergency management professionals must draw across the boundaries of phy-
sical/social science, technology, engineering, and mathematics. The COVID-19 pandemic
has shown that traditional approaches to responding to health disasters, reducing risk and
mitigating losses are inadequate. Starting in the 1950s, the emphasis was on the disciplines
of civil defense and humanitarian relief.
Role play, scenario methods and game simulations have been successfully used across a
wide variety of learning environments ranging from military training, high school driver’s
education to the diplomatic art of negotiation. Role play and game simulation provide a
learning-by-doing experience and thus have been shown to increase emergency manage-
ment eectiveness in various application ranging from hospital disaster planning and public
health to the mental and social aspects of health emergencies. Modeling and simulation
tools have been shown to build higher cognitive skills for emergency management training
and education and to help with individual and group learning in crisis simulations. There
are a number of emergency medicine elds where role play and game simulations are used
successfully in improving graduate and undergraduate learning outcomes. Despite the suc-
cesses of role play and game simulations across a wide range of educational environments,
there is a need for more such learning tools within in emergency healthcare education.
Our ongoing research seeks provide a bridge between the emergency response focus ge-
nerally lled by community and technical colleges and theoretical focus of graduate level
courses. Future work seeks to develop prototype simulations for undergraduate emergency
medical curricula. A further outcome of the project will be a framework and workplan to
leverage the lessons learned in this paper for complete development, implementation and
faculty support materials of emergency medicine simulations.
REFERENCES
Chen, R., Lu, D., & Pan, Y. (2003). Generating Textures of Irregular Objects from Models
and Photo Sequences. Journal of Image and Graphics, 8.
312
http://doi.org/10.17993/3ctecno.2020.specialissue5.303-312
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Abril 2020
Kron, F. W., Gjerde, C. L., Sen, A., & Fetters, M. D. (2010). Medical student attitudes
toward video games and related new media technologies in medical education. BMC
Medical Education, 10(50), 1–10. https://doi.org/10.1186/1472-6920-10-50
Kunkler, K. (2006) The role of medical simulation. The International Journal of Medical
Robotics and Computer Assisted Surgery, 2(3), 203–210. https://onlinelibrary.wiley.com/
doi/10.1002/rcs.101
McDermott, W. (2018). The PBR guide. Part 1. Allegorithmic. https://academy.substance3d.
com/courses/the-pbr-guide-part-1
Sauerland, S., Jaschinski, T., & Neugebauer, E. A. (2010). Laparoscopic versus
open surgery for sus-pected appendicitis. Cochrane Database of Systematic Reviews, (10).
https://doi.org/10.1002/14651858.CD001546.pub3
