3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
21
SUPERIMPOSED ANATOMICAL STRUCTURES IN
AUGMENTED REALITY
Vladimir Ivanov
Herzen State Pedagogical University of Russia,
Saint-Petersburg, (Russia).
E-mail: voliva@rambler.ru ORCID: https://orcid.org/0000-0001-8194-2718
Nikolay Kalakutskiy
Pavlov First State Medical University of St. Petersburg,
Saint-Petersburg, (Russia).
E-mail: kalakutsky@yandex.ru ORCID: https://orcid.org/0000-0002-6851-5073
Alexander Klygach
Herzen State Pedagogical University of Russia,
Saint-Petersburg, (Russia).
E-mail: voolf00@yandex.ru ORCID: https://orcid.org/0000-0002-2984-0201
Sergey Strelkov
Herzen State Pedagogical University of Russia,
Saint-Petersburg, (Russia).
E-mail: sergin3d2d@gmail.com ORCID: https://orcid.org/0000-0002-4830-5407
Recepción:
07/08/2020
Aceptación:
29/09/2020
Publicación:
13/11/2020
Citación sugerida Suggested citation
Ivanov, V., Kalakutskiy, N., Klygach, A., y Strelkov, S. (2020). Superimposed anatomical structures in
augmented reality. 3C Tecnología. Glosas de innovación aplicadas a la pyme. Edición Especial, Noviembre 2020,
21-31. https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
22
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
ABSTRACT
The paper considers an approach to visualization of anatomical structures using augmented
reality, which allows for more accurate planning of operations and reduces the degree of
surgical intervention.
To achieve this goal, a technology was developed using a frame with a volumetric optical
marker, which allows you to correlate a virtual 3D model with the patient. As a result, it
was possible to achieve an accurate positioning of the 3D model and obtain a reliable
visualization of the location of internal anatomical structures. At the same time, the parallel
application of spatial mapping technology allows you to x the 3D model at a certain point
in space and, in case of loss of marker tracking, save the correct location of the 3D model,
taking into account that the patient is in a stationary state.
This approach can be eectively applied in surgery, if it is possible to x the skeleton relative
to any bone structure. Options for surgical intervention can be dierent, starting with
the installation of prostheses, ending with the removal of cancerous tumors. The article
discusses a project to develop a technology for using augmented reality in planning and
conducting maxillofacial surgery. The capabilities of the technology, as well as the prospects
for its use, are highlighted.
KEYWORDS
Medicine, Surgery, Maxillofacial surgery, Augmented reality, Markers, 3D graphics,
Visualization.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
23
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
1. INTRODUCTION
Worldwide, the modern standard for jaw cancer surgery is the simultaneous reconstruction
of a defect with a vascular graft. To reduce the risk of intra- and postoperative complications,
improve the postoperative quality of life of patients and increase the accuracy of operations,
much attention is paid to their careful preoperative planning (Hartman, Spauwen, & Jansen,
2002). The complexity of planning reconstructive operations in oncology lies in the need to
combine data from dierent studies to simultaneously visualize several body tissues (bones,
blood vessels, nerves, soft tissues) in order to increase the objectivity of determining the
boundaries of the resection, optimizing the choice of a vascularized graft and determining
donor and recipient vessels. All the above tasks correspond to an oine “map” of the
patient, used in the developed augmented reality program, which is compiled on the basis
of CT or MRI data and can be supplemented by ultrasound, dopplerography, angiography,
etc.
Augmented Reality (AR) is a combination of virtual reality and with the “oine map” of
a patient and allows the user to interact in real time between various material and virtual
objects, as if they exist in a single environment (Bartella et al., 2019). Using this feature,
we can increase the productivity and quality of services in many areas, especially in head
and neck surgery. The proposed solution will help the doctor easily see where the internal
anatomical structures are located during the operation. We oer an advanced solution for
tracking optical markers with spatial imaging based on patient CT data. The surgeon during
the operation through glasses sees a 3D visualization of the received data, due to which he
less looks at the monitor with the data of diagnostic tests and constantly monitors the state
of the surgical wound, which reduces the risk of intraoperative complications, including
bleeding, which leads to an increase in the accuracy of surgery, reducing its duration and
reducing the risk of complications. Also, this project can be widely used for educational
purposes.
Currently, many scientic papers on the use of augmented reality in various elds of
medicine have been published: neurosurgery, cardiology, urology, plastic surgery, including
dentistry and maxillofacial surgery (Ayoub & Pulijala, 2019; Elmi-Terander et al., 2020; Kim,
Kim, & Kim, 2017; Kwon, Park, & Han, 2018). However, most of them are descriptions
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
24
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
of individual clinical cases or their series, which certainly does not have a very high class
of evidence (Bosc et al., 2019). Available literature reviews also note the lack of clinical trial
results for a more objective assessment of this method. Therefore, the goal of our work is
to conduct a multicenter randomized clinical trial in the treatment of patients with jaw
cancer, which shows their resection with simultaneous plastic surgery of the defect with a
vascularized bone graft with microtechnology. A feature of basic research is their careful
planning, strict adherence to the research protocol, good statistical analysis of the data
obtained, as well as a unied scale for assessing the data obtained (calibrated investigator)
and their analysis by an uninvolved specialist, to reduce the likelihood of bias on the part
of the scientist when interpreting the data, compliance All of the above conditions will
allow you to perform a high-class evidence study, the results of which will be useful for the
development of modern science.
On the basis of the Department of Maxillofacial and Reconstructive Surgery of the First
St. Petersburg Medical University named after Acad. I.P. Pavlova reconstructive plastic
surgery to eliminate jaw defects due to malignant and benign neoplasms, post-traumatic
defects and deformations of various genesis with the help of vascularized bone and soft-
tissue aps with microsurgical technique has been carried out for more than 30 years by a
team led by professor N.V. Kalakutsky During this time, vast clinical experience has been
accumulated in the treatment of this group of patients, which will facilitate the introduction
of new technologies and improve the quality of the study.
According to the example of the following clinical case of using augmented reality
technology in dentistry, within the framework of the planned study, it is supposed to be
used when performing a jaw resection for cancer with simultaneous plastic surgery of the
defect with a bone graft with microtechnology.
2. METHODOLOGY
2.1. MEDICAL CASE
For the rst case, we took a patient who has a small hollow cluster of cells that is grouped
together - a cyst in the left nasal sinus (Figure 1), which was recommended for removal.
It also has a complex and uneven shape. Therefore, to help the doctor better understand
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
25
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
where he is during planning and work, we have developed a visual solution with the display
of anatomical structures on top of what the doctor sees.
Figure 1. Cyst in left sinus marked in red.
2.2. OVERLAY 3D MODELS OF THE SKULL IN THE REAL WORLD
For the selected task, Augmented Reality is the optimal visual solution.
Augmented Reality (AR) presents information in the right context of the real world. To
do this, the system must know where the user is and what he is looking at. Usually the user
observes an object through a display that displays a camera image along with augmented
reality. Thus, in practice, the system needs to determine the location and orientation of the
camera relative to the object. In the case of Hololens glasses, it is almost the same. To do
this, use a separate built-in camera, and as a display - two holographic projectors.
In AR technology, tracking markers are used for display on the display and in the camera -
AR codes that help align the 3d model from the user’s point of view. In our problem, when
an object should be viewed from dierent sides, we used a special marker called cuboid,
where on each side there is a unique AR code. The Hololens camera has a limited resolution
of 720p and a low dynamic range compared to modern cameras. This causes problems
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
26
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
with accurate tracking. To improve this, we have added the wrong organic structure, which
provides additional tracking capabilities.
Figure 2. Type of marker.
In addition to the problem of aligning the object with the camera, our case is complicated
by the fact that the marker must be xed on the persons head, strictly in a certain position
and with a certain orientation. For this task, we built a 3D model of the head from CT data
using the Materialize Mimics software. Then we designated the areas where the marker will
align and x. Based on the selected areas, a three-dimensional wireframe model was built
with a platform for attaching a marker. The frame was made in such a way that it could
be easily put on and xed in the correct position (Figure 3). Finally, it was printed on a 3D
printer (Figure 4), and the marker was attached at a predetermined location.
Figure 3. Left image - A constructed model of the face and skull based on CT data; in the middle are designated
areas for xing the frame, on the right is the model of the frame with a zone for the marker on top.