3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
21
SUPERIMPOSED ANATOMICAL STRUCTURES IN
AUGMENTED REALITY
Vladimir Ivanov
Herzen State Pedagogical University of Russia,
Saint-Petersburg, (Russia).
E-mail: voliva@rambler.ru ORCID: https://orcid.org/0000-0001-8194-2718
Nikolay Kalakutskiy
Pavlov First State Medical University of St. Petersburg,
Saint-Petersburg, (Russia).
E-mail: kalakutsky@yandex.ru ORCID: https://orcid.org/0000-0002-6851-5073
Alexander Klygach
Herzen State Pedagogical University of Russia,
Saint-Petersburg, (Russia).
E-mail: voolf00@yandex.ru ORCID: https://orcid.org/0000-0002-2984-0201
Sergey Strelkov
Herzen State Pedagogical University of Russia,
Saint-Petersburg, (Russia).
E-mail: sergin3d2d@gmail.com ORCID: https://orcid.org/0000-0002-4830-5407
Recepción:
07/08/2020
Aceptación:
29/09/2020
Publicación:
13/11/2020
Citación sugerida Suggested citation
Ivanov, V., Kalakutskiy, N., Klygach, A., y Strelkov, S. (2020). Superimposed anatomical structures in
augmented reality. 3C Tecnología. Glosas de innovación aplicadas a la pyme. Edición Especial, Noviembre 2020,
21-31. https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
22
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
ABSTRACT
The paper considers an approach to visualization of anatomical structures using augmented
reality, which allows for more accurate planning of operations and reduces the degree of
surgical intervention.
To achieve this goal, a technology was developed using a frame with a volumetric optical
marker, which allows you to correlate a virtual 3D model with the patient. As a result, it
was possible to achieve an accurate positioning of the 3D model and obtain a reliable
visualization of the location of internal anatomical structures. At the same time, the parallel
application of spatial mapping technology allows you to x the 3D model at a certain point
in space and, in case of loss of marker tracking, save the correct location of the 3D model,
taking into account that the patient is in a stationary state.
This approach can be eectively applied in surgery, if it is possible to x the skeleton relative
to any bone structure. Options for surgical intervention can be dierent, starting with
the installation of prostheses, ending with the removal of cancerous tumors. The article
discusses a project to develop a technology for using augmented reality in planning and
conducting maxillofacial surgery. The capabilities of the technology, as well as the prospects
for its use, are highlighted.
KEYWORDS
Medicine, Surgery, Maxillofacial surgery, Augmented reality, Markers, 3D graphics,
Visualization.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
23
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
1. INTRODUCTION
Worldwide, the modern standard for jaw cancer surgery is the simultaneous reconstruction
of a defect with a vascular graft. To reduce the risk of intra- and postoperative complications,
improve the postoperative quality of life of patients and increase the accuracy of operations,
much attention is paid to their careful preoperative planning (Hartman, Spauwen, & Jansen,
2002). The complexity of planning reconstructive operations in oncology lies in the need to
combine data from dierent studies to simultaneously visualize several body tissues (bones,
blood vessels, nerves, soft tissues) in order to increase the objectivity of determining the
boundaries of the resection, optimizing the choice of a vascularized graft and determining
donor and recipient vessels. All the above tasks correspond to an oine “map” of the
patient, used in the developed augmented reality program, which is compiled on the basis
of CT or MRI data and can be supplemented by ultrasound, dopplerography, angiography,
etc.
Augmented Reality (AR) is a combination of virtual reality and with the “oine map” of
a patient and allows the user to interact in real time between various material and virtual
objects, as if they exist in a single environment (Bartella et al., 2019). Using this feature,
we can increase the productivity and quality of services in many areas, especially in head
and neck surgery. The proposed solution will help the doctor easily see where the internal
anatomical structures are located during the operation. We oer an advanced solution for
tracking optical markers with spatial imaging based on patient CT data. The surgeon during
the operation through glasses sees a 3D visualization of the received data, due to which he
less looks at the monitor with the data of diagnostic tests and constantly monitors the state
of the surgical wound, which reduces the risk of intraoperative complications, including
bleeding, which leads to an increase in the accuracy of surgery, reducing its duration and
reducing the risk of complications. Also, this project can be widely used for educational
purposes.
Currently, many scientic papers on the use of augmented reality in various elds of
medicine have been published: neurosurgery, cardiology, urology, plastic surgery, including
dentistry and maxillofacial surgery (Ayoub & Pulijala, 2019; Elmi-Terander et al., 2020; Kim,
Kim, & Kim, 2017; Kwon, Park, & Han, 2018). However, most of them are descriptions
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
24
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
of individual clinical cases or their series, which certainly does not have a very high class
of evidence (Bosc et al., 2019). Available literature reviews also note the lack of clinical trial
results for a more objective assessment of this method. Therefore, the goal of our work is
to conduct a multicenter randomized clinical trial in the treatment of patients with jaw
cancer, which shows their resection with simultaneous plastic surgery of the defect with a
vascularized bone graft with microtechnology. A feature of basic research is their careful
planning, strict adherence to the research protocol, good statistical analysis of the data
obtained, as well as a unied scale for assessing the data obtained (calibrated investigator)
and their analysis by an uninvolved specialist, to reduce the likelihood of bias on the part
of the scientist when interpreting the data, compliance All of the above conditions will
allow you to perform a high-class evidence study, the results of which will be useful for the
development of modern science.
On the basis of the Department of Maxillofacial and Reconstructive Surgery of the First
St. Petersburg Medical University named after Acad. I.P. Pavlova reconstructive plastic
surgery to eliminate jaw defects due to malignant and benign neoplasms, post-traumatic
defects and deformations of various genesis with the help of vascularized bone and soft-
tissue aps with microsurgical technique has been carried out for more than 30 years by a
team led by professor N.V. Kalakutsky During this time, vast clinical experience has been
accumulated in the treatment of this group of patients, which will facilitate the introduction
of new technologies and improve the quality of the study.
According to the example of the following clinical case of using augmented reality
technology in dentistry, within the framework of the planned study, it is supposed to be
used when performing a jaw resection for cancer with simultaneous plastic surgery of the
defect with a bone graft with microtechnology.
2. METHODOLOGY
2.1. MEDICAL CASE
For the rst case, we took a patient who has a small hollow cluster of cells that is grouped
together - a cyst in the left nasal sinus (Figure 1), which was recommended for removal.
It also has a complex and uneven shape. Therefore, to help the doctor better understand
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
25
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
where he is during planning and work, we have developed a visual solution with the display
of anatomical structures on top of what the doctor sees.
Figure 1. Cyst in left sinus marked in red.
2.2. OVERLAY 3D MODELS OF THE SKULL IN THE REAL WORLD
For the selected task, Augmented Reality is the optimal visual solution.
Augmented Reality (AR) presents information in the right context of the real world. To
do this, the system must know where the user is and what he is looking at. Usually the user
observes an object through a display that displays a camera image along with augmented
reality. Thus, in practice, the system needs to determine the location and orientation of the
camera relative to the object. In the case of Hololens glasses, it is almost the same. To do
this, use a separate built-in camera, and as a display - two holographic projectors.
In AR technology, tracking markers are used for display on the display and in the camera -
AR codes that help align the 3d model from the user’s point of view. In our problem, when
an object should be viewed from dierent sides, we used a special marker called cuboid,
where on each side there is a unique AR code. The Hololens camera has a limited resolution
of 720p and a low dynamic range compared to modern cameras. This causes problems
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
26
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
with accurate tracking. To improve this, we have added the wrong organic structure, which
provides additional tracking capabilities.
Figure 2. Type of marker.
In addition to the problem of aligning the object with the camera, our case is complicated
by the fact that the marker must be xed on the person’s head, strictly in a certain position
and with a certain orientation. For this task, we built a 3D model of the head from CT data
using the Materialize Mimics software. Then we designated the areas where the marker will
align and x. Based on the selected areas, a three-dimensional wireframe model was built
with a platform for attaching a marker. The frame was made in such a way that it could
be easily put on and xed in the correct position (Figure 3). Finally, it was printed on a 3D
printer (Figure 4), and the marker was attached at a predetermined location.
Figure 3. Left image - A constructed model of the face and skull based on CT data; in the middle are designated
areas for xing the frame, on the right is the model of the frame with a zone for the marker on top.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
27
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
Figure 4. Printed frame.
After that, using the same CT dataset, we extracted a model of the patient’s skull and cysts
using Mimics. In the case of a cyst, it was isolated by CT by the doctor in layers, because
such structures are clearly not readable on CT, and the program cannot automatically
detect them. These two models were exported to Unity for further development under
Hololens glasses (Figure 5).
Figure 5. 3D model of a skull and cyst for augmented reality. The cube determines the size and position of the
marker.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
28
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
3. RESULTS
3.1. APPLICATION DEVELOPMENT FOR HOLOLENS
We developed the application using the Unity environment, which has special tools for
HoloLens. There are also additional modules for Unity that expand the capabilities. One
of them is Vuforia, designed for augmented reality using marker tracking.
In addition, a simple user interface was added using a gesture that the glasses recognize
- a click to hide the skull and a click and drag to change the opacity when viewing the
hologram.
To improve tracking performance, we also used a feature called spatial mapping, which is
available in HoloLens. Spatial mapping provides additional information about the position
and orientation of glasses in the environment with high accuracy.
Figure 6. View of the skull and cyst through the HoloLens glasses.
Figure 7. View of an isolated cyst through HoloLens glasses.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
29
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
4. CONCLUSIONS
The proposed approach using holographic representation of anatomical structures can
be potentially used in surgery planning and during a procedure with some restrictions.
Currently, marker based solutions on HoloLens devices have few limitations, such as low
resolution cameras that are sensitive to lighting conditions and insucient computing
power of glasses. This causes a delay between 100-500 milliseconds and produces an
error threshold in hologram positioning between 1-5mm. Such problems can be solved by
upgrading hardware in the future glasses releases and possibly by switching to a geometry
tracking solution, where markers can be represented by geometrical objects with any shape.
REFERENCES
Ayoub, A., & Pulijala, Y. (2019). The application of virtual reality and augmented
reality in Oral & Maxillofacial Surgery. BMC Oral Health, 19(1), 238. https://doi.
org/10.1186/s12903-019-0937-8
Bartella, A. K., Kamal, M., Scholl, I., Schier, S., Steegmann, J., Ketelsen,
D., Hölzle, F, & Lethaus, B. (2019). Virtual reality in preoperative imaging in
maxillofacial surgery: implementation of “the next level”? British Journal of Oral and
Maxillofacial Surgery, 57(7), 644-648. https://doi.org/10.1016/j.bjoms.2019.02.014
Bosc, R., Fitoussi, A., Hersant, B., Dao, T. H., & Meningaud, J. P. (2019).
Intraoperative augmented reality with heads-up displays in maxillofacial surgery: a
systematic review of the literature and a classication of relevant technologies. Int
J Oral Maxillofac Surg, 48(1), 132-139. https://doi.org/10.1016/j.ijom.2018.09.010
Elmi-Terander, A., Burström, G., Nachabé, R., Fagerlund, M., Ståhl, F.,
Charalampidis, A., Edström, E., & Gerdhem, P. (2020). Augmented reality
navigation with intraoperative 3D imaging vs uoroscopy-assisted free-hand surgery
for spine xation surgery: a matched-control study comparing accuracy. Scientic
Reports, 10(1), 707. https://doi.org/10.1038/s41598-020-57693-5
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
30
https://doi.org/10.17993/3ctecno.2020.specialissue6.21-31
Hartman, E. H., Spauwen, P. H., & Jansen, J. A. (2002). Donor-site complications in
vascularized bone ap surgery. Journal of Investigative Surgery, 15(4), 185-197. https://
doi.org/10.1080/08941930290085967
Kim, Y., Kim, H., & Kim, Y. O. (2017). Virtual Reality and Augmented Reality in Plastic
Surgery: A Review. Archives of Plastic Surgery, 44(3), 179-187. https://doi.org/10.5999/
aps.2017.44.3.179
Kwon, H. B., Park, Y. S., & Han, J. S. (2018). Augmented reality in dentistry: a current
perspective. Acta Odontologica Scandinavica, 76(7), 497-503. https://doi.org/10.1080/0
0016357.2018.1441437
