3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
79
AUGMENTED REALITY BASED GESTURE DETECTION &
OBJECT CREATION SYSTEM USING XCODE & ARKIT
Sallar Khan
Sir Syed University of Engineering and Technology, Karachi, (Pakistan).
E-mail: Sallarkhan_92@yahoo.com ORCID: https://orcid.org/0000-0001-8988-3388
Syed Abbas Ali
N.E.D University of Engineering and Technology, Karachi, (Pakistan).
E-mail: saaj.research@gmail.com ORCID: https://orcid.org/0000-0001-6014-1559
Muhammad Nadeem
Sir Syed University of Engineering and Technology, Karachi, (Pakistan).
E-mail: mnadeem79@gmail.com ORCID: https://orcid.org/0000-0002-9271-5008
Raj Chawla
Sir Syed University of Engineering and Technology, Karachi, (Pakistan).
E-mail: rajchawlas72@gmail.com ORCID: https://orcid.org/0000-0003-1289-3849
Recepción:
04/09/2020
Aceptación:
29/09/2020
Publicación:
13/11/2020
Citación sugerida Suggested citation
Khan, S., Abbas, S., y Chawla, R. (2020). Augmented reality based gesture detection & object creation
system using XCode & ARKit. 3C Tecnología. Glosas de innovación aplicadas a la pyme. Edición Especial,
Noviembre 2020, 79-91. https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
80
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
ABSTRACT
In this modern era of mobile applications, Augmented Reality (AR) is becoming one of
the emerging areas of implementation for researchers around the globe. While live gesture
detection and 3D model creation still need more attention from researchers. In this paper, we
present an interactive AR character that directly interacts with real objects. The interactive
AR character automatically determines how to behave and to control real objects. This
current research presents three studies that test the social psychological eects of Augmented
Reality. In this research, we are using Apple’s IDE for native iOS development on swift,
XCode for UI design and architectural functionalities of our application. We used the
ARKIT library to import all the necessary functions and classes to manipulate and use
according to our needs. Finally, we successfully deployed an IOS application which can
detect live gestures of our hand movements and then create 3D models with the help of
our hand gestures.
KEYWORDS
Augmented Reality, Hand Gestures, 3D Model Creation, IOS Application, Apple, XCode,
ARKit.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
81
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
1. INTRODUCTION
Augmented Reality (AR) is a grand vision where the digital domain blends with the physical
world. Information not only follows a person, but also her very gaze: looking at an object
is enough to retrieve and display relevant information, amplifying her intelligence. Though
research on AR has advanced over the last several decades, AR technology has yet to reach
the mass-market. The minimum requirement for AR is a display, a camera for tracking, and
a processing unit. These are also the components of camera phones, predicted to account
for more than 80 percent of total worldwide mobile phone sales by 2010.
The average person learns better by observing and listening something than by simply
reading something. We will be using this specic property of the human mind to accelerate
learning. Although there are thousands of videos on the internet about almost every
eld of life but they’re mostly from the perspective of video makers. By combining AR
with learning and practicing we will provide immersive learning experience to our users
in a way that the information settles in the long-term memory, with control over their
interaction with the object they experiment as the way they want it. AR being the in the
top 5 technologies of 2019 is momentous for the upcoming future as users incline towards
augmented virtual experiences. With AR users can experience and interact with things that
would be physically inaccessible or inconvenient otherwise. Our product will cater to users
from students to professionals belonging to diversied elds of life and practice. The ability
to touch and interact with augmented models in real time space is a new to the industry of
Augmented Reality and our app is one of a kind with its ability to utilize real world spaces
and objects to detect and write with our bare hands or using any object, from your ordinary
pen, to any object with a pointed end. Users can write and then save their work to resume
or view later.
2. RELATED WORK
This research describes experiences with development of an AR application for
augmentation of an industrial robot. The paper focuses on the description of the application
requirements and prototype implementation (Löfvendahl, 2014). Another application is
Robotic Modeling Assistant (RoMA), an interactive fabrication system providing a fast,
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
82
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
precise, hands-on and in-situ modeling experience. As a designer creates a new model using
RoMA AR CAD editor, features are constructed concurrently by a 3D printing robotic arm
sharing the same design volume.
The partially printed physical model then serves as a tangible reference for the designer as
she adds new elements to her design. RoMA’s proxemics-inspired handshake mechanism
between the designer and the 3D printing robotic arm allows the designer to quickly
interrupt printing to access a printed area or to indicate that the robot can take full control
of the model to nish printing. RoMA lets users integrate real-world constraints into a
design rapidly, allowing them to create well-proportioned tangible artifacts or to extend
existing objects. We conclude by presenting the strengths and limitations of our current
design (Peng et al., 2018).
An app that is based on usage of AR in historical recreation as well as it elaborates previous
work that has been done in this area and gives knowledge about technologies that allows
creating application for AR oriented historical site. This app will showcase various ancient
artifacts and heritage sites across the world showing how important history is how we can
learn from the immaculate and genius architecture and solutions of the past that those
people came up with much lesser resources (Desai, 2018).
An Augmented Reality application for mobile devices that promoted and supports the
learning of geometric gures. The application is named AGeRA, consists of a geometry
book and software capable of reading special markers inserted into the book’s content.
When this book is placed in front of the camera of a mobile device, 3D objects, sounds,
animations, and other interactive elements leap from book pages making learning more
immersive and interesting. Tests were made with teachers and students and showed good
acceptance of the application to support the teaching of geometry (Neto, 2013).
An android based app has been developed in Indonesia help the Indonesian students to
learn Batik. Pre-test and post-test are administered to check whether the application is
improving the spatial intelligence of the students. This android based application was based
on Augmented Reality Batik Ikonik (ARtikon) Joyful. First registration to the app is required
by logging in with matching usernames and password. After successfully logging in the
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
83
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
camera will be activated. All the 2D and the 3D objects will reect in the Batik patterns by
this app while the object will remain unidentied if the object is in 2D (Widiaty et al., 2017).
KUFSGT a school of Global tourism at Kyoto University of Foreign studies collaborated
with MAVR, an immersive learning group in Japan, for augmented learning environments.
With the help of AReiantation, application which simplies the workow of AR contents
so in seconds the idea can be prototyped. On the open campus they created an area for the
activities they call ARVR experience zone. These activities included KUFSGO and MY
HOMETOWN PROJECT (Hawkinson, 2019). An application was developed in India that
scans the menu of any restaurant to provide ratings to each dish on the menu by the help of
OCR. These rating were given by the sentiment analysis over the reviews provided by the
customers. For a better experience, these menus have an AR system that projects the rating
for the dishes (Wang, Chen, & Lang, 2015).
A researcher developed an innovation by the implementation of Augmented Reality
technology in natural sciences learning of elementary school. The method that was used
in this research was the method based on the Ball and George theory, which contained
ten stages: Research and information collecting, Planning, Develop preliminary form of
product, Preliminary eld testing, Main product revision, Main eld testing, Operational
product revision, Operational eld testing, Final product version and Dissemination and
implementation (Fakhrudin, 2018).
A research was conducted; the purpose of this research was to calculate the impact of
Augmented Reality mobile applications on the learning motivation of health science
students of University of Cape Town. The ACRS (attention, relevance, condence and
satisfaction) model measured the impact of Augmented Reality on a student’s motivation,
and the Instructional Materials Motivation Survey guided the way to design the research
instrument. A total of 78 participants used the Augmented Reality mobile application,
and the result showed that satisfaction, attention and condence factors were signicantly
increased. However, the decrease in relevance factor was proved to be insignicant (Khan
et al., 2019).
A new software system is developed by Brown University researchers that turns cell phones
into Augmented Reality portals, enabling users to place virtual building blocks, furniture
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
84
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
and other objects into real-world backdrops, and use their hands to manipulate those objects
as if they were here. The developers are hoping the new system that can be a tool for artists,
designers, game developers and others to experiment with Augmented Reality (AR). The
team will present at the ACM Symposium on User Interface Software and Technology
(UIST 2019) in New Orleans. The source code for Android is freely available for download
on the researchers’ website, and iPhone code will follow soon (Brown University, 2019).
Physics education applications AR, which has been developed extensively in recent years,
have a lot of restrictions in terms of accuracy and performance. The purpose of our research
is to develop a real-time simulation system for physics education that is based on parallel
processing. In this research, we present a video see-through AR (Augmented Reality) system
that includes an environment recognizer using a depth image of Microsoft’s Kinect V2 and
a real-time soft body simulator based on parallel processing using GPU (Graphic Processing
Unit). Soft body simulation can provide more realistic simulation results than rigid body
simulation, so it can be more eective in systems for physics education. We have designed
and implemented a system that provides the physical deformation and movement of 3D
volumetric objects and uses them in education. We plan to use the stand-alone AR device
including one or more cameras to improve the system in the future (Sung et al., 2019).
Mobile devices are becoming a common target for Augmented Reality applications,
especially for showing contextual information in buildings or construction sites. A
prerequisite of contextual information display is the localization of objects and the device
in the real world. In this research, we will present our approach to the problems of mobile
indoor localization with a building model. The approach does not use external sensors
or input. Accurate external sensors such as stationary cameras are expensive and dicult
to set up and maintain. Relying on already existing external sources may also prove to be
dicult, as especially inside buildings, Internet connections can be unreliable and GPS
signals can be inaccurate. Therefore, we try to nd a localization solution for Augmented
Reality devices that can accurately localize itself only with data from internal sensors
and preexisting information about the building. If a building has an accurate model of
its geometry, we can use modern spatial mapping techniques and point-cloud matching
to nd a mapping between local device coordinates and global model coordinates. We
use normal analysis and 2D template matching on an inverse distance map to determine
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
85
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
this mapping. The proposed algorithm is designed to have a high speed and eciency, as
mobile devices are constrained by hardware limitations. We show an implementation of the
algorithm on the Microsoft HoloLens, test the localization accuracy, and or use cases for the
technology (Herbers & König, 2019).
Industrial Augmented Reality (AR) applications demand high on the visual consistency
of virtual-real registration. To present, the marker-based registration method is most
popular because it is fast, persistent, and convenient to obtain the registration matrix.
The registration matrix should multiply an o set matrix that describes the transformation
between the attaching position and the initial position of the marker relative to the object.
However, the o set matrix is usually measured, calculated, and set manually, which is not
accurate and convenient. This paper proposes an accurate and automatic marker object of
set matrix calibration method. First, the normal direction of the target object is obtained by
searching and matching the top surface of the CAD model. Then, the spatial translation is
estimated by aligning the projected and the imaged top surface. Finally, all six parameters
of the o set matrix are iteratively optimized using a 3D image alignment framework.
Experiments were performed on the publicity monocular rigid 3D tracking dataset and an
automobile gearbox.
The average translation and rotation errors of the optimized o set matrix are 2.10 mm and
1.56 degree respectively. The results validate that the proposed method is accurate and
automatic, which contributes to a universal o set matrix calibration tool for marker-based
industrial AR applications (Yin et al., 2019).
This current research presents three studies that test the social psychological eects of
Augmented Reality.
Study 1 examined participants’ task performance in the presence of embodied agents and
replicated the typical pattern of social facilitation and inhibition. Participants performed
a simple task better, but a hard task worse, in the presence of an agent compared to when
participants complete the tasks alone. Study 2 examined nonverbal behavior. Participants
meet an agent. Participants wearing the headset never sat directly on the agent when
given the choice of two seats, and while approaching, most of the participants chose the
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
86
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
rotation direction to avoid turning their heads away from the agent. A separate group of
participants chose a seat after removing the Augmented Reality headset, and the majority
still avoided the seat previously occupied by the agent. Study 3 examined the social costs of
using an Augmented Reality headset with others who are not using a headset. Participants
talked in dyads, and Augmented Reality users reported less social connection to their
partner compared to those not using Augmented Reality. Overall, these studies provide
evidence suggesting that task performance, nonverbal behavior, and social connectedness
are signicantly aected by the presence or absence of virtual content(Miller et al., 2019).
In the research eld of Augmented Reality (AR), applications using interactive characters
have been developed as the form of giving users information such as LEGO assembly
guidance and explanation about historical artifacts. Even though these characters respond
to interaction with users, they could not create substantial eects or changes in a real space.
Therefore, this limitation makes users reduce their coexistence with the AR characters.
In this paper, we present an interactive AR character that directly interacts with real
objects. The interactive AR character automatically determines how to behave and to
control these objects. At rst, we make working space populated by AR characters that has
a real object with which the AR character can interact. As an interactive AR character, we
implement ARMate, which presents realistic responses according to changes of real objects
manipulated by a user in real time. We develop ToyCart as a physical object that includes
hardware devices for movement, and ARMate can control ToyCart. Finally, we expect that
our AR character can increase coexistence through real object-based interaction (Kang &
Woo, 2011).
3. METHODOLOGY
3.1. XCODE
Using apple’s IDE for native iOS development on swift, we used XCode to design the UI
and architectural functionalities of our application. We used the ARKIT library to import
all the necessary functions and classes to manipulate and use according to our needs. We
then designed the UI of the application based on the Apple’s modern iOS design language.
The progression of development of our app was divided into 2 phases, rst being the UI/
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
87
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
UX of the application, and second being the main programming and functionality of the
application.
As the application is based on iOS and developed on XCode, we used the following libraries
to construct our application.
3.2. ARKIT
Apple’s AR Kit is a powerful library and tool that enables us to use already written code
according to our functionality needs and modify it accordingly. Using the camera on
iPhone or iPad it can detect live people in the real environment and track their motions
and movements. We used this feature to track the hand or object movement that will enable
us to draw and write in real time using the front or back camera of the device using our
hands or objects. AR Kit also enabled us to make anything 3D after drawing it in 2D on
any surface using its animation tools. ARKit is based on two main features, the camera in
a location for a 3D space and the second main feature is the detection of horizontal plane.
To make it possible the camera of your phone assumes that it is in an actual 3D space
and placing some 3D object will pinpoint in real 3D space. Then he ARKit detects the
horizontal surface and places the object on top of it. So how the ARKit can perform this?
This is done through a simple technique known as Visual Inertial Odometry (VIO). In 3D
space, to track the location of the device with the help of motion sensors fused with camera
frames, this technique is known as VIO. By detecting the edge points in the image with high
contrast we can track the motion of the camera frames.
It estimates where the device is in the 3D space, by detecting how much of these points
relatively moved to each other from one frame to another frame. This is the sole reason why
ARKit couldn’t work with a white wall in the background or when the device is moving
too fast to detect the 3D space which causes to create blurred images. We can create a new
ARKit project from New > Project > Augmented Reality App. It is more accessible to begin
the AR tutorial with the ocial Apple ARKit sample, which make plane detection more
feasible.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
88
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
4. RESULTS
By successfully integrating everything together, this is the result of our AR application.
Firstly as seen below the yellow box detects your ngers or any other pointing object and
then starts tracking it:
Figure 1. Tracking process of hand and its gestures.
After successfully track locking the nger/object, a red dot appears which will act as a pen
to draw on the surface below:
Figure 2. Implementation of drawing with the help of hand gestures.
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
89
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
Result of this can be seen below as after successfully tracking and drawing:
Figure 3. The desired 3D model is drawn successfully.
5. CONCLUSION
AR is one of the fastest future technologies that will soon become a huge part of our
everyday lives, and with huge conglomerates adopting it such as Google, Apple, Facebook,
it will soon be a must with everything.
Keeping that in mind, and the fact that immersive learning is so much of a better experience
than the more traditional forms of learning such as text and video, it is bound to get
attention and appreciation for the awe factor it cases. Technology has come a long way and
not to use it to its full potential would be a waste of all the hard work put in by scientists and
engineers to bring that innovation to the table of everyday common man so he can rejoice
life and maybe one step at a time make it easier and convenient for him.
REFERENCES
Brown University. (2019). New augmented reality system lets smartphone users get
hands-on with virtual objects. ScienceDaily. https://www.sciencedaily.com/
releases/2019/10/191016153703.htm
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
90
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
Desai, N. (2018). Recreation of history using augmented reality. ACCENTS Transactions
on Image Processing and Computer Vision, 4(10), 1–5. https://doi.org/10.19101/
tipcv.2017.39019
Fakhrudin, A. (2018). The Implementation of Augmented Reality Technology in Teaching
Natural Sciences to Improve Elementary Students’ Learning Achievement. Al-Ta
Lim Journal, 25(1), 13. https://doi.org/10.15548/jt.v25i1.374
Hawkinson, E. (2019). Student Designs in Augmented Tourism. https://medium.com/ready-
teacher-one/student-designs-in-augmented-tourism-bc168bc5f7f
Herbers, P., & König, M. (2019). Indoor localization for augmented reality devices using
BIM, point clouds, and template matching. Applied Sciences (Switzerland), 9(20). https://
doi.org/10.3390/app9204260
Kang, C., & Woo, W. (2011). ARMate: An Interactive AR Character Responding to Real
Objects. In Chang M., Hwang WY., Chen MP., Müller W. (eds) Edutainment Technologies.
Educational Games and Virtual Reality/Augmented Reality Applications. Edutainment 2011.
Lecture Notes in Computer Science, vol. 6872. Springer, Berlin, Heidelberg. https://
doi.org/10.1007/978-3-642-23456-9_3
Khan, T., Johnston, K., & Opho, J. (2019). The Impact of an Augmented Reality
Application on Learning Motivation of Students. Advances in Human-Computer
Interaction, Article ID 7208494. https://doi.org/10.1155/2019/7208494
Löfvendahl, B. (2014). Augmented Reality Applications for Industrial Robots. Umeå University.
Department of Applied Physics and Electronics. https://www.diva-portal.org/
smash/get/diva2:706980/FULLTEXT01.pdf
Miller, M. R., Jun, H., Herrera, F., Villa, J. Y., Welch, G., & Bailenson, J. N.
(2019). Social interaction in augmented reality. PLoS ONE, 14(5), 1–27. https://doi.
org/10.1371/journal.pone.0216290
Neto, F. M. M. (2013). Technology platform innovations and forthcoming trends in
ubiquitous learning. In Technology Platform Innovations and Forthcoming Trends in Ubiquitous
Learning. https://doi.org/10.4018/978-1-4666-4542-4
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020
91
https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91
Peng, H., Briggs, J., Wang, C. Y., Guo, K., Kider, J., Mueller, S., Baudisch, P.,
& Guimbretière, F. (2018). Roma: Interactive fabrication with augmented reality
and a Robotic 3D printer. Conference on Human Factors in Computing Systems - Proceedings,
2018(April). https://doi.org/10.1145/3173574.3174153
Sung, N. J., Ma, J., Choi, Y. J., & Hong, M. (2019). Real-time augmented reality
physics simulator for education. Applied Sciences (Switzerland), 9(19), 2019. https://doi.
org/10.3390/app9194019
Wang, C. Y., Chen, Y., & Lang, Z. (2015). Restaurant Menu Expert A digital image
processing pipeline to increase the accuracy of the state-ofthe-art OCR algorithm.
https://web.stanford.edu/class/ee368/Project_Autumn_1516/Reports/Wang_
Chen_Lang.pdf
Widiaty, I., Riza, L. S., Danuwijaya, A. A., Hurriyati, R., & Mubaroq, S. R. (2017).
Mobile-based augmented reality for learning 3-dimensional spatial Batik-based objects.
Journal of Engineering Science and Technology, 12(Special Issue 10), 12–22. https://www.
researchgate.net/publication/336603527_EDUCATIONAL_DIGITAL_MEDIA_
FOR_TRADITIONAL_FOOD_OF_KAMPUNG_ADAT_CIREUNDEU_AN_
ETHNOPEDAGOGY_PERSPECTIVE
Yin, X., Fan, X., Yang, X., Qiu, S., & Zhang, Z. (2019). An automatic marker-
object oset calibration method for precise 3D augmented reality registration in
industrial applications. Applied Sciences (Switzerland), 9(20). https://doi.org/10.3390/
app9204464
