IMPLEMENTATION OF HAND GESTURE-
CONTROLLED MOUSE USIN
G ARTIFICIAL
INTELLIGENCE
Piyush Pilare
B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of
Engineering and Management Nagpur, (India).
Coral Mahato
B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of
Engineering and Management Nagpur, (India).
Chanchal Khergade
B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of
Engineering and Management Nagpur, (India).
Shubham Agrawal
B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of
Engineering and Management Nagpur, (India).
Prasheel Thakre
Assistant Professor, Department of Electronics and Communication Shri Ramdeobaba College of
Engineering and Management Nagpur, (India).
E-mail: thakrepn2@rknec.edu
Reception: 16/09/2022 Acceptance: 01/10/2022 Publication: 29/12/2022
Suggested citation:
Pilare, P., Mahato, C., Khergade, C., Agrawal, S., and Thakre, P. (2022). Implementation of hand gesture-
controlled mouse using artificial intelligence. 3C Tecnología. Glosas de innovación aplicadas a la pyme, 11(2),
71-79. https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79
https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143
Ed. 42 Vol. 11 N.º 2 August - December 2022
71
ABSTRACT
This article presents a proposed mouse system. In this paper, we discussed the implementation of an
artificially intelligent hand gesture-controlled mouse that uses computer vision to execute mouse
functions using the Colour detection technique. The virtual mouse uses the current python and
computer vision algorithms for the recognition of the Masked/colored region and works seamlessly
without any extra hardware requirements. A computer may be controlled remotely using hand motions,
and it is capable to perform cursor movement, left-clicking, and right-clicking without the need for a
hardware mouse.
KEYWORDS
Hand Gestures, mouse, Image capture, Preprocessing, Masking.
https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143
Ed. 42 Vol. 11 N.º 2 August - December 2022
72
1. INTRODUCTION
AI hand control technology is a brand-new development that allows people with limited hand usage to
control their computers via hand movement recognition. It works by tracking the movements of a
person’s hand with a webcam or inbuilt camera. The software recognizes the position of the person’s
hand in space and replays that movement on the screen as a mouse cursor movement. The result is that
users can control their cursor with simple movements of their hands. Many of the current iterations of
AI hand control technology have been created for people with conditions like arthritis, carpal tunnel
syndrome, or paralysis Ghute et al. [2018]. Such disorders result in limited hand use and often make it
impossible for people to use a computer mouse. AI hand-control technology allows such individuals to
regain computer control, opening up the world of online information and entertainment once more to
people who have been cut off from it. The hand gesture-controlled mouse would surely lead to
massive development in the field of technology Pradhan et al. [2015].
1.1. EXISTING SYSTEM
All I/O operations are controlled virtually with the assistance of masking. This implemented system
aims to regulate a mouse cursor using the fingertips of the individual. A custom locking algorithm is
employed to convert masked region coordinates from a virtual screen into a full-screen one to be
accustomed to controlling the mouse Mali et al. [2022]. The suggested mouse system captures images
using either a built-in camera or a webcam while evaluating the quality of the camera.
The virtual mouse framework can also get aware of problems with hitting in places like things that
don't have room to accommodate an actual mouse and be tailored for those who have issues in
gripping and can't handle an actual mouse. Assuming that if something is finished with a mouse, it
may also be through with your daily webcam. An electronic device could be a handheld pointing
device most ordinarily used for manipulating objects on the pc screen.
1.2. PROPOSED SYSTEM
The virtual space between the web camera and the user is described as the "Virtual Monitor," where a
mouse cursor can be moved using a masked object and background subtraction. Grif H. T. et al.
[2018]. We have mainly concentrated on the movement of the pointer and basic operation of the
mouse like dragging and left-clicking Titlee et al. [2017]. The Python module called OpenCV is a
library of programming functions, mostly focused on computer vision in real time. It uses a sampling
streaming endpoint to listen to new tweets in real-time and draws them onto a virtual globe according
to whatever positional information is included Grif H. T. et al. [2018] Prof. Shital Pawar et al. [2022].
It could be modified to a further extent by introducing some new features and adding hand gestures.
Instead of using masking, we can control the mouse with the help of just your hand.
https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143
Ed. 42 Vol. 11 N.º 2 August - December 2022
73
1.3. USE OF PROPOSED WORK
In this system, we are controlling the mouse pointer by using colored tape or caps on the
fingertips and with the open and Close gestures we are operating the mouse, we are using
inbuilt libraries in Python to get the pointer coordinates and match them with the masked
region from the camera to screen resolution.
2. METHOD
The methodology of each system component will be covered separately. These are the
different sections:
2.1. CAMERA SETUP
The web-cam manages the mouse operations. We need to use a Video Capture object to
capture a video. After that, we can capture frame-by-frame. We could apply color detection
techniques to any image by making minor algorithm changes.
2.2. PREPROCESSING THE FRAMES
The web camera is continually active and captures the video during the program's duration
thanks to an infinite loop. Next, each frame that was gathered in RGB (the default) color
system is converted to HSV color format. Grif et al. [2016].
2.3. MASKING TECHNIQUE
We are producing a certain region with the help of a mask by the use of the original image
using the threshold image and applying AND operation when we do that the masked region is
then highlighted and we get the region we want and we can propose some rules to that area
Chienet al. [2015] Grif et al. [2015].
2.4. OPERATION
The detected coordinate is then used to convert camera resolution to actual screen resolution
Wen et al. [2012]. The position of the mouse is then set, however, it will take some time to
shift the mouse pointer. Once the current mouse position and the designated mouse location
are the same, we started a loop.
https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143
Ed. 42 Vol. 11 N.º 2 August - December 2022
74
Fig 1. FlowChart of proposed Method.
2.5. CLICKING
In this instance, dragging and clicking the object is used to complete the action. It is
comparable to making an open gesture, but since there is only one item, we simply need to
determine its center. And we'll position that there where our mouse pointer will be.
2.6. DRAG
We introduce a variable called "flag" to implement dragging. It will be set to 1 if it was clicked
in the past. So, after finding the open gesture, we click it and then double-check that the flag is
set to 1. If it is set to one, the drag action is performed; if not, the mouse move operation is
performed.
3. RESULT AND EVALUATION
Many of the current iterations of AI hand control technology have been created for people with
conditions like arthritis, carpal tunnel syndrome, or paralysis. This implemented system aims
to regulate a mouse cursor using the fingertips of the individual. AI hand control technology is
a brand-new development that allows people with limited hand usage to control their
computers via hand movement recognition. A custom locking algorithm is employed to
convert masked region coordinates from a virtual screen into a full-screen one to be
accustomed to controlling the mouse Suriya et al. [2014]. The software recognizes the position
of the person’s hand in space and replays that movement on the screen as a mouse cursor
movement. The suggested mouse system captures images using either a webcam or a built-in
camera while considering the camera quality, allowing us to reduce the number of system
components. This creates a greater impact on the Environment by stopping the E-waste
generation from keyboard and mouse and their wires.
https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79
3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143
Ed. 42 Vol. 11 N.º 2 August - December 2022
75