IMPLEMENTATION OF HAND GESTURE-

CONTROLLED MOUSE USIN

G ARTIFICIAL

INTELLIGENCE

Piyush Pilare

B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of

Engineering and Management Nagpur, (India).

Coral Mahato

B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of

Engineering and Management Nagpur, (India).

Chanchal Khergade

B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of

Engineering and Management Nagpur, (India).

Shubham Agrawal

B.Tech Student, Department of Electronics and Communication Shri Ramdeobaba College of

Engineering and Management Nagpur, (India).

Prasheel Thakre

Assistant Professor, Department of Electronics and Communication Shri Ramdeobaba College of

Engineering and Management Nagpur, (India).

E-mail: thakrepn2@rknec.edu

Reception: 16/09/2022 Acceptance: 01/10/2022 Publication: 29/12/2022

Suggested citation:

Pilare, P., Mahato, C., Khergade, C., Agrawal, S., and Thakre, P. (2022). Implementation of hand gesture-

controlled mouse using artificial intelligence. 3C Tecnología. Glosas de innovación aplicadas a la pyme, 11(2),

71-79. https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

ABSTRACT

This article presents a proposed mouse system. In this paper, we discussed the implementation of an

artificially intelligent hand gesture-controlled mouse that uses computer vision to execute mouse

functions using the Colour detection technique. The virtual mouse uses the current python and

computer vision algorithms for the recognition of the Masked/colored region and works seamlessly

without any extra hardware requirements. A computer may be controlled remotely using hand motions,

and it is capable to perform cursor movement, left-clicking, and right-clicking without the need for a

hardware mouse.

KEYWORDS

Hand Gestures, mouse, Image capture, Preprocessing, Masking.

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

1. INTRODUCTION

AI hand control technology is a brand-new development that allows people with limited hand usage to

control their computers via hand movement recognition. It works by tracking the movements of a

person’s hand with a webcam or inbuilt camera. The software recognizes the position of the person’s

hand in space and replays that movement on the screen as a mouse cursor movement. The result is that

users can control their cursor with simple movements of their hands. Many of the current iterations of

AI hand control technology have been created for people with conditions like arthritis, carpal tunnel

syndrome, or paralysis Ghute et al. [2018]. Such disorders result in limited hand use and often make it

impossible for people to use a computer mouse. AI hand-control technology allows such individuals to

regain computer control, opening up the world of online information and entertainment once more to

people who have been cut off from it. The hand gesture-controlled mouse would surely lead to

massive development in the field of technology Pradhan et al. [2015].

1.1. EXISTING SYSTEM

All I/O operations are controlled virtually with the assistance of masking. This implemented system

aims to regulate a mouse cursor using the fingertips of the individual. A custom locking algorithm is

employed to convert masked region coordinates from a virtual screen into a full-screen one to be

accustomed to controlling the mouse Mali et al. [2022]. The suggested mouse system captures images

using either a built-in camera or a webcam while evaluating the quality of the camera.

The virtual mouse framework can also get aware of problems with hitting in places like things that

don't have room to accommodate an actual mouse and be tailored for those who have issues in

gripping and can't handle an actual mouse. Assuming that if something is finished with a mouse, it

may also be through with your daily webcam. An electronic device could be a handheld pointing

device most ordinarily used for manipulating objects on the pc screen.

1.2. PROPOSED SYSTEM

The virtual space between the web camera and the user is described as the "Virtual Monitor," where a

mouse cursor can be moved using a masked object and background subtraction. Grif H. T. et al.

[2018]. We have mainly concentrated on the movement of the pointer and basic operation of the

mouse like dragging and left-clicking Titlee et al. [2017]. The Python module called OpenCV is a

library of programming functions, mostly focused on computer vision in real time. It uses a sampling

streaming endpoint to listen to new tweets in real-time and draws them onto a virtual globe according

to whatever positional information is included Grif H. T. et al. [2018] Prof. Shital Pawar et al. [2022].

It could be modified to a further extent by introducing some new features and adding hand gestures.

Instead of using masking, we can control the mouse with the help of just your hand.

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

1.3. USE OF PROPOSED WORK

In this system, we are controlling the mouse pointer by using colored tape or caps on the

fingertips and with the open and Close gestures we are operating the mouse, we are using

inbuilt libraries in Python to get the pointer coordinates and match them with the masked

region from the camera to screen resolution.

2. METHOD

The methodology of each system component will be covered separately. These are the

different sections:

2.1. CAMERA SETUP

The web-cam manages the mouse operations. We need to use a Video Capture object to

capture a video. After that, we can capture frame-by-frame. We could apply color detection

techniques to any image by making minor algorithm changes.

2.2. PREPROCESSING THE FRAMES

The web camera is continually active and captures the video during the program's duration

thanks to an infinite loop. Next, each frame that was gathered in RGB (the default) color

system is converted to HSV color format. Grif et al. [2016].

2.3. MASKING TECHNIQUE

We are producing a certain region with the help of a mask by the use of the original image

using the threshold image and applying AND operation when we do that the masked region is

then highlighted and we get the region we want and we can propose some rules to that area

Chienet al. [2015] Grif et al. [2015].

2.4. OPERATION

The detected coordinate is then used to convert camera resolution to actual screen resolution

Wen et al. [2012]. The position of the mouse is then set, however, it will take some time to

shift the mouse pointer. Once the current mouse position and the designated mouse location

are the same, we started a loop.

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

Fig 1. FlowChart of proposed Method.

2.5. CLICKING

In this instance, dragging and clicking the object is used to complete the action. It is

comparable to making an open gesture, but since there is only one item, we simply need to

determine its center. And we'll position that there where our mouse pointer will be.

2.6. DRAG

We introduce a variable called "flag" to implement dragging. It will be set to 1 if it was clicked

in the past. So, after finding the open gesture, we click it and then double-check that the flag is

set to 1. If it is set to one, the drag action is performed; if not, the mouse move operation is

performed.

3. RESULT AND EVALUATION

Many of the current iterations of AI hand control technology have been created for people with

conditions like arthritis, carpal tunnel syndrome, or paralysis. This implemented system aims

to regulate a mouse cursor using the fingertips of the individual. AI hand control technology is

a brand-new development that allows people with limited hand usage to control their

computers via hand movement recognition. A custom locking algorithm is employed to

convert masked region coordinates from a virtual screen into a full-screen one to be

accustomed to controlling the mouse Suriya et al. [2014]. The software recognizes the position

of the person’s hand in space and replays that movement on the screen as a mouse cursor

movement. The suggested mouse system captures images using either a webcam or a built-in

camera while considering the camera quality, allowing us to reduce the number of system

components. This creates a greater impact on the Environment by stopping the E-waste

generation from keyboard and mouse and their wires.

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

Fig 2. Hand Gesture for clicking and dragging.

Fig 3. Hand Gesture for Deselecting.

Table I. Performance analysis (Where S-successful and U-unsuccessful).

Mouse

operations No. Detection Drag Left-

Click Move

1 S S S S

2 S S S S

3 S S S S

4 S U S S

5 S S S S

6 S S S S

7 S S U S

8 S U S S

9 S S S S

10 S S S S

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

Fig 4. Number of Mouse Operation Performed Successfully.

The Histogram displays how frequently each of the four operations is carried out correctly and

accurately.

The correctness of the system was assessed using the below formula to calculate the

performance of the system.

Where TF is the total number of operations performed and DF is the number of operations

successfully recognized.

Hence, the accuracy of the system is 92.5

4. CONCLUSION

Gesture recognition is used to give the best human-machine interface. Gesture recognition is

essential for the creation of new human-computer interface techniques. It facilitates more

natural interactions between people and machines Grif etal. [2015]. Many different

applications, including robot control and sign language recognition for the deaf and dumb, can

use gesture recognition. This technique has applications in many different fields, including

augmented reality, prosthetics, computer graphics, gaming, and biomedical devices. Our

system's Digital Canvas, becoming increasingly popular among artists, allows them to use the

Virtual Mouse technology to create 2D or 3D images while using hands like brushes and a VR

headset as a display. Patients who lack control over their limbs can benefit from this device.

Modern gaming consoles have incorporated computer visuals and gaming technology to create

interactive games that track player actions and translate them into commands Chowdhury et al.

[2020].

This work can be further extended to make the system more adaptable to various lighting

conditions and background complexity. It is possible to create a user interface that is both

efficient and complete in terms of mouse functionality. Additionally, it would be excellent to

look into cutting-edge mathematical techniques for image processing and look into other

hardware options that would produce more precise hand detections. This study illustrated the

possibilities for streamlining user interactions with personal computers and hardware systems

in addition to illustrating the various gesture operations that users may perform.

Accuracy=DF/ TFx100

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

REFERENCES

[1] S. R. CHOWDHURY, S. PATHAK AND M. D. A. PRAVEENA (2020). Gesture Recognition

Based Virtual Mouse and Keyboard. 4th InternationalConference on Trends in

Electronics and Informatics (ICOEI). doi: 10.1109/ICOEI48184.2020.9143016.

[2] HORATIU-STEFAN GRIF, ZOLTAN GERMAN, ADRIAN GLIGOR (2015). Hand Posture Mouse.

Procedia Technology, 19, 766-771, https://doi.org/10.1016/j.protcy.2015.02.108.

[3] HORATIU-STEFAN GRIF, TRIAN TURC (2018). Human Hand gesture based system for Mouse

cursor control. Procedia Manufacturing, 22, 1038-1042, https://doi.org/10.1016/

j.promfg.2018.03.147.

[4] HORATIU-STEFAN GRIF, CORNEL CRISTIAN FARCAS (2016). Mouse Cursor Control System

Based on Hand Gesture. Procedia Technology, 22, 657-661, https://doi.org/10.1016/

j.protcy.2016.01.137.

[5] M. S. GHUTE, M. ANJUM AND K. P. KAMBLE (2018). Gesture-Based Mouse Control.

Second International Conference on Electronics, Communication, and Aerospace

Technology(ICECA), (pp. 710-713). doi:10.1109/ICECA.2018.8474905

[6] MALI, Y., MALANI, H., MAHORE, N., & MALI, R. (2022). Hand Gesture Controlled Mouse.

International Research Journal of Engineering and Technology.

[7] P. -Y. Chien, Y. -H. Miao and J. -I. Guo (2015). A 3D hand tracking design for gesture

control in complex environments. VLSI Design, Automation and Test (VLSI-DAT), 1-4.

doi:10.1109/VLSI-DAT.2015.7114577

[8] A. PRADHAN AND B. B. V. L. DEEPAK (2015). Obtaining hand gesture parameters using

image processing. International Conference on Smart Technologies and management for

computing, Communnication, Controls, Energy and Materials. IEEE.

[9] PROF. SHITAL PAWA R, S. M. (2022). Virtual Mouse using Hand Gesture. Gradiva Review

Journal, 8(1).

[10] R. SURIYA AND V. V IJAYACHAMUNDEESWARI (2014). A Survey on hand gesture

recognition for simple mouse control. International Conference on Information

Communication and Embedded Systems (ICICES2014). IEEE.

[11] TITLEE, R., RAHMAN, A. U., ZAMAN, H. U., & RAHMAN, H. A. (2017). A novel design of

an intangible hand gesture controlled computer mouse using vision based image

processing. 3rd International Conference on Electrical Information and Communication

Technology (EICT). IEEE.

[12] Y. WEN, C. HU, G. YU AND C. WANG. (2012). A robust method of detecting hand gestures

using depth sensors. IEEE International Workshop on Haptic Audio-Visual

Environments and Games (HAVE 2012) Proceedings. IEEE.

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022

AUTORS BIOGRAPHY

Mr. Piyush Deorao Pilare is pursuing 4th year B.E. in the department of Electronics and

Communication Engineering, at Shri Ramdeobaba College of Engineering and

Management, Nagpur.

E-mail: pilarepd@rknec.edu

Ms. Coral Mahato is pursuing 4th year B.E. in the department of Electronics and

Communication Engineering, at Shri Ramdeobaba College of Engineering and

Management, Nagpur.

E-mail: mahatocr@rknec.edu

Ms. Chanchal Khergade is pursuing 4th year B.E. in the department of Electronics and

Communication Engineering, at Shri Ramdeobaba College of Engineering and

Management, Nagpur.

E-mail: khergadecs@rknec.edu

Mr. Shubham Agrawal is pursuing 4th year B.E. in the department of Electronics and

Communication Engineering, at Shri Ramdeobaba College of Engineering and

Management, Nagpur.

E-mail: agrawalsr_10@rknec.edu

P. N. Thakre has received Bachelor’s degree in Electronics Engineering from RTM

Nagpur University in 2010. He has done M.Tech. in Electronics Engineering from Shri

Guru Gobind Singhji Institute of Engineering and Technology, Nanded University in

2013. Presently he is pursuing Ph. D. from Shri Ramdeobaba College of Engineering

and Management, RTM Nagpur University, under the fellowship of Visvesvaraya PhD

Scheme for Electronics & IT. His research area includes Non-Orthogonal Multiple

Access (NOMA) for 5G Wireless Communication Systems and Wireless channel

Estimation Algorithms. Presently he is working as Assistant Professor in Electronics &

Communication Engineering Department, Shri Ramdeobaba College of Engineering and

Management, Nagpur.

https://doi.org/10.17993/3ctecno.2022.v11n2e42.71-79

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254-4143

Ed. 42 Vol. 11 N.º 2 August - December 2022