3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

A NON-INTRUSIVE METHOD FOR DRIVER

DROWSINESS DETECTION USING FACIAL

LANDMARKS

Sivakumar Pothiraj

Professor, ECE Department, Kalasalingam Academy of

Research and Education (Deemed to be University).

Krishnankoil, (India).

E-mail: siva@klu.ac.in

ORCID: https://orcid.org/0000-0003-1328-8093

Rampranav Vadlamani

Student, ECE Department, Kalasalingam Academy of

Research and Education (Deemed to be University).

Krishnankoil, (India).

E-mail: vrampranav9@gmail.com

ORCID: https://orcid.org/0000-0001-9270-6884

Bonthu Pavan Kumar Reddy

Student, ECE Department, Kalasalingam Academy of

Research and Education (Deemed to be University).

Krishnankoil, (India).

E-mail: vrampranav9@gmail.com

ORCID: https://orcid.org/0000-0003-1718-040X

Recepción:

16/10/2019

Aceptación:

14/10/2020

Publicación:

30/11/2021

Citación sugerida:

Pothiraj, S., Vadlamani, R., y Reddy, B. P. K. (2021). A non-intrusive method for driver drowsiness

detection using facial landmarks. 3C Tecnología. Glosas de innovación aplicadas a la pyme, Edición Especial,

(noviembre, 2021), 71-85. https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

ABSTRACT

Driver weariness is one of the real reasons behind accidents. Recognizing the sleepiness

of the driver is one of the best methods for estimating driver weariness. The motivation

behind this paper is to build up a drowsiness detection system that works by monitoring

the eye movement of the driver and alerting the driver by producing an alarm or vibration

when the person is found drowsy. This paper shows a non-intrusive model for the fatigue

detection dependent on processing video streams of an individual's face. The proposed

model is not quite the same as meddlesome techniques dependent on natural methodology

(Electroencephalogram, Electrooculogram and some sensors), which require gadgets

explicitly. Unlike traditional image processing techniques, we use computer vision and

machine learning technique to display a prototypal adaptation of a real-time system with

individual feedback to monitor and identify when the driver may be sleepy directly from a

web camera. The drowsiness detection model depends on face alignment and then evaluation

of the Eye Aspect Ratio (EAR) which uses Histogram of oriented gradient (HOG) features

combined with Support Vector Machine (SVM) classier for blink detection. Utilizing such

a system, it is conceivable to alarm the client of the threat of nodding o, so that enough

actions can be made, diminishing the risk of human mistake and avoiding accidents.

KEYWORDS

Non-Intrusive Method, Facial Landmarks, Support Vector Machine, Eye Aspect Ratio

(EAR).

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

1. INTRODUCTION

Drowsiness is an automatic human physical movement. Since tiredness can be specically

identied with the human xation and liveliness, drowsiness identication has been

connected in elds like in human behaviour investigation, fatigue detection, readiness level

estimation and so on. When the driver experiences drowsiness state, he starts losing the

control and he might be suddenly deviated from the road and may hit an obstacle or another

car. Driver weariness is one of the critical factors in many auto collisions. Safe driving is a

noteworthy worry of social orders everywhere throughout the world. A huge number of

individuals are seriously harmed because of drivers nodding o at the wheels every year.

According to the poll conducted by the National Sleep Foundation’s Sleep, 60% of people

have fallen sleepy while driving. However, many people cannot tell whether they are about

to fall asleep or not. 30% of car crashes have been brought about by drowsiness alone.

The signs of the driver drowsiness are:

• Hard to focus, frequent blinking.

• Continuous Yawning

• Daydreaming; wandering/disconnected thoughts

• Diculty in keeping his head up

• Might be feeling of restless.

Various techniques have been proposed for fatigue/drowsiness detection. These

techniques are based on Image Processing, Electroencephalograph, Electrooculogram,

Electrocardiogram, and articial neural network. The image processing-based techniques

are divided into three categories. These categories are matching of a template technique,

detecting of an eye blink, the technique based on yawning, identifying the pose of the head.

The techniques to measure drowsiness are classied into two types.

• Intrusive Techniques.

• Non- Intrusive Techniques.

Intrusive techniques require the use of some explicit gadgets which always require some

measuring gadgets to be in contact with human body which causes some sort of disturbance

to the person so these are often referred to as contact type method (Electroencephalogram,

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

Electrooculogram), whereas in Nonintrusive techniques there is no need of such devices

which require contact and it requires high-quality cameras for capturing images or

monitoring the video stream, it is also known as contactless method.

In this paper, we propose a system for drowsiness detection which uses computer vision

techniques which uses a machine learning library. Object recognition, tracking is a standout

amongst the most created research spaces in the eld of image processing and computer

vision. It is used in several useful applications in all the available domains like video-stream

analysis, industrial automation, internet of things, robotics, security and surveillance, and in

many more applications. The similar idea is used to detect the human body to identify eye,

mouth, arms and recognizing its shape and movement, making a diagnosis and deriving

human behavior. There are many algorithms for blink detection which are trained on a

wild dataset.

In the proposed system, we monitor the driver continuously and intimate the driver about

the drowsiness when the specied conditions are met. We use facial landmarks technique

which identies the face region. The algorithm behind the facial landmarks is the Histogram

of oriented gradients combined with support vector machine classier. The detector we use

is a pre-trained model trained on HELEN dataset. A threshold value of eye aspect ratio will

be kept. The person is monitored continuously, and the eye aspect ratio is evaluated each

time and compared with the threshold value to decide whether the state of the eye is open

or not. It will be implemented using OpenCV in Python.

2. RELATED WORKS

Yan et al. (2016) proposed a model for real-time drowsiness detection depending on

PERCLOS evaluation and grayscale image processing to identify whether the driver is

sleepy or not. Their algorithm was comprised of three steps. First, it identies the position

of the face of the driver and converts them into grey-scale images, and then the position of

eyes is localized using a small template. Secondly, the data obtained in the rst step is used

to establish a drowsiness model and at last, depending on the individual drowsiness model

the developed system monitors the driver’s state without interruption. Once the driver is

found to have sleepiness state, an alarm is produced to alert the driver.

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

Leo and Sankar (2015) used Haar based cascade classier for identication of eyes and

the Histogram oriented gradients with Support vector machines for the detection of blink.

When the eye blinks were detected the PERCLOS value for the eyes was calculated. If the

computed value is beyond six seconds, it classies the person as drowsy. First, the driver's face

is monitored through a camera and then Viola and Jones's method was used to identifying

the face of the driver. Once the face is identied the Haar features are extracted and then

the blink detection algorithm is used to determine the blink in the video stream and then

the PERCLOS is evaluated.

According to Amin and Rahmati (2011), the developed system acquires the images in a

sequence and given as input in the rst step. The system follows 4 steps: Detection of

the face, extraction of facial components, tracking of facial components and detection of

drowsiness. In that system, the background subtraction method is used to detect the face

region in the image and this step is repeated until the face region is obtained accurately. The

facial components like eyes, eyebrows and mouth region are extracted. The method used in

the second step is Horizontal projection and template matching. The results obtained in the

third step is matched with the reference template for tracking. Using the reference template,

the person can be identied as feeling sleepy or not.

3. METHODOLOGY

This paper presents an algorithm to precisely estimate the position of eyes using facial

landmarks in a computationally efcient way. Our proposed model has 4 phases.

• Monitoring for the face in a video stream.

• Applying Facial Landmarks.

• Evaluating the eye aspect ratio (EAR).

• Drowsiness detection.

Unlike some traditional image processing techniques for extracting eye regions or to detect

blinks, which typically involves the following processes:

(1) Face Detection (2) Eye Localization (3) Localizing the white eye region (4) Identifying

whether the white region disappears for a certain time which indicates the blink of an eye

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

(5) Evaluate some parameters related to detect the state of an eye(open or close). Using

these methods, the output can be achieved but the result may vary when there is a head

movement, or any facial expression variation is noticed.

Our goal is to detect the eyes using a shape prediction technique. The machine learning

library ‘dlib’ has the facial landmark detector which has been trained based on HELEN

dataset. The algorithm behind this detector is the combination of Histogram of oriented

gradients (HOG) and Support Vector Machines classier.

A. Histogram of oriented gradients

The functions we use from dlib library has been trained based on the HOG algorithm.

HOG is a feature descriptor which is a portrayal of an image that simplies the image by

removing unnecessary information and extracting useful information The feature vector

generated by the algorithms is given into an image classication algorithm like Support

Vector Machine (SVM) to generate accurate outcomes.

In the HOG feature descriptor, the directions of gradients distribution are used as features.

Gradients values are mapped to 0-255. Pixels with large negative change will be black,

pixels with large positive change will be white, and pixels with little or no change will be

gray.

B. Facial Landmarks

Dlib is a landmark’s facial detector with pretrained models. It has two types of models.

They are

• 5-point landmark detector.

• 68-point landmark detector.

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

Figure 1. 5-point detector.

Source: own elaboration.

The 5-point detector localizes:

• 2 points for the left eye.

• 2 points for the right eye.

• 1 point on the nose.

Whereas the 68-point detector localizes regions along the eyes, eyebrows, nose, mouth and

jawline.

Figure 2. 68-point detector.

Source: own elaboration.

In terms of speed, the 5-point detector is 8-10% faster than the 68-point detector and it also

consumes 10 times less memory. So, the 5-point detector is fast and the most appropriate

use of 5-point detector is the face alignment.

But for our proposed work since we need to compute eye aspect ratio, if we use 5-point

detector we cannot evaluate the eye aspect ratio as there were no vertical points for eye

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

region so we are using the 68 point detector where we have six points per eye and the eye

aspect ratio can be computed easily to produces the accurate results.

C. Computing Eye Aspect Ratio

The Eye Aspect Ratio is an estimate of the eye-opening state. If the computed Eye Aspect

Ratio value is lower than a threshold value, we could tell that the person's eyes are closed.

The indexes of the eyes are taken into consideration to compute the eye aspect ratio value.

Figure 3. Eye aspect Ratio.

Source: own elaboration.

The numerator of the EAR equation calculates the distance between the vertical eye

landmarks while the denominator calculates the distance between horizontal eye landmarks.

When the eye is detected open the EAR value is almost constant and when the eye is closed

(blink) the EAR suddenly drops to zero.

We can use this simple equation to avoid image processing techniques and the blinking

detection simply depends on the ratio of vertical and horizontal eye landmark distances.

We use SciPy package to compute the Euclidean distance between eye facial landmarks We

also include play sound library so that a suitable alert can be produced when the EAR value

is below a threshold value.

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

Figure 4. Block Diagram of Drowsiness detection system.

Source: own elaboration.

Figure 5. Flowchart of our system.

Source: own elaboration.

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

4. RESULTS

The face of the driver was monitored continuously, and the landmarks were applied when

the face was found. Even if there is a tilt in the head movement the detector was able to

identify the eye regions if the face was in the scope of the camera. Once the eye regions are

obtained the eye aspect ratio is evaluated. The threshold we kept is 0.28 as it would be more

approximate value for almost 70% of the people. And when the eye aspect ratio computed

was found to be below the threshold value then an alarm was produced. And if the EAR

value is above the threshold value then the EAR would be continuously monitored until we

turn o the device. We observed that the CNN based detector works well for non-frontal

faces at odd angles where HOG based detector struggles.

Unfortunately, CNN based detector is computationally heavy and is not suitable for real-

time video now. We use HOG detector as it is highly ecient and there is no need of

detecting any faces at odd angles. When the face is detected the facial landmarks will be

applied and eye regions will be extracted and to make it visible, we applied draw Contour

method. The indexes of the eye are sent to the eye aspect ratio method and computed for

each eye and prints the average value. The output shows the EAR value obtained.

Figure 6. EAR Output.

Source: own elaboration.

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

Figure 7. Drowsiness Detected.

Source: own elaboration.

Figure 8. EAR Values.

Source: own elaboration.

When the eye aspect ratio computed is below the threshold value i.e. 0.28 and it lasts for

40 consecutive frames then drowsiness will be detected, and this process will be repeated

continuously.

5. CONCLUSIONS

In this paper, a non-intrusive method for drowsiness detection was implemented using

detector from dlib library and OpenCV in Python. The predictor we used is a 68-landmark

predictor. The person will be monitored continuously and when detected the landmarks

will be applied and EAR value is evaluated and then compared that with the threshold

value. The proposed approach uses Eye Aspect Ratio and Eye Closure Ratio with adaptive

thresholding to detect driver’s drowsiness in real-time. This is useful in situations when the

drivers are used to arduous workload and drive unceasingly for long distances. The system,

https://doi.org/10.17993/3ctecno.2021.specialissue8.71-85

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue

Noviembre 2021

if integrated, can reduce the number of fatalities and harms that happen regularly due to

these drowsy states of the drivers.

The developed system works well under the daylight conditions and in any resolutions. If

the device should be used under dark conditions as well, then we can implement the system

using night vision cameras. This makes the system suitable to work in any conditions.

ACKNOWLEDGEMENT

Authors want to express their gratitude to ECE Department at Kalasalingam Academy of

Research and Education for allowing them to utilize the computing facilities in DST-FIST