Fatigue detection software

In an operational environment scenario where operating systems are dependent on human performance, fatigue can be defined as an inclination to degrade performance. Thus, fatigue is an indicator of baseline risk for the occurrence of errors and accidents.

Globally mining operations are at risk of fatigued workers. Sleepiness and fatigue increase human error and contribute to accidents which can be fatal. Factors compounding fatigue levels in mine workers include; disruptions in circadian rhythms due to shift work, exposure to noise, vibration and chemicals, monotonous and repetitive nature of tasks and night shift driving. Studies recognise a connotation between lifestyle and fatigue. Mine workers in developing countries depend on unreliable public transport systems which add additional commuting hours to their workday. These workers are more susceptible to poor quality and quantity of sleep.

Fatigue is a form of impairment. In 2011, Australian Coroner Annette Hennessy compared fatigue to drunk driving.[1] Fatigued workers are simply less alert and more likely to exercise poor judgement. It's especially risky because often a tired operator is the worst judge of how fatigued he or she may be. David Edwards PhD, Global Mining Safety Solutions Manager at Caterpillar Inc. compares it to asking a drunk person if they believe they are too intoxicated to drive.[2]

Vehicles and driving are recognised as a critical risk in mining environments. Vehicle to vehicle and vehicle to human interactions are generally fatal. The real monetary cost of accidents extends beyond compensation and insurance pay-outs, medical expenses and investigation costs. Fatal accidents often result in the temporary suspension of operations and loss of production. World class mining operations aspire to a fatality free environment and publish their annual safety performance in their Annual Report. There is a global expectation for mines to reduce injuries, eliminate fatalities and prevent catastrophic incidents.

Most mines and commercial truck fleets rely on soft controls such as procedures and other counter-measures to manage fatigue. Common counter-measures that could potentially alleviate fatigue and improve alertness levels in haul truck drivers include; rest days, sleep management, well-designed shift work schedules and structured breaks during the shift, health screening and counselling, education programmes, food and fluid intake and devices for measuring driver's alertness.

The consequences of fatigue are specifically evident in road traffic safety statistics. However, it is not only drivers of light and commercial vehicles that are at risk. Across all industries shift workers are vulnerable to fatigue-related incidents especially during night shift. Safety statistics are not always available and seldom record the causal factors of the incident. In this section road safety statistics are used to illustrate the context of the fatigue problem.

Driving fatigue generally refers to the state in which a driver possesses physiological and mental function deficiencies, and where driving skills decline objectively, usually after an extended period of driving. A driver that is asleep behind the wheel will not act to avoid a collision or accident and for this reason the accident is much more likely to cause severe injuries or death.[3] Fatigue-related road accidents are three times more likely to result in severe injury or death. A great proportion of these accidents occur between the hours of 14h00-16h00 and 02h00-06h00. During these two time periods drivers are more likely to become drowsy, which increases the chance of accidents.[4]

Statistics show that a leading cause of fatal or injury-causing traffic accidents is a diminished vigilance level. In the trucking industry, 57% of fatal truck accidents are due to driver fatigue. It is the number one cause of heavy truck crashes.[4]

According to the National Sleep Foundation's 2005 Sleep in America poll, 60% of adult drivers – about 168 million people – say they have driven a vehicle while feeling drowsy in the past year and 13% of them admitted to have done so at least once a month.[4]

The National Highway Traffic Safety Administration (NHTSA) conservatively estimates that 100,000 police-reported crashes are the direct result of driver fatigue each year. This resulted in an estimated 1,550 deaths, 71,000 injuries, and $12.5 billion in monetary losses.[4]

In Australia, 60–65% of truck haulage accidents are directly related to operator fatigue and 30% of all crashes are related to fatigue.[5]

The complex interaction of the major physiological factors responsible for sleepiness – circadian rhythms and the homeostatic drive for sleep – pose formidable technical challenges to the design and development of fatigue detection systems. The technology must be robust and capable of high accuracies in diverse operational environments with constantly changing conditions and varying customer needs.[6]

To meet the requirements of efficiency and functionality the technology should comply with the following guidelines:[7]

• It should measure what it is operationally and conceptually intended to measure and be consistent in these measurements over time. Thus, a device designed to measure eye blinks (operationally) and alertness (conceptually) should measure these all the time for all drivers.
• The software technology used in the device should be optimised for sensitivity and specificity. False negatives should be minimised through accurate and reliable detection of reduced alertness levels. False positives should be minimised through accurate and reliable identification of safe driving and operator vigilance.
• The device should be robust, reliable and capable of continuous operation over extended periods, such as a shift. Maintenance and replacement cost should not be excessive.
• Be capable of real time monitoring of driver or operator behaviour.
• The device should be capable of accurately operating under various operational conditions during the day, at night and under illuminated conditions. Accuracy should not be compromised by conditions in the operator cab, such as humidity, temperature, vibration, noise, etc.
• Audible warning signals should not startle the operator and should be adjustable over a reasonable range. The signals should be distinct and audible under operating conditions to not be confused with other alarms and signals.

Irrespective of the obvious safety benefits fatigue detection devices offer, successful acceptance of the technology depends on whether the operator perceives the benefits as greater than the cost. User acceptance is influenced by the following factors:[8][7]

• Ease of use: the technology should be understandable and intuitive in its operation. The operator should be familiar with the capabilities, limitations and operational parameters in all operating conditions. The device output should be easily and correctly interpreted by operators with differing cognitive and physical ability. The operator's vision of the road and other controls should not be obscured.
• Ease of learning: the success of the technology depends on its congruency with the mental model of the operator, how easy it is to understand, recall and retain the information and to react thereupon. Most importantly, the operator should trust the device's accuracy to maximise 'hits' and eliminate false or nuisance alarms.
• Perceived value: the operator should perceive the technology as contributing to a safer and more alert driving experience, but at the same time it should not create a state of over-reliance. The device should benefit the operator in his own fatigue management programme. It should be clear that the device is completely safe to use without any negative side effects on the health of the operator. Operator data, captured and transmitted to a central control room should be completely confidential.
• Advocacy: a critical component of user acceptance is measured by the willingness of operators to purchase and endorse the technology. For the perceived safety benefits of the device, market uptake will increase when it is supported by the intended users – operators, fleet managers, trucking associations, safety departments etc.
• Driver behaviour: the allocation of the operator's attention to maintain safe driving should not be negatively influenced by interacting with the device. Extended exposure to the technology should create positive influence in driver behaviour as well as lifestyle changes with regard to fatigue management.

There were significant advancements in fatigue monitoring technology the past decade. These innovative technology solutions are now commercially available and offer real safety benefits to drivers, operators and other shift workers across all industries.[9]

Software developers, engineers and scientists develop fatigue detection software using various physiological cues to determine the state of fatigue or drowsiness. The measurement of brain activity (electroencephalogram) is widely accepted as the standard in fatigue monitoring. Other technology used to determine fatigue related impairment include behavioural symptom measurements such as; eye behaviour, gaze direction, micro-corrections in steering and throttle use as well as heart rate variability.

Fatigue detection software analyse behaviour and warning signs to determine the onset of fatigue. The technology has the potential to be a highly accurate tool for detecting the early stages of fatigue in drivers and minimise the likelihood of incidents. The technology allows operators in real time to visually identify their levels of alertness. Operators can proactively assess different approaches to maintain alertness and manage their fatigue levels.

Electroencephalography (EEG) is a technique that reports the electrical brain activity non-invasively.[10] It was discovered by Hans Berger in 1924 and evolved over more than 90 years to the advanced technology of today. A dramatic reduction in size, weight and cost of EEG instrumentation and the potential to communicate wirelessly with other digital systems paved the way to extend the technology to previously unsuspected fields, such as entertainment, bio-feedback and support for learning and memory training. Experimentation and product development around this technology include fatigue detection applications.

New EEG fatigue detection software measures an individual's ability to resist sleep.[10] Micro-sleep only occurs when an individual fails to resist sleep, it does not occur when an individual chooses to rest. Operators of heavy mobile equipment are accustomed to resist sleep; it comes natural and is almost a subconscious behaviour. However, when an individual's ability to resist sleep diminishes, there is a risk of a micro-sleep. The ability to resist sleep is therefore the most relevant safety measure for equipment operators. The underlying measurement behind the technology is brain activity. Electroencephalogram has been the gold standard in sleep and fatigue science.[10] Being a more direct physiological measure, it provides improved accuracy by avoiding erroneous measurements related to the external environment.

Apart from developing practical wearable technology, the universal mapping of EEG information to a useful measurement is required for accurate fatigue monitoring in an operating environment. Although EEG analysis is well advanced, scientists found that due to natural physiological person-to-person variations, rigorous rules to interpret brain activity cannot effectively be applied to the entire population. This implies that a rule-based approach to EEG fatigue measurements would be impractical, as each physiological variation would require a specific rule applicable to a specific person.

To overcome this problem, scientists developed the Universal Fatigue Algorithm based on a data-driven approach. Drowsiness is a state determined by independent non-EEG measures. The Oxford Sleep Resistance Test (OSLER test) and the Psychomotor Vigilance Test (PVT) are the most commonly used measures in sleep research.[10] Both tests were used to establish the sample dataset for development of the Universal Fatigue Algorithm. The algorithm was developed from real EEG of a large number of individuals. Artificial intelligence techniques were then used to map the multitude of individual relationships. The implication is that the result gets progressively universal and significant as more data from a wider range of individuals are included in the algorithm. In addition to an unseen-blinded experiment approach, testing of the algorithm is also subject to independent external parties.[10]

PERCLOS is a drowsiness detection measure, referred to as the percentage of eyelid closure over the pupil over time and reflects slow eyelid closures or droops rather than blinks.[11] Various real-time operator drowsiness detection systems use PERCLOS assessment and propriety developed software to determine the onset of fatigue. Each technology developer use a unique set-up and combination of hardware to improve the accuracy and ability to track eye movement, eyelid behaviour, head and face poses under all possible circumstances.[11]

Some systems rely on a camera module on a rotating base that is mounted on the dashboard inside the cab. The device has a large field of view to accommodate operator head movements. The equipment uses eye-tracking software with a structured illumination approach that depends on the high contrast between the pupils and the face to identify and track the operator's pupils.

Alternatively, flexible and mobile tracking systems provide head and face tracking which include eye, eyelid and gaze tracking. These systems now provide real time feedback without the use of wire, magnets or headgear.

Although studies confirmed a correlation between PERCLOS and impairment, some experts are concerned by the influence which eye-behaviour unrelated to fatigue levels may have on the accuracy of measurements. Dust, insufficient lighting, glare and changes in humidity are non-fatigue related factors that may influence operator eye-behaviour. This system may therefore be prone to higher rates of false alarms and missed instances of impairment.[10]

The computer vision system utilises an unobtrusive dashboard mounted camera and two infra-red illumination sources to detect and track the facial features of the operator. The system analyses eye closures and head poses to determine early onset of fatigue and distraction. The fatigue detection algorithm calculates AVECLOS. This is the percentage of time the eyes are fully closed during a one-minute interval.[12]

The technology was developed for the domestic and commercial markets and is currently being tested in a Volvo demonstration vehicle.

Recently, fatigue detection system software has been modified to run on Android mobile phones. The technology utilises the mobile phone camera which is mounted in a stand on the cab dashboard to monitor operator eye movement. The developers of the system preferred to use eyelid movement technique.[13] The robust system is capable of tracking fast head movements and facial expressions. External illumination is limited which reduce operator interference. Other potential techniques were found to have drawbacks with the application of the specific hardware. Yawning detection makes it difficult to precisely detect lip positions. Detection of head nodding requires electrodes to be fixed to the scalp.

Further, deep learning methods for action recognition have also been successfully applied on mobile devices.[14] Deep learning techniques do not require separate feature selection steps to identify eye, mouth or head positions and have the potential to further increase prediction accuracy.

App-based technologies have also been released that do not use cameras, but instead leverage the Bowles-Langley Test (BLT)[15] through a simple 60-second game-like experience. Companies who have released fatigue impairment apps with this type of technology include Predictive Safety, based in Denver, Colorado, USA and Aware360 based in Calgary, Alberta, Canada.

The technologies discussed in previous sections, opened up the automotive safety landscape for various manufacturers to add new safety features to their production models. The drivers of the development of these features can be contributed as either regulatory pressure or the enhancement of the value offering of their product through added features.

New developments in the car industry is as follows:[16]

• Further development is undertaken by NVIDIA, the chip supplier to Audi, Mercedes, Tesla and others. NVIDIA is developing the co-pilot, an artificial-intelligence tool that can learn the behaviours of individual drivers and determine abnormal behaviour.
• For early drowsiness detection, Plessey Semiconductors developed sensors, to be placed in a seat, that monitor changes in heart rate.
• Bosch, a German supplier of technology to many automotive companies, is developing a camera-based system that will monitor head and eye movements, as well as, body posture, heart rate and body temperature.
• Valeo, another supplier of automotive technology, is developing an infrared camera system that will monitor children in the rear seat as well as the driver's shoulder, neck and head movements, looking for deviations from the norm.
• Mercedes's Attention Assist monitors a driver's behaviour for the first 20 minutes behind the wheel to get a baseline of behaviours. Then, the system checks those against as many as 90 indexes, such as steering wheel angle, lane deviation and external factors such as wind gusts and pothole avoidance.

The application for these systems are not only limited to car manufacturers, but third party technological companies as well. These companies have developed hardware like the Anti Sleep Pilot and Vigo. Anti-Sleep Pilot is a Danish device that can be fitted to any vehicle that uses a combination of accelerometers and reaction tests. The Vido is a smart Bluetooth headset that detects signs of drowsiness through eye and head motion to alert users.

By 2013 it was estimated that about 23% of new registered cars had various degrees of drowsiness detection systems in place. The importance of these systems can be contributed to safety regulatory bodies including these systems in their rating systems. Regulatory systems like the Euro NCAP system primarily focuses on occupant safety ratings, pedestrian rating and child occupant ratings through the release of an overall 5-star rating. In 2009 a new category was added in the form of Euro NCAP Advance safety assist systems, The Euro NCAP Advanced reviews active safety monitoring systems of new car models and aims to provide car buyers with clear guidance about the safety benefits offered by these new technologies.

Here is a list of some advanced safety systems recently developed by car manufacturers.[16]

• Steering pattern monitoring, Vision Enhancements and Autonomous Emergency braking

Primarily uses steering input from electric power steering system, radar systems and cameras. These systems could facilitate autonomous braking in the case of drowsiness or distraction, when a driver physically does not act quickly enough. It also has the facility of autonomous driving in the prevention of an accident, when the driver reacts too slowly or not at all.

• Vehicle position in lane monitoring

Uses lane monitoring camera and radar sensors. These systems can assist and warn you when you unintentionally leave the road lane or when you change lane without indication, commonly due to fatigue. These features are commonly referred to as blind spot monitoring, lane keep assist or lane departure monitoring.

• Driver eye/face monitoring

Requires a camera watching the driver's face, referred to as attention assist, these systems detect and warns drivers to prevent them falling asleep momentarily whilst driving.

• Physiological measurement

Requires body sensors for measure parameters like brain activity, heart rate, skin conductance and muscle activity. It is not limited to car drivers only. Studies have also been done in assessing neuro-physiological measurements as a method to improve alertness of aircraft pilots.

• Fatigue (safety)
• Fatigue Avoidance Scheduling Tool
• Shift work