Intelligent Techniques for Improving the Aviation Operations

1. Intelligent Techniques for Improving the Aviation Operations Professor Lakhmi C. Jain PhD, ME, BE(Hons), Fellow (Engineers Aust), KES Founder http://www.kesinternational.org/organisation.php

2. Courtesy: Universal press

3. Adelaide City

4. Australia

5. KES Internationalwww.kesinternational.org • KES regularly provides networking opportunities for professional community through one of the largest conferences of its kind in the area of KES.

6. KES Conferences • 1997-99 Adelaide, Australia • Brighton, UK • Osaka, Japan • Crema, Italy • Oxford, UK • New Zealand • Australia • UK • Italy • 2008 Croatia

7. Chile • 2010 UK • 2011 Germany • 2012 Spain • 2013 Japan • 2014 Poland • 2015 Singapore • 2016 UK

8. 2015 KES Conferences • Innovations through Knowledge Transfer 2015 • Agents and Multi-Agent Systems 2015 • Sustainability in Energy and Buildings 2015 • Intelligent decision Technologies 2015 • Intelligent Interactive Multimedia Systems and Services 2015

9. 2015 KES Conferences • Innovations in Medicine and Healthcare 2015 • Knowledge-Based and Intelligent Information and Information Systems 2015 • Archiving for the Future 2015 • Innovations in Music 2015

10. KES Journals • International Journal of Knowledge-Based Intelligent Engineering Systems, IOS Press, The Netherlands • http://www.iospress.nl/html/13272314.php • Intelligent Decision Technologies, IOS Press, The Netherlands, 2007-present. • http://www.iospress.nl/loadtop/load.php?isbn=18724981

11. BOOK SERIES KES members are involved in book series published by leading publishers. • Smart Innovation, Systems and Technologies: www.springer.com/series/8767 • Intelligent Systems Reference Library: www.springer.com/series/8578 • Advanced Information and Knowledge Processing: www.springer.com/series/4738

12. University of South Australia Intelligent Flight Data Monitoring System for Improving the Safety of Aviation Operations .

14. AIMS and Objectives • Develop an intelligent Flight Data Monitoring (FDM) system incorporating supervised learning • Identify adverse trends and potential unsafe deviations from the accepted norms of flight operations • Provide early warning signals to flight crew, and/or air traffic control staff • Explanation to flight crew as to why the current flight operation is considered safe/unsafe

15. Specific Objectives • To acquire flight data and other relevant information for flight operations and safety • Develop CI-based algorithms with incremental learning capabilities to categorise flight data and perform risk prediction • To extract rules to explain the prediction from the CI-based algorithms • To evaluate the effectiveness of the FDM system for aviation operations and safety.

16. University of South Australia Background • Air travel in modern turbine passenger aircraft has become extremely safe. • This has largely been attributed to increased mechanical reliability, increased reliability of on‑board automated systems and the wide spread development and implementation of flight crew training in Crew Resource Management (CRM) .

17. University of South Australia Background • However, even with all these advances in aviation safety there remains a stubborn remnant of air crashes which are seemingly not eradicable. • Of these accidents, worldwide, Helmreich and Foushee have suggested that 70% are due to flight crew actions or in some cases inactions. HUMAN ERROR. • This is despite the fact that pilots are extremely technically competent and well trained in CRM. • There is no question that flight crews are highly trained to operate in the technical and human environments of the cockpit.

18. University of South Australia Background • Why do such accidents happen and, perhaps more disturbingly, why do they continue to happen? • Research has shown that most are due to a momentary loss of concentration or awareness during which the flight crew did not consciously notice that a necessary event did not occur, or that an adverse event did occur.

19. University of South Australia Background • In order to function pilots build up a mental model of what is happening around them. • New information they receive will be perceived and restructured in terms of this model until an event happens which forces an unsettling recognition that the model is actually false. This is termed loss of situation awareness (SA). • If this happens too late on in a critical process, the result can be an adverse event.

20. University of South Australia Background • One solution to the problem is increased automation. • However despite the high reliability and accurate flight path control automation can actually decrease a flight crew’s “awareness of parameters critical to flight path control through out-of-the-loop performance decrements, over-reliance on automation, and poor human monitoring capabilities. • Weiner describes reports of pilots unintentionally creating flight paths to wrong locations which went undetected and resulted in collision with a mountain. • This type of accident is referred to as a controlled-flight-into-terrain accident or CFIT.

21. University of South Australia The Statistics CFIT and approach-and-landing accidents (ALAs) accounted for 80% of the fatalities in commercial transport- aircraft accidents (Flight Safety Foundation, Recent Study). The FSF Approach-and-landing Accident Reduction Task Force Report concluded that the two primary causal factors for such accidents are: “omission of action/inappropriate action” and “loss of positional awareness in the air”.

22. University of South Australia Current Technology • In 1974 the Federal Aviation Administration (FAA) mandated that all heavy airliners be fitted with a GPWS. This has lead to a decrease in CFIT accidents however there continues to be a relatively large number of fatalities attributed to ALA or CFIT accidents. • The GPWS uses information from the radar altimeter and air data computer to determine the aircraft’s vertical distance from the terrain below. The system is limited because it only perceives vertical separation between the aircraft and the ground directly below the aircraft in real time. • Since 2003 the GPWS has been replaced by Enhanced GPWS (EGPWS) on all turbine aircraft with 10 or more passenger seats. The EGPWS has a predictive terrain hazard warning function.

23. University of South Australia Current Technology How does it work? • The EGPWS compares the aircraft’s position and altitude derived from the Flight Management and Air Data computers with a 20MB terrain database. In the terrain database the majority of the Earth’s surface is reduced to a grid of 9x9 km squares. Each square is given a height index. In the vicinity of airports the grid resolution is increased to squares of 400m x 400m. The height index and the aircraft’s predicted 3 dimensional position 20 to 60 seconds into the future are compared to see if any conflict exists. • If it does the EGPWS displays an alert or warning to the flight crew. Other than to initially alert the pilots of “TERRAIN” up to 40-60 s before impact or warn the pilots to “PULL UP” up to 20-30 s before impact it does not offer any other solution to the potential problem.

24. Proposed Solution • Develop a Flight Data Monitoring System. • It involves collecting and analysing data recorded during flight operations • Identifying and rectifying adverse trends and deviations from accepted norms of flight operations and safety • Understanding flight operations by tracking trends and detecting flaws before they lead to major incidents • Developing preventive and/or corrective actions such as increased training.

25. Flight Data Monitoring Systems • A number of commercial products are available in the market. • Battele, USA • Teledyne, USA • CEFA, France • Sagem, France • Flightscape, Canada • (These products do not incorporate online learning capabilities)

26. Flight Data Monitoring Systems • Most successful work reported thus far is by Battele and NASA. It is a software tool that aggregates large volumes of flight data and then uses statistical cluster-based techniques to find the unexpected or the abnormal events. • NASA Ames research Centre also reported a clustering-based techniques to find the expected or the abnormal events.

27. Proposed Solution • Computational Intelligence (CI)-based to process and analyse flight data • Ability to incrementally learn and absorb knowledge of aviation instructors/trainers • A variant in the family of Adaptive Resonance Theory (ART) called Fuzzy ARTMAP (FAM) is under investigation

28. Fuzzy ARTMAP (FAM) • FAM is a supervised neural network • It is able to perform on-line or off-line learning • FAM is an integration of Neural net and Fuzzy system. This integration brings the learning capability of neural net and the reasoning capability of fuzzy system. • These are the reasons of selecting FAM as a core engine of the FDM.

29. FAM Architecture Input pattern Target Class

30. Methodology • FAM network monitors the flight data and identify normal as well as a typical flight operations. • Data is related to flight dynamics, aircraft status, weather and environmental variables. • The FAM network examines these data as a combined flight pattern and yields a predicted risk pertaining to whether the flight pattern is within or outside the normal operating conditions. • In FAM, each recognition category, which is associated with a target output, corresponds to an If-Then rule. The features stored in each recognition category can be expressed directly as rule antecedents. • There is an increase in rules and recognition categories with time. • It is proposed to use a pruning strategy to remove insignificant and noisy recognition categories. • It is planned to tag each rule with a confidence factor to reflect its significance.

31. The Operation Phase of FDM • Flight crew performs some flight operations. Flight data as well as relevant environmental and weather information are captured and these form the input to the FDM. • If the input excites a recognised category (due to previous learning) in the FDM knowledge base, then a prediction (safe or unsafe flight pattern) with the corresponding If-Then explanation is retrieved and displayed to the user. • However, if the input pattern does not excite any existing recognition categories in the FDM knowledge base, then the user is so informed of the unknown flight pattern.

32. The Supervised Learning Phase of FDM • Flight patterns recorded from the flight crew during the operation phase are used for learning with the FAM model. • The flight patterns, together with the associated predictions, are first evaluated, one-by-one, by domain experts (qualified flight instructors/check-and-training pilots). • Update of the FDM knowledge base can only be initiated if and only if a prediction pertaining to a particular flight pattern is confirmed by domain experts. • The FDM system is able to continue learning and absorbing new flight patterns into its knowledge base, even after its development. • The capability of supervised learning initiated by domain experts is made possible by the characteristics of the FAM model in overcoming the stability-plasticity dilemma, i.e. absorbing knowledge continually without corrupting its previously acquired knowledge.

33. Characteristics • In order for the system to be reliable it must be have a large database from which to learn and make classification decisions of safe/unsafe, and this database must be expandable. • In human terms, the system must be able to know the difference between a safe and unsafe flight situation, and have the ability to learn whether a new and previously unseen operation is safe. • Both of these deployment requirements are met through the use of supervised learning by domain experts.

34. University of South Australia Pilots Fear Automation!

35. An In-flight agent to monitor pilot situation awareness

36. Air crash examples Case 1: Eastern airlines flight 401, 29 December 19721, • Departed from John F Kennedy International Airport New York at 9:20p.m. • Started the Approach to Miami Airport at approx. 11:30p.m. • Captain noticed nose gear indicator light problem • Started cycling around the airport at 2000ft • First officer switched on the Auto pilot • Captain , Flight engineer and First officer started resolving the problem. • Altitude hold mode was accidentally switched off • Started slow descent • Crash Reason for the crash: The crew were concentrating on the nose gear indicator light, failed to realize the drop in altitude. 1. NTSB report ,1973,http://www.airdisaster.com/reports/ntsb/AAR73-14.pdf

37. Air crash examples contnd..

38. Air crash examples contnd..

39. Air crash examples Case 2: Kenyan airways Flight KQA507 , 5 May 20072 • Took off from Douala to Nairobi at 23:06 on 5th of May • After reaching 1000 ft, started losing height, went unnoticed by the pilots • Pilots assumed that the auto pilot was engaged • Because of the bad weather & dark night there was no visual reference, still no instrument scanning was done. • Erratic inputs from the pilot in last few seconds resulted in rapid loss of height and crash. There were no survivors in the plane. • Reason for the crash : loss of control by the crew as a result of spatial disorientation, failing to perform instrument scanning in the absence of external visual references. 2. http://www.ccaa.aero/surete-et-securite-aerienne-141/aviation/actualite/384,technical-investigation-.html

40. Air crash examples contnd.. Case 3: Turkish Airlines Flight, 25 February 20093 • Started Approach to SchilpolAirport at approx. 10:14a.m. • Left hand radio altimeter showed wrong altitude reading (-8ft) • Auto pilot was on and responding to the change Auto throttle reduced the engine power • The flight crew failed to the control column manually • The pitch altitude increased but the airspeed was decreasing • Crash • Reason for the crash: The Crew failed to recognize the reduction in airspeed & decrease in thrust setting. When the speed decreased at a height of 750 feet, pilots should have noticed the speed & height if they had scanned the airspeed indicator and the artificial horizon indicator respectively on the primary flight displays. 3.http://aviation-safety.net/database/record.php?id=20090225-0

41. Air crash examples Case 4: Air France Flight 447, 1 June 20094 • The aircraft disappeared over the mid-Atlantic without providing any clue about the cause of accident. • The investigation which took over two years, revealed that the captain was confused when the auto pilot was disengaged. • The airspeed was very low , instead of pushing the control forward the pilot pulled it back resulting into stall. • The copilot tried to take over, he was not successful in comprehending the situation. • The pilots lost control of the aircraft and it crashed in to the mid-Atlantic. • Reason for the crash: Pilots failed to cross- check the instruments even after receiving the auto pilot disengaged warning. Though there was no major mechanical failure and accident would have been avoided, repeated mistakes of pilots resulted in the fatal air crash which claimed 228 lives. 4. http://www.popularmechanics.com/technology/aviation/crashes/what-really-happened-aboard-air-france-447-6611877

42. Major Cause of All Four Crashes LOSS OF SITUATION AWARENESS

43. What is situation Awareness (SA)? • Situation awareness is the ability to perceive information from the environment, understanding the meaning and predicting future events based on these factors.(Endsley,1995)

44. Limitations • A number of these systems try to detect conflicts but do not resolve them. • Existing inflight systems overload pilot with information on top of already existing complex system.

45. What has not been done Some of the key questions that need to be addressed in SA research should include: • How to monitor a pilot’s actions? • What parameters are needed? • How to translate an action into cognition?

46. Monitoring cognition , attention Cognition and attention can be monitored by observing: • Brain waves • Heart rate • Body temperature or • Eye movement • Facial expression Eye movement tracking and facial expression monitoring can be done non-intrusively, but all other methods require devices to be attached to the human body.

47. Experimental Setup X-plane Flight simulator setup faceLAB5 Eye tracker

48. Sample data collection

49. System Diagram

50. Flight Instrument Scan • It is observed from accident databases that inappropriate instrument scan have led to fatal flight accidents. • Instrument scanning skills are very important for instrument pilots. • Cross -checking the readings of all the instruments together to make a collective decision is an important aspect of flight instrument scan in instrument flying. • As a result any single instrument is not fixated for longer than a defined threshold. • Federal Aviation Administration(FAA) has classified instruments as primary and supporting instruments for each flight maneuver.