Using Program Generation to Support V&V in the Next Generation Air Transportation System

1. Using Program Generation to Support V&V in the Next Generation Air Transportation System Ewen Denney (joint work with Xavier Bouyssounouse Johann Schumann Karen Tung-Cate) August, 2007

2. NGATS • US air transportation system (airports and airspace) rapidly reaching capacity • Major effort to develop integrated solutions for a safe, efficient, and high-capacity airspace system • Dynamic Airspace Configuration • Traffic Flow Management • Separation Assurance • Safety analysis

3. Trajectory Prediction Trajectory Prediction predicts the future progress of individual aircraft on the basis of • the current aircraft state, • estimates of pilot and controller intent, • expected environmental conditions, and • models of aircraft performance and procedures • What is a trajectory? • Actual flight path (“ground truth”) seldom known • Tracking data: radar, altitude, weather • Intent: flight-plan (pilot/ATC), way points, jet-routes • Route constraints

4. TP Client Applications • Flight Plan Distribution and Sector Loading: prediction of the Center Sequence (i.e., Air Traffic Control facilities to be transited) • Flight Data Processing Systems: estimation of hand-over conditions, collision avoidance • Basic Decision Support Tools: monitor flight plan and tracking data to advise controllers of suggested metering/sequencing and potential conflicts • Advanced Decision Support Tools: directly suggest solutions (e.g., instructions for conflict resolution, accurate prediction of cruise-descent and climb-cruise profiles in presence of tactical maneuvers)

5. Sources of Uncertainty • Data errors: incomplete, inaccurate, corrupted or untimely data (e.g., flight plan and radar track data inputs, airspace/navigation database) • Modeling errors: simplifications, omissions and uncertainty in the mathematical models used in the system (e.g., aircraft performance modeling database) • Intent errors: unpredictability of pilot and controller intent

6. Tracking Database • Trajectory info in NAS • Track reading (radar position and barometric altitude): noisy measurement of actual AC movement • Flight plan: “intended path” of the AC • Initial flight plan • Amendments to flight plan (often not in system) CM_SIM database assembles aircraft trajectory data from multiple noisy sources: raw, cleaned, and derived data, weather, pilot intent, etc.

7. Tracking Data • AC_DATA: radar tracking data • Noisy radar readings (every 12 secs) • ADD_FLIGHT_PLAN: flight plan data • AMEND_FLIGHT_INFO: additional intent data • weather and waypoint data are separate AMEND_FLIGHT_INFO 15 FDX1169/AFW.0015 000000002000 -NS- F -NS- AFW/1043 GRABE/1049 OKMX/1049 GRABE012038/1052 TIKYS260015/1055 EAKER/1057 ATOKA313002/1059 MLC304033/1104 LOBOE085022/1104 LOBOE058028/1106 69194/1106 69195/1107 TUL200033/1107 69184/1109 TUL/1111 ADD_FLIGHT_PLAN 15 AAL8/PHNL.0016 ZFW AAL8 0016 -NS- -NS- 1114 H/B763/QHNL./.GATES247077..3254N/9702W..KDFW TXO221069 1100 390 455 -NS- ESTIMATED_FP ATC_ARRIVAL AC_DATA 52 BKA302/ATL.0002 806.6335 540.9157 333850.00N 0913712.00W 28100 0.00 400.65 0.00 259.26 0.00 41.29 1 -NS- 28 N N F F ZFW 0.0000 403.0000 259.2621 AC_DATA 52 AAL8/PHNL.0016 217.1337 539.1776 333634.00N 1032302.00W 39000 0.00 450.52 0.00 97.61 0.00 52.69 2 93 93 N N F F ZFW 0.0000 510.0000 97.6074 ASSIGN_CONFIG 15 AAL8/PHNL.0016 CM KDFW SOUTH_4_VFR

8. Trajectory Synthesis • Mature discipline with complex algorithms to • integrate equations of motion, aircraft models • analyze weather • interpret pilot intent • Can represent trajectories using Generalized Altitude Profile • Domain-specific language for trajectories: GenAlt • Input request constrains possible trajectories • TS tool outputs parametric trajectory and point sequence of states

9. V&V Challenge • TS tool highly safety-critical but difficult to test • Complex algorithms • Lots of domain knowledge • Many different “kinds” of trajectories • Many different “kinds” of requests • Huge database of raw data • US national airspace tracking data • Weather data • Aircraft manufacturer models • Field tests • Need flexible data selection and analysis mechanism • Want to write analysis queries at high-level • Goal: • develop DSL for trajectory analysis • use program synthesis to generate analysis code

10. V&V Goals • Validate TS algorithm • Test with representatives and edge-cases of important trajectory classes • Populate trajectory database (based on tracking data, etc.) • Contractor building database infrastructure • Use for testing/assessment of trajectory prediction algorithms • Determine faulty assumptions in models • Compare trajectory predictors • Initial analysis techniques • Clustering wrt. • Point data • Trajectory data • Change-point detection • Decompose trajectories into segments • Can use as features for clustering

11. Clustering • Given: data-points that originate from 3 different sources/processes • Data points are: • iid • Gaussian distributed Problem: you don’t see the colors

12. Clustering • Clustering task: Estimate the parameters µ, such that P(x| µ,) is maximal Clustering also produces most likely class-assignment (“coloring”) of data points and class frequency . Usually, an iterative machine-learning algorithm (EM, k-means) is used

13. AutoBayes • Customizable Bayesian data analysis code generator • Synthesizes custom algorithms • C/C++, Matlab, … • Different model domains: MLE, MAP, mixture models, change-point detections, etc. • Different PDFs • Different algorithms (EM, k-means, kd-trees) • Analysis or sampling (Monte Carlo simulation) • Can be extended with new algorithms • Input is statistical model of analysis problem

14. Example Data Model • Specify: • num. classes • num. datapoints • distribution of data • goal function

15. Trajectory Features • Each trajectory (i.e., sequence of AC locations over time) can be characterized by a set of scalar features: • % in level flight, climbing, descent • Endpoints • Min distance to other AC • Min/max/mean v/speed, g/speed, g/accel, Mach/CAS/TAS • Diff raw and cleaned data • Turning maneuvers: • Total, max, sum • Altitude and heading • Could smooth wrt. turbulence

16. Clustering Analyses • Simple clustering • Cluster into disjoint sets of ACs: each represents set of AC trajectories • Cluster into disjoint time segments: each represents all AC info during that segment • Cluster into disjoint sets of ACs at specific time point • Cluster on • Tracking data • GenAlt requests • Synthesized trajectories

17. Example Analysis • Approx. 300x300nm around DFW • Approx. 1 hour of data • 215 AC trajectories • Cluster wrt. several features • Blue and cyan are landing and approach to DFW

18. Example: Endpoints • DFW forms one cluster (cyan) • Other points not obviously clustered • through flights • cut-off by time

19. Example: Close AC-distance • AC in descent to/departure from DFW: small distance to airport (?) (d~1.2nm) • En route flights which exhibit small minimal distances to other AC (d~0.97nm)

20. Change-point Detection • Data analysis technique for time series • Detect (rapid) changes in • Intensity • Noise • etc.

21. Mach/CAS Transition • Mach number • Velocity relative to sound (M = v/a) • Depends on altitude • Calibrated airspeed (CAS) • Indicated airspeed, corrected for errors • Mach/CAS transition point • During ascent: constant CAS then constant Mach

22. Climb/Descend Profiles mach • Accurate AC models for climb profiles are necessary for accurate trajectory prediction • Mach/CAS profiles can be extracted from radar trajectory data • Convert ground speed to Mach/CAS profile • Cluster according to Mach/CAS values for different aircraft types and maneuvers • Change-point detection in climb/descend maneuvers cas alt Generate code to find transition points: Mach/CAS, climb/cruise, …

23. Finding Transition Points const nat N, t_0; const double sigma_sq double a,b,c. data double mach(0..N-1). mach(t) ~ N( if(T < t_0)? a+b*t : c, sigma_sq). max pr(mach|{a,b,c}) for {a,b,c}. • Pre-process to find “climbs” • Find where transition from slope to constant occurs • Noisy data • Should also find best Mach/CAS transition a = 0.69428 b = 0.0072091 c = 0.44532 t_0 = 35 Red: actual trajectory Blue: estimated profile

24. Next Steps • Characterize important trajectory features • Generate test cases for TS from clusters • Combine extracted transitions and parameters with other features • Covariance analysis wrt. AC types • Extension of statistical model by domain knowledge (“priors”) • e.g., airport locations • Incorporation of other data • flight plans • weather data (e.g., for Mach/CAS) • Relating physical trajectories (tracking data), synthesized trajectories, and TS requests • Recursive clustering algorithms (large data sets)

25. Longer Term • Domain-specific language to describe • analysis types • trajectory features • data filters • weather conditions • changes of flight plans • specific airports, flight types, flight segments • Integrate generator with • trajectory database • TS tool