Autonomous Vehicle Verification Challenges, State-of-Art, and Testing Framework

1. Autonomous Vehicle Verification Challenges, State-of-Art, and Testing Framework Advanced Mobility Insitute Dr. Rahul Razdan Flpolyami.com

2. Florida Poly Introduction Population • Total number of students is 1378 • 1362 Undergraduate, 15 Grad • 94.8% in-state residents • 21 states and 16 countries represented Quality • Average class size of 25 • Student-to-faculty ratio of 18:1 • FTIC retention of 72% Florida Polytechnic was created as a statewide institution

3. Road at peak throughput only 5% of the time… ...and then only 10% covered with cars Productive use 1% Energy used to move the person Inertia Aerodynamics Rolling resistance Auxiliary power Transmission losses 86% of fuel never reaches the wheels Engine losses More than 33,000 road fatalities in US $300B annually in cost Idlig 0.8% looking for parking 0.5% sitting in congestion >95% Caused by human error 2.6% driving The typical American car spends 96% of its time parked Why is Autonomous Important ? Transport Today: 99% Waste of $3/Mile [nauto]

4. AV and Shared Mobility Solutions to Many Problems

5. Perception Decision Action Sensor 1 Vehicle Control Sensor Fusion & Tracking Sensor 2 Threat Assessment Decision Making Driver Interaction Sensor n AV Technology Structure

6. State of Art: AV Sensors Autonomous Requirement Rating:H = High,M=Medium,L=Low Radar LiDAR Camera

7. State-of-Art: Regulation

8. AV Safety Regimes Med/High Velocity Examples: country roads, suburban intersections Level 3 Level 2 Level 1 High Velocity Limited Access Situations Examples: Turnpike, Interstates Low Velocity, Sensor Feedback System Sufficient Examples: Planned Communities, third-world Countries Complexity of the Situation

9. AV Verification Challenges

10. Current Verification Paradigm • Characterization, model, simulate, verify : sim/model test track time (1991 NYT) • What works well ? Well characterized situations. • What does not ? Global effects, reliability, final test ➔ major role of test tracks

11. Theoretical Analysis Augmented Data Replay Model/Software- In-the-Loop Efficiency Real Traffic Data Replay Offline Hardware- In-the-Loop Online Real Traffic Test Track Vehicle Hardware- In-the-Loop ProcessControllability State-of-Art: Verification Methods

12. Theoretical Analysis Augmented Data Replay Model/Software- In-the-Loop Efficiency Offline Real Traffic Data Replay Online Hardware- In-the-Loop Real Traffic Vehicle Hardware- In-the-Loop Test Track Sensor Data Realism State-of-Art: Verification Methods

13. Real World Closure Velocity Issue Research (Learning) Theory, Simulation Known-Known Testing Real World (Learning) Test Track, Controlled Real World Known-Unknown Testing Real World (Production) Test Track (Diagnosis) Test Case Acceleration (must be faster than real world) Unknown-Unknown Testing

14. OEM

15. Tier 1

16. Sensors/Semi

17. Development Tools

18. The BI Intelligence Research Team

19. Open Verification Challenges

20. Advanced Mobility Institute • Research Institute: • Focus Test and Verification for AV technology • Local Partners: Suntrax, JTA, CF • Global Partners: Mathworks, NI, Ansys, etc • Research Thrusts: • Rare Scenario Generation and Test • Sensor/Object Recognition Verification • EMC Interference Issues • Human/Machine Communication • Transportation OS

21. Multi-Disciplinary Team Dr. Razdan Dr. Sargolzaei Dr. Ala J. Alnaser Dr. Akbas Dr. Vargas Dr. Alsweiss Gen Polumbo Dr. Sahawneh

22. AV Verification Architecture Test Seeding and Coverage Key Methods • Abstraction and Separation of Concerns • Pseudo-Random Test Generation • Assertions and Coverage Analysis Focus Separation of Concerns • Sensor reliability/calibration at component level • Object recognition at higher level • Planning and Decision Making Focused • Physics with Real-Time Automotive Controls ABSTRACT SCENERIO MODEL (ASM) Control and Assertion Abstractions Baseline Newtonian Physics

23. Scenario Abstraction Architecture ScenarioSimulation Constraints Scenario Test Generation Industry Regulations Scenario Database - Converge Matrix Abstract Scenario Model (ASM) Scenario Test Track Scenario Abstraction from Real Life Test Track Diagnostic

24. Implications of Model

25. Implications of Model

26. Florida SunTrax Florida Turnpike/Florida Poly Collaboration: • Co-located near Poly Campus (2.25 mile trace, 400 acres) • Turnpike Testing Tolling Technology • Infield Focused on Autonomous Research • Architected for Industrial Zone Expansion and Economic Development Engine • Breaking ground 2017 and finishing 2019 Florida Trend (Oct 2016): FDOT announces partnership with Florida Polytechnic University to develop SunTrax test facility

27. US DOT AV Proving Grounds Central Florida Partnership: • City of Orlando, FDOT, Turnpike, NASA • One of Ten Proving Grounds for AV • UCF (simulation), Poly (AV Testing) Research Efforts • Multi-purpose: urban, rural, restricted • Multi-user: tourist, language, commuter US DOT (Jan 2017): U.S. Department of Transportation Designates 10 Automated Vehicle Proving Grounds to Encourage Testing of New Technologies

28. “Rare” Scenario Generation and Test • Accelerate AV learning/verification with 3 sigma • Basis for Signoff Framework for Regulators But my AV worked in Michigan and California ?

29. Sensor/Object Recognition Verification • Environment testing of Sensor/Obj Systems But my AV worked in Arizona ?

30. EMC Interference • V2V, V2I, Environment, and Weather EMC • Many radios, EMC impacts safety systems Storm didn’t know DSRC is reserved ?

31. Human/AV communication • Humans have a driving language, AVs need it That AV drives like an mad old lady !

32. Transportation OS • Transportation System Inefficient • Use Market Economics and Network Ideas What if we could dynamically use roads ?

33. Public Transport Potential Impact: • 24x7 • Reuse of Infrastructure (utilization) • On-call Access Simplifications: • Limited Routes (lanes) • Manage Intersections and Pedestrians • Lower Speeds

34. Logistics Potential Impact: • Warehouse Operations • Outdoor Logistics Operations • Long Haul and Last Mile Simplifications: • Controlled Environments • Human Leverage (“follow-me” model)

35. Planned Communities Massive Impact: • Enable driverless elderly communities • Increase Safety and Health Simultaneously • More Efficient Architectural Designs Simplifications: • Lower Speeds • Controlled Environments • Less Complex Interaction Models

36. Agriculture Potential Impact: • 24x7 Operation • Higher Level of Capability (ex fruit picking robots) Simplifications: • Lower Speeds • Less Complex Interaction Models

37. Conclusions • AV is a game-changer (access, economics, environment) • AV Test and Verification is critical unsolved problem ! • State-of-Art built for “action” new capabilities required for “perception” and “decision-making” aspects of AV • Proposed well-defined Abstract Scenario Model (ASM) • Enables Regulators to Communicate with OEMs • Enables Backwards Compatibility • Enables tests of kwn-unkwn and unkwn-unkwn issues. • Enables decomposition of decision-making and perception

38. THANK YOUwww.flpolyami.com

39. Backup

40. AV and Shared Mobility Solutions to Many Problems

41. AV and Shared Mobility Solutions to Many Problems

42. AV and Shared Mobility Solutions to Many Problems

43. AV and Shared Mobility Solutions to Many Problems

44. AV and Shared Mobility Solutions to Many Problems

45. Process Vehicle Driver Environment Active Safety System Technology Architecture