Reinforcement Learning Traffic Control for Energy Optimization and Throughput

1. Reinforcement Learning-based Traffic Control to Optimize Energy Usage and Throughput (HPC4Mobility) Thomas P KarnowskiEmail: karnowskitp@ornl.govPhone: 865-574-5732 Oak Ridge National Laboratory National Transportation Research Center GRIDSMART HPC4Mobility Stakeholders Workshop, Oct 1 2018 Husain Aziz, Wael Elwasif, Travis Johnston, Thomas Naughton, Sean Oesch Project ID:EEMS036 This presentation does not contain any proprietary, confidential, or otherwise restricted information

2. Our work has two primary objectives: GRIDSMART horizon-to-horizon camera view • Teach a GRIDSMART camera how to estimate fuel consumption in its visual field (DATA (and HPC) FOCUSED) • Teach an array of GRIDSMART-instrumented traffic lights how to change their signal phasing & timing to improve fuel economy (HPC focused) *https://www.energy.gov/eere/vehicles/energy-efficient-mobility-systems https://www.energy.gov/eere/vehicles/articles/new-initiatives-will-use-supercomputers-improve-transportation-energy

3. This project will use computer vision sensing and reinforcement learning to develop traffic control policies that improve energy usage • Reinforcement Learning learns key actions in response to environmental states to create a solution that achieves a goal in an optimal manner • Key examples include games where playing strategies can be learned without explicit programming • Traffic control use cases have been published and achieved good results • The “state” of the environment for this work will be sensed by GRIDSMART cameras Can GRIDSMART cameras be used to sense traffic-energy states and control lights to improve overall energy efficiency?

4. GRIDSMART views horizon-to-horizon using fisheye cameras and computer vision for vehicle detection • Vehicle sizes are estimated and provided as data products for traffic engineers The fisheye lens allows for simple, often single-camera installations with low maintenance, but is a sub-optimal imager with less resolution at distances

5. Approach / Strategy : Data Focus • Use real-world imaging data and vehicle dynamic data from naturalistic driving studies to estimate fuel consumption of imaged vehicles • Build a training set of ground-level images and corresponding GRIDSMART images to perform vehicle classification with GRIDSMART devices GridSMART Camera Installation Simultaneous ground detections and GRIDSMART imagery will be correlated to develop a training set for vehicle classification by GRIDSMART cameras Ground Truth Make Model GRIDSMART capture tagged with make/model Ground sensor capture (Leverages existing vehicle detection work)

6. Data update: • Process is in place to build data set with synchronized ground sensor, vehicle tracking, and ground-truth classification and is ongoing • To understand the “art of the possible”, we are also using datasets of vehicle make & models to understand the potential and likely errors • Use baseline methods to estimate accuracy and model camera resolution • Use high-performance computing to improve the classification process

7. Teach a GRIDSMART camera to estimate fuel consumption Our process is in place to build a data set of vehicle types to teach the cameras how to estimate fuel consumption GROUND VIEW IMAGE OF VEHICLE

8. Teach a GRIDSMART camera to estimate fuel consumption Google query for image of 2012 Ford Transit Connect CLASSIFICATION OF GROUND IMAGE

9. Teach a GRIDSMART camera to estimate fuel consumption Ford Transit Connect: 28 mpg GRIDSMART view Google query for image of 2012 Ford Transit Connect Ground view

10. Baseline to estimate vehicle fuel consumption • Used pre-trained convolutional neural network* and modified it to classify vehicles by make & model and body style • Data set from Gebru (substitute while building larger GRIDSMART view training set) • Achieved classification accuracy comparable to Gebru, but also studied degradation effects of distance to vehicle Gebru, T., et al. "Fine-Grained Car Detection for Visual Census Estimation." AAAI. Vol. 2. No. 5. 2017. Krizhevsky,A. et al. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.

11. But what is the estimate of vehicle fuel consumption? • Make/model classification yields “perfect” results when the correct make/model is estimated, but degrades significantly when incorrect Choosing the mean MPG across the data set yields 6.1 RMS error Length/width linear regression: 2.8 without measurement error 5.85 with measurement error Gebru, T., et al. "Fine-Grained Car Detection for Visual Census Estimation." AAAI. Vol. 2. No. 5. 2017. Krizhevsky,A. et al. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.

12. How can we improve the fuel efficiency estimate? • Use High-Performance Computing to evolve a better MPG estimate! • Deep Learning must be tailored to an application domain, which can require months of manual effort with a significant amount of expertise • Inspired by Gregor Mendel’s pea plant experiments, MENNDL automatically creates and evolves DL neural networks that best suit an application and runs on 100% of Titan and Summit • MENNDL creates networks that typically go beyond what a human expert would have thought to try • DL network tailored to an application within hours and with very little expertise • 2018 Gordon Bell Finalist • 2018 R&D 100 Award Finalist • Scaled to 4,000 nodes of Summit and 152.5 PFlops measured, 167 PFlops projected • MENNDL could also allow trade-offs between performance and other objectives (CPU VS GPU; latency; network size; etc) Old New Evolved

13. Data progress to date: Summary • We have interfaced to ORNL GRIDSMART cameras and even improved their infrastructure: • GRIDSMART cameras now get time updates from the ORNL instrumentation network • We helped identify a broken fan on the Melton Valley GRIDSMART controller • We have completed a set of computer vision tools for segmenting and tracking vehicles • We are acquiring more data samples even now • We have established a baseline performance based on length/width as well as actual machine learning standards • And we seek to improve this using MENNDL, ORNLs HPC solution for evolving and improving deep learning classification

14. Approach / Strategy : HPC Simulation Focus • HPC simulations will be performed using historic data for traffic and GRIDSMART sensing with new computational intelligence capabilities to demonstrate adaptive signal control across large-scale urban areas • Simulations will be used to train and test reinforcement learning algorithms that optimize energy efficiency and throughput • Targeting large grids (goal: 25x25) although smaller topologies will be tested through the development phase

15. HPC Software Design • Use MPI in a grid (targeting up to 25x25) • Staten Island = initial model for BoE estimates due to ease of data availability, population • Traffic flow simulation focuses on the environment at the lights and vehicle characteristics Vehicle Type RPM Velocity / acceleration Maneuver Lane … Vehicle Type RPM Velocity / acceleration Maneuver Lane … Vehicle Type RPM Velocity / acceleration Maneuver Lane … RL policy and algorithm dictates actions based on simulated GRIDSMART “view” of world

16. HPC Simulation progress to date • Applied for Director Discretionary HPC hours and HPC4-X hours (approved) • Formulated initial HPC design using adjacency matrices to characterize grid characteristics • Created classes in Python for HPC deployment • Simulations will run in parallel with extensive logging for experimentation on parameter impacts (including baselines) • Goal: enable learning of policies across simulations • Simulations will be scoped to focus on intersections and modeling of GRIDSMART sensing

17. Summary • Relevance: Create a technology solution to increase mobility energy productivity using GRIDSMART cameras, computer vision, and reinforcement learning for adaptive signal control • Approach: • Use historic data and imagery to create methods to estimate fuel consumption of vehicles at intersection • Use these estimates of camera performance in HPC simulations with reinforcement learning to develop energy-efficient traffic control • Collaborations: GRIDSMART of Knoxville, TN • Technical Accomplishments: • Acquired historic data sets and visual data • Process for obtaining ground truth data at real intersections is in place and in progress • Adapted baseline methods for “visual fuel consumption estimation” • Using HPC (MENNDL) to evolve better visual fuel consumption estimator • Started HPC simulation design & execution • Future Work: • Complete data acquisition and characterization • Complete and execute HPC simulations • Plans for deployment with GRIDSMART National Transportation Research Center

18. BACKUP SLIDES

19. Models for fuel consumption based on vehicle classification & dynamics • Correlate vehicle body types with fuel economy estimates to create classes for visual estimates of fuel economy using acquired datasets and machine learning • Model the instantaneous fuel consumption using naturalistic driving study data from over 1 million intersection traversals and estimates of vehicle dynamics • The Second Strategic Highway Research Project (SHPR2) consists of 2 years of driving data from over 3,000 participants in the United States • Selected trip data through signalized intersections will be used to create dynamics models for fuel consumption

20. Outcomes • The key outcome of this work will be control strategies generated through HPC specific to GRIDSMART sensing technology, for integration into real-world settings • Per task outcomes: • Data set to add GRIDSMART vehicle typing capability • Fuel consumption models • HPC framework for RL + sensing technology • Simulations for performance evaluation

21. Machine Learning Design • Premise: For every data set, there exists a corresponding neural network architecture that performs ideally with that data • What’s the ideal neural network architecture (i.e., hyper-parameters) for a particular data set ? • Widely-used approach: intuition / guessing • Pick some deep learning software (Caffe, Torch, Theano, etc) • Design a set of parameters that defines your deep learning network • Try it on your data • If it doesn’t work as well as you want, go back to step 2 and try again.

22. AutoML • Developing ML often involves multiple steps • Data preprocessing • Feature engineering • Feature selection • Parameter optimization • Training & Evaluation • AutoML is the process of automating the entire end to end process

23. Software design (and concept) is based on a “prototype” project • GRIDSMART predecessor Aldis formulated original concept of fisheye lens for traffic light actuation and control with an “eye” toward energy savings • Proof of concept initial findings in 2007: • Idle time reduced through the use of vehicle classification • Performance of system improved as computer vision model improved • Used “Dyna-Q” algorithm from Sutton to update policy with a two-layer neural network based function approximation • Since then, advances in Q-learning algorithms have occurred and open source implementations are available