Selection of Behavioral Parameters: Integration of Case-Based Reasoning with Learning Momentum

1. Selection of Behavioral Parameters: Integration of Case-Based Reasoning with Learning Momentum Brian Lee, Maxim Likhachev, and Ronald C. Arkin Mobile Robot Laboratory Georgia Tech Atlanta, GA This research was funded under the DARPA MARS program.

2. Integrated Multi-layered Learning • CBR Wizardry • Guide the operator • Probabilistic Planning • Manage complexity for the operator • RL for Behavioral Assemblage Selection • Learn what works for the robot • CBR for Behavior Transitions • Adapt to situations the robot can recognize • Learning Momentum • Vary robot parameters in real time THE LEARNINGCONTINUUM: Deliberative (pre-mission) . . . Behavioral switching . . . Reactive (online adaptation) . . .

3. Motivation • It’s hard to manually derive behavioral controller parameters. • The parameter space increases exponentially with the number of parameters. • You don’t always have a priori knowledge of the environment. • Without prior knowledge, a user can’t confidently derive appropriate parameter values, so it becomes necessary for the robot to adapt on its own to what it finds. • Obstacle densities and layout in the environment may be heterogeneous. • Parameters that work well for one type of environment may not work well with another type. • A solution is to provide adaptability to the system while remaining fully reactive.

4. Context for Case-based Reasoning (CBR) • Spatial and temporal features are used to select stored cases from a case library. • Cases contain parameters for a behavior-based reactive controller. • Selected parameters are adapted for the current situation. • The controller is updated with new parameters that should be more appropriate to the current environment.

5. CBR Module Spatial Feature Matching Temporal Feature Matching Feature Identification Random Selection Process Case Library Sensors Case Switching Decision Case Application Case Adaptation

6. Context for Learning Momentum (LM) • A crude form of reinforcement learning. • If the robot is doing well, try doing what it’s doing a little more, otherwise try something different. • Behavior parameters are continually changed in response to progress and obstacles. • Static rules for pre-defined situations are used to update behavior parameters. • Different sets of rules for parameter changes can be used (ballooning versus squeezing).

7. LM Strategies • Ballooning • Alter parameters so the robot reacts to obstacles at larger distances than normal to push it out of box canyon situations. • Squeezing • Alter parameters so the robot reacts to obstacles only at shorter distances than normal so it can move between closely spaced obstacles. • Example ballooning rule: if ( situation == NO_PROGRESS_WITH_OBSTACLES ) obstacle_sphere_of_influence += 0.5 meters else obstacle_sphere_of_influence -= 0.5 meters

8. LM Module Short Sensor History Situation Matching Sensors Parameter Deltas Parameter Adaptation Adapted parameters Old parameters Behavioral Parameters

9. Effects of CBR and LM When Used Separately • Reported in ICRA 2001 • Effects of CBR • Distances traversed were shorter • Time taken was shorter • Effects of LM • Completion rates were much higher for dense obstacles • Completion times were higher than those for successful non-adaptive robots

10. Why Integrate? • Want discontinuous switching + continuous searching in the parameter space. • CBR is not continuous • Parameter changes are triggered by environment changes or case time-outs. • Case library is manually built to provide only ballpark solutions for different environment types. • LM does not make large, discontinuous changes • LM may take a while to adapt to large environmental changes. • LM cannot change strategies at run time • The LM strategies of ballooning and squeezing are tuned for different environments.

11. Currently Used Behaviors • Move to Goal • Always returns a vector pointing toward the goal position. • Avoid Obstacles • Returns a sum of weighted vectors pointing away from obstacles. • Wander • Returns vectors pointing in random directions. • Bias Move • Returns a vector biasing the robot’s movement in a certain direction (i.e. away from high obstacle densities), and is set by the CBR module. • Only used when CBR is present.

12. Adjustable Behavioral Parameters • Move to goal vector gain • Avoid obstacle vector gain • Avoid obstacle sphere of influence • Radius around the robot inside of which obstacles reacted to • Wander vector gain • Wander persistence • The number of consecutive steps the wander vector points in the same direction • Bias Move vector gain • Bias Move X, Bias Move Y • These are the components of the vector returned by Bias Move

13. Integration Base System Actuators Core Behavior-Based Controller Sensors Behavioral Parameters

14. Integration Addition of CBR Module Actuators Core Behavior-Based Controller Sensors CBR Module Behavioral Parameters Updated Parameters

15. Integration Addition of LM Module Actuators Core Behavior-Based Controller Sensors CBR Module Behavioral Parameters LM Module Updated Deltas and Parameter Bounds Updated Parameters

16. Simulation Setup • Heterogeneous Environments • varying obstacles density, order, and size • 350 x 350 meters • Homogeneous Environments • even obstacle distribution • random obstacle placement and size • two environments with 15% density and two environments with 20% density • 150 x 150 meters

17. CBR-LM in Simulation

18. Simulation Results For a Heterogeneous Environment

19. Simulation Results For a Heterogeneous Environment

20. Simulation Results For a Homogeneous Environment

21. Simulation Results For a Homogeneous Environment

22. Simulation Observations • Beneficial Attributes of CBR are Preserved. • We see quick, radical changes in behavior. • Time taken is about the same as CBR only. • Beneficial Attributes of LM are not always apparent. • Results can probably be attributed to a well-tuned case library. • If the case library is good enough, LM should not be needed.

23. Physical Robot Experiments • RWI ATRV-Jr robot • Forward and rear LMS SICK laser scanners • Odometry, compass, and gyroscope for localization • Straight-line start to goal distance of about 46 meters • Outdoor environment with trees and man-made obstacles • CBR-LM, CBR, LM, and non-adaptive systems were compared • The squeezing strategy was used in the LM-only experiments. • Data was averaged over 10 runs per adaptation algorithm

24. Outdoor Run

25. Physical Experiments Results • All valid runs were able to reach the goal. • Both CBR and LM beat the non-adaptive system. • The CBR-LM integrated system gave the best performance.

26. Difference From Simulation • CBR-LM outperformed CBR on the physical robot more than in simulation. • The case library for the real robot may not have been as well tuned as the simulation library.

27. Conclusions • A performance increase is not guaranteed. • For a well-tuned case library, there may be little for LM to do. • Integration of CBR and LM can result in a performance increase • observed up to 29% improvement in steps over CBR • Benefits of LM are more likely to be apparent when the CBR case library is not well-tuned (which is likely to be the case for real robots.) • LM could be used to dynamically update the case library with better sets of parameters.