From Teleoperation to Autonomy

1. From Teleoperation to Autonomy • Define Intelligent Robot • Be able to describe at least two differences between AI and engineering approaches to robotics • Be able to describe the difference between telepresence and semi-autonomous control • Have some feel for the history and societal impact of robotics History -AI -Engineering Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

2. Intelligent Robot • Mechanical creature which can function autonomously • Mechanical= built, constructed • Creature= think of it as an entity with its own motivation, decision making processes • Function autonomously= can sense, act, maybe even reason; doesn’t just do the same thing over and over like automation History -AI -Engineering Teleop Case Studies Programming Summary Review Chapter 1

3. What are Robots? • Autonomous mechanical creatures • Capek 1921: R.U.R. • Intelligent because teleoperation doesn’t work, doesn’t scale • Physically situated, but now software agents or softbots • Principles from robotics influenced AI community, esp. planning • Combines programming, networks, operating systems, algorithms, … everything about CS into a system (the ultimate software engineering project) History -AI -Engineering Teleop Case Studies Programming Summary Review www.fradulent.org/rur.htm Chapter 1

4. Robots Constantly in the Press History -AI -Engineering Teleop Case Studies Programming Summary Review www.sony.com courtesy of MIT AI Lab www.irobot.com courtesy of Honda Chapter 1

5. ½ iRobot PackBot Less Famous Cousins at WTC Inuktun microTracks Chapter 1

6. Why Robots? Dirty, Dangerous, Dull Tasks • JV2010, TRADOC, JFCOM, all branches even down to the organic level • Reconnaissance, MOUT, denial of area, consequence management, logistics, demining History -AI -Engineering Teleop Case Studies Programming Summary Review www.friendlymachines.com Replace Humans with Robots Chapter 1

7. Why Robots? Better Than Bio • Robots at WTC… • voids smaller than person could enter • voids on fire or oxygen depleted • NBC Response • Lose ½ cognitive attention with each level of protection • Level A=12.5% of normal ability Void:1’x2.5’x60’ History -AI -Engineering Teleop Case Studies Programming Summary Review Void on fire Do Things that Living Things Can’t Chapter 1

8. Major Robot Modalities: UAV, UGV, UUV • Unmanned Aerial Vehicles • drones since Vietnam: Global Hawk, UCAV • easy: nothing to hit • hard: mission sensing, human-in-the-loop control • Unmanned Ground Vehicles • since 1967 • easy: can always stop and think, a priori maps • hard: perceiving, e.g., light vegetation vs. wall • Unmanned Underwater Vehicles • ROVs since 1960s • easy: run tethers • hard: platform operation in unfriendly environment History -AI -Engineering Teleop Case Studies Programming Summary Review Mobility (platform), Perception, Communications +HRI, Control (Intelligence), Power Chapter 1

9. A Brief History… Chapter 1

10. Industrial Manipulators • “Tommy” type of robots: deaf, dumb, and blind • High precision, fast repetition • Usually no sensing of the environment • Welding can be off by an inch… History -AI -Engineering Teleop Case Studies Programming Summary Review Chapter 1

11. 3 Ways of Controlling a Robot • “RC-ing” • you control the robot • you can view the robot and it’s relationship to the environment • ex. radio controlled cars, bomb robots • operator isn’t removed from scene, not very safe History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

12. 3 Ways of Controlling a Robot • teleoperation • you control the robot • you can only view the environment through the robot’s eyes • don’t have to figure out AI History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

13. 3 Ways of Controlling a Robot • semi- or full autonomy • you might control the robot sometimes • you can only view the environment through the robot’s eyes • ex. Sojouner with different modes • human doesn’t have to do everything History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

14. Remote Local Sensor Communi- cation Display Mobility Control Effector Power Components of a Telesystem(after Uttal 89) • Local • display • Local control device • Communication • Remote • sensor • mobility • effector • power Chapter 1

15. Example History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Remote Local Chapter 1

16. Typical Run History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

17. Problems That You Saw • no feedback, couldn’t really tell that the robot was stuck but finally got free • robot doesn’t have “proprioception” or internal sensing to tell you what the flippers were doing. No crunching noises, no pose widget to show the flippers • no localization, mapping-> no idea how far traveled • partial solution: better instrumentation (but can’t do dead reckoning well) • operator doesn’t have an external viewpoint to show itself relative to the environment • solution: two robots, one to spot the other • communications dropout, even though ~3 meters away • lighting conditions went from dark to very bright • hard for computer vision or human to adjust History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

18. But good for unmodeled events History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

19. Communications is Important:DarkStar+7 seconds=DarkSpot • 7 second communications lag (satellite relay) • “interruption” lag on part of operator History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

20. Predator:~7:1 human to robot ratio • 4 people to control it (52-56 weeks of training) • one for flying • two for instruments • one for landing/takeoff • plus maintenance, sensor processing and routing • lack of self-awareness– in Kosovo, come along side in helicopter and shoot down History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Leo’s unofficial Predator page Chapter 1

21. Summary of Teleop Problems • cognitive fatigue • communications dropout • communications bandwidth • communications lag • too many people to run one robot (hidden cost) History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

22. Telesystems Best Suited For: • the tasks are unstructured and not repetitive • the task workspace cannot be engineered to permit the use of industrial manipulators • key portions of the task require dexterous manipulation, especially hand-eye coordination, but not continuously • key portions of the task require object recognition or situational awareness • the needs of the display technology do not exceed the limitations of the communication link (bandwidth, time delays) • the availability of trained personnel is not an issue History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

23. Teleop Improvements: Telepresence • Telepresence • improves human control, reduces simulator sickness and cognitive fatigue by providing sensory feedback to the point that teleoperator feels they are “present” in robot’s environment • increases demands on bandwidth History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

24. Teleop Improvements:Supervisory Control • Semi-autonomous • Supervisory Control • human is involved, but routine or “safe” portions of the task are handled autonomously by the robot • is really a type of mixed-initiative • Shared Control/ Guarded Control • human initiates action, interacts with remote by adding perceptual inputs or feedback, and interrupts execution as needed • robot may “protect” itself by not bumping into things • Traded Control • human initiates action, does not interact History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

25. Teleop Improvements:Mixed-Initiative • Levels of Initiative • do only what told to do (teleoperation) • recommend or augment (cognitive augmentation) • act and report • act on own and supervise itself (autonomy) History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

26. “No Hands Across America” • 1994 • CMU NavLab • Pittsburgh to San Diego • 2897 miles total • 2849 autonomously • Autonomous or Mixed-Initiative? History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

27. Level of Initiative teleoperation cognitive augmentation act and report act on own Application janitorial robot medical robot (telemedicine) high flying surveillance drone combat aerial vehicle resupply system for bringing water to fire fighters guard dog robot “nurse” robot Mixed-Initiative Matching Game History Teleop -Motivation -Components -Problems -Alternatives Case Studies Programming Summary Review Chapter 1

28. Collaborative Teleoperation 1 3 mpg: June 2, 2000 SRDR Miami Beach: view from Inuktun as it falls mpg: June 2, 2000 SRDR Miami Beach: view from Inuktun from hoisted position 2 • Urban is stuck, Inuktun can’t help from current perspective • Driven off 3rd floor • Hoisted to 2nd floor by tether • Has better view, changing configuration & rocking extend view still: June 2, 2000 SRDR Miami Beach Chapter 1

29. 2000 AAAI Mobile Robot • 2 robots helping each other reduced collision errors, sped up time navigating confined space, righting History Teleop Case Studies Programming Summary Review Chapter 1

30. Example:Mixed-Initiative & Collab. Teleop • 9/2000 DARPA Tactical Mobile Robots demonstration • Robot used an intelligent assistant agent to look for signs of snipers hiding in urban rubble • motion • skin color • difference in color • thermal (IR camera) • Human navigated mother robot using viewpoint of 2nd robot (not in picture) • Once deposited the human moved the daughter robot, and either saw a sniper or was alerted by the agent History Teleop Case Studies Programming Summary Review Chapter 1

31. AI provides the “other stuff” • knowledge representation • understanding natural langugage • learning • planning and problem solving • inference • search • vision History Teleop Case Studies Programming Summary Review Chapter 1

32. Young Frankenstein 4 specialists: 1 vehicle 1 specialist: 1 vehicle 1 specialist: n vehicles 1 specialist: 1 modality 1 specialist 1 soldier Example User Expectation of AI • Proposed Goal: 1:1 soldier:any robot,where 1 soldier is responsible for 1 or more active robots but does not have to pay continuous attention to them. UAVs as theater assets MAVs as organic assets Field recon- figurable UUVs MAV-UGV cooperative monitoring Flocks of MAVs Chapter 1

33. More Reasonable Expections Consolidation agents with “tactical” autonomy, toolkits Vehicle success is still based on human, but robot is “in front” History Teleop Case Studies Programming Summary Review Dedicated Autonomy Systems Human intermittent attention as team coordinator, not with individuals Mass-produced dedicated agents Cooperating “pack” or “herd” agents Reconfigurable AutonomySystems Human primary responsibility as a tool builder, expert advisor. Peer-level communication Field- reconfigurable agents Chapter 1

34. Programming Notes • You always need telesystem or human intervention as a backup • at some point a human will need to take control • embed in your design • “Roboticists automate what is easy and leave the rest to the human”- Don Norman • The user interface is absolutely critical • User interface make up 60% of commercial code • Useful= is the program purpose useful? • usually given to designer via specifications and requirements • Usable= can a human use it efficiently? • designer must conduct usability studies • avoid “if I can use it, some one else will” History Teleop Case Studies Programming Summary Review Chapter 1

35. Example of How an “Internal” Display Can Hurt • gamer joystick plus laptop with video & audio • robot state: battery, comms, orientation, camera, encoders • was not used on rubble pile at WTC because it scared off rescuers: too complicated, too long to boot, too toy • now integrated with Land Warrior– used in Afghanistan History Teleop Case Studies Programming Summary Review iRobot PackBot video, FLIR, 2 way audio Chapter 1

36. Summary • Teleoperation arose a partial solution to autonomy • cognitive fatigue, high comms bandwidth, long delays, and many:one human to robot ratios • Telepresence tries to reduce cognitive fatigue through enhanced immersive environments • Semi-autonomy tries to reduce fatigue, bandwidth by delegating portions of the task to robot • mixed-initiative • Teleop isn’t simple and improvements aren’t just “better user interfaces” History Teleop Case Studies Programming Summary Review Chapter 1

37. Review Questions • What is an intelligent robot? • What is the difference between engineering and AI robotics? • What are 3 types of control? • What are the parts of a telesystem? • What are problems with teleoperation? • What’s the difference between telepresence and semi-autonomous control? • What are the levels of initiative (mixed-initiative)? • What are alternatives to traditional teleoperation? History Teleop Case Studies Programming Summary Review Chapter 1