An Extensible Platform for Interactive, Entertaining Social Experiences with an Animatronic Character

1. The Entertainment Technology Center Presents: An Extensible Platform for Interactive, Entertaining Social Experiences with an Animatronic Character Will Bosley, Dave Culyba, Sabrina Haskell, Andrew Hosmer, TJ Jackson, Shane Liesegang, Seema Patel, Christine Skarulis, Peter Stepniewicz, Jim Valenti, Salim Zayat, and Brenda Harger Carnegie Mellon University August 1st, 2005

2. OUTLINE • Motivation • Goals • Related Work • Hardware • Robot • Kiosk • Software • Control Software • Content Authoring Tools • Rapidly Developed Content • Future Work

3. MOTIVATION • What is the Interbots Initiative? • Recent Explosion in Consumer Robotics Industry • Popularity of AIBO and Robosapien demonstrate that Entertainment Robotics is a viable Industry • Unlike traditional theme park animatronics, novelty is in interactivity and unique experience

4. GOALS • Leverage the success of household entertainment robotics to introduce interactivity and uniqueness to higher end custom animatronics • Develop a fully interactive, autonomous animatronic character that is both believable and entertaining • Give non-technologists the ability to author content for interactive animatronic experiences • Develop an extensible platform that allows artists to rapidly design and create complete, entertaining social experiences with animatronic characters

5. RELATED WORK • Within academia, most similar to that of Breazeal et al • Animatronic robot Leonardo centers around human-robot collaboration and learning • Our robot, Quasi, is influenced by behavioral, emotional, and environmental motivators, but does not replicate artificial intelligence • Interactive Robot Theatre featuring an anemone-like creature • Work differs in depth of interactivity • Within industry, most similar to Disney Animatronics • Lucky the dinosaur • In many ways, we attempt to bridge academia and industry

6. THE IBI PLATFORM • In 14 weeks a team of 6 interdisciplinary graduate students created: • A believable animatronic character named Quasi • A suite of software tools that allow for the development for interactive social experiences with Quasi • The platform allows for experiences to be developed quickly and easily. No programming or technical knowledge is required.

7. Quasi the Robot HARDWARE Quasi’s Kiosk

8. FABRICATION

9. QUASI’S HARDWARE

10. KIOSK HARDWARE

11. THE SOFTWARE • Control Software • Character State Control System (CSCS) • Real-time Show Control System (RSCS) • cVision, Inter-Process Communication (IPC), iButton Reader, MIDI Reader, Babble • Content Authoring Tools • Behavior Authoring Tool (BAT) • Maya Plug-In • Macromedia Director & Flash

12. CONTROL SOFTWARE • Character State Control System (CSCS) • Backend run-time decision mechanism for a character • Depends on set of system inputs and network of related states and transition rules (essentially a FSA) • This behavioral model is stored in a database format • Inputs are split into internal “factors” and external “facts” • Facts represent knowledge acquired from the character’s environment • Direct (e.g. distance reading from IR sensor) • Indirect (interpreted, e.g. “someone’s standing in front of me”) • Factors represent character’s internal environment • Emotional State and Motivations (happiness, aggression, loneliness, cleanliness, attention)

13. CONTROL SOFTWARE • A character’s behavioral database is (at the high level) a system of “superstates” • Original Quasi personality included Idle, Sleep, Greeting, Farewell, and GiveCandy superstates • Each superstate (except the default Idles) has a set of entry requirements • Superstate is entered when facts and factors meet entry requirements • Superstates also have “stay requirements” • Superstates are not connected, when stay requirements aren’t met CSCS finds all superstates that match entry requirements and randomly picks one, weighted by priority

14. CONTROL SOFTWARE • Within each superstate is a network of substates • Traditional FSA, substates are connected by transitons • Upon entry to a substate: • Adjustments are made to internal factors • Actions take place (e.g. animations, lighting changes, sound, candy dispensing, video display, etc.) • After adjustments, all exit transitions considered. If none, CSCS exits current superstate.

15. CONTROL SOFTWARE • Real-Time Show Control System (RSCS) • Actual software interface to hardware • Receives message from CSCS specifying which animation file to load, which DMX channels to play them on, and any sound files that need to be broadcast • Sends DMX data to servos, and any other DMX controlled hardware • Has ability to linearly blend animations together, and seamlessly transition from one animation to another

16. CONTROL SOFTWARE • cVision • Face tracking tool; implementation of Intel’s OpenCV face tracking library • Inter-Process Communication (IPC) • Message passing protocol that utilizes Inter-process Communication Library developed in 1994 at CMU by Reid Simmons • MIDI Reader • Listens to input signals from sensors, converts them to IPC messages and transmits them to CSCS • iButton Reader • Allows Quasi to uniquely identify people through log-in • Babble • Implementation of open source Sphinx voice recognition package developed at Carnegie Mellon. Enchanced to pass messages over IPC.

17. CONTENT AUTHORING TOOLS • Goal: • Create content authoring tools that are so intuitive and easy to use that no prior programming or technical skills are required • Accomplished By: • Leveraging power of popular mass-market software that artists are already familiar with • Creating custom, graphical authoring tools

18. CONTENT AUTHORING TOOLS • Behavior Authoring Tool • One of the most powerful components of IBI Platform • Used prior to run-time • Graphical application for composing networks of substates and superstates • Written in Macromedia Flash, interfaces to database through SQL commands • Provides visualization of each superstate, making it easy to understand complete picture of a character’s personality

19. CONTENT AUTHORING TOOLS Superstate Panel

20. CONTENT AUTHORING TOOLS • Maya Plug-In • Industry standard application for 3D modeling and animation • Natural choice for opening up animation of animatronic characters to 3D animatiors • Plug-in is a MEL script that converts keyframes to DMX values for appropriate DMX channels; values are stored in a file that is read in by RSCS and sent to servos • Not limited to joint rotations– can control anything in robot’s environment that can talk to DMX and is keyable to some attribute in Maya: lights, colors, etc. • Also created Animation Editor Tool (AET) • For those unfamiliar with Maya • Consists of a series of sliders for controlling joints and lights

21. CONTENT AUTHORING TOOLS Animation Editor Tool

22. RAPIDLY DEVELOPED CONTENT • Usasbility of Content Authoring Tools put to the test in Fall 2004 Building Virtual Worlds course at Carnegie Mellon • Two week project cycles • Interdisciplinary teams of 4-5 students (1 programmer, 1 texture painter, 1 3D modeler, 1 designer, and possibly 1 audio artist) • None of the students had prior experience with IBI Platform • Tools met expectations admirably– all teams were able to rapidly develop complete, entertaining experiences

23. Quasi’s Interactive Dreams Addressed problem of how to keep Quasi engaging while allowing his motors to “rest” Quasi’s Photo Kiosk Allows guests to have their picture taken with Quasi E-mails picture to guest Adds picture to album Displays Quasi’s life story RAPIDLY DEVELOPED CONTENT

24. RAPIDLY DEVELOPED CONTENT • Quasi-oke • Finale of Fall 2004 BVW show, a spot typically reserved for the best piece of the semester

25. FUTURE WORK New portable cable driven robot

26. FUTURE WORK Guided Performance Interface

27. FUTURE WORK • Virtual Show Control System • Sends DMX Signals to 3D model in Panda3D instead of RSCS • Character Authoring Tool • Expands BAT to allow for plug and play hardware

28. ACKNOWLEDGEMENTS • Brenda Harger • Todd Bowers • Zac Pavlov • Dan Kanitz • The ETC • Art Institute of Pittsburgh

29. QUESTIONS? • For more information please visit: • www.interbots.org • www.etc.cmu.edu