Introduction to AIBO Programming

1. Introduction to AIBO Programming Bogdan Raducanu Centre de Visiò per Computador E-mail: bogdan@cvc.uab.es

2. Contents • Introduction • AIBO’s anatomy • Programming Environments (Remote Framework) • Application: AIBOLive • Integration between Remote Framework and OpenCV • Configuration of MSVC++ • RoboCup 2005

3. Introduction • AIBO is the name SONY assigned to an ambitious concept to create a revolutionary generation of robots designed to become part of people’s live • By pursuing the integration between SONY’s state-of-the-art technology and latest breakthroughs in AI, the result was an intelligent companion, gradually adapting to the environment, capable of expressing emotions, very skilful and with an inherent desire to entertain • What AIBO means? • 1) friend, companion (jap.) • 2) Artificial Intelligence roBOt • - Introduced in 1999 for entertainment purposes, it rapidly has been adopted by the scientific community

4. - Why AIBO is so special? • unlike other robots, which are designed to perform a certain sequence of operations, AIBO has not a specific goal • AIBO’s behavioral patterns will develop as it learns and grows • - it is able to communicate with people by displaying some emotions and understading some verbal commands • its human-like behavior is implemented through a series of insticts and senses • since it is learning through direct interaction with the environment and the people it cohabitates with, each AIBO is unique

5. AIBO’s instincts: • affective instinct (desire to communicate with people) • search instict (desire to satisfy its curiosity) • movement instict (desire to move about) • recharge instinct (desire to find its power supply) • sleep instinct (desire to have a rest) • AIBO’s senses: • sense of touch • sense of hearing • sense of sight • sense of balance

6. AIBO’s anatomy (ERS7-M2)

7. Inside AIBO: - 576 MHz 64bit RISC CPU - Built-in Wireless LAN module - 64 MB RAM - Volume switch: set the sound speaker level to your preference - Wireless On/Off switch: activate or deactivated the Wireless LAN transmission - Lithium-ion battery - Memory Stick™ (MS) for AIBO Mind 2 software

10. To feel • - touch sensors on the back, on top of the head and chin • touch sensors on each paw – detect the type of the surface (the walk style is adapted automatically) • To see • a camera on the tip of the nose - recognize faces, favorite places and objects • recognize the CyberCode attached to the energy station pole • infrared (IR) distance sensors mounted near the camera and on the chest • the combination between the camera and IR sensors allow to have a 3D perception of the sorroundings

11. To hear • stereo microphones located on each side of the head • AIBO can detemine the sound direction • it can understand some commands • To express • matrix of LEDs on the frontal part of the head • onboard loudspeaker • wireless status LEDs - on the rear part of the ears • (blue – remotely controlled, pink – play mode, orange – clinic mode, green – has been lifted)

12. Hour (10) Afraid Happy Bone found Sad Furios Clinic mode Displaying expression through LEDs

13. Accesories

14. Programming Environments (Remote Framework) • AIBO’s SDKs allow to write programs that either executes on AIBO or on a PC and controls AIBO by using a wireless LAN: • Open-R is a cross-platform development environment based on gcc (C++) • R-CODE is a script language • Remote Framework is a Windows PC application development environment based on Visual C++ • Motion Editor – creates motion patterns for AIBO

15. Why Remote Framework (RFW)? • can take advantage of powerful CPUs, large memory areas, and large persistent (hard drive) storage • perform streaming of the data captured by camera and microphones • make applications with rich user interfaces (AIBO Entertainment Player) • it can be easily integrated with other libraries (IPL, OpenCV) • get access to AIBO status (sensor information, battery level, etc.) • easy to debug and test some of AIBO functions • make use of 3000+ built-in motions

16. RFW Modules - Client Application - user application that utilizes AIBO RFW - AIBO RFW API - AIBO Remote Framework libraries - VirtualAIBO Server - the server program which connects AIBO and the AIBO Remote Framework application Remark: From a client application, multiple AIBOs (up to 16) can be simultaneously controlled. However, in this case, there is only one server program.

17. Remote Framework is outlined by the red dotted line

18. The AIBO Remote Framework has four dynamically linked libraries (DLLs): • - VAIBOClient.dll - Connect/Disconnect to AIBO. Enable/Disable FTP, Sensor, Image, and Audio services that are internal AIBO functions. Send commands to AIBO to change posture, walk, etc. The VAIBOClient.dll is called VirtualAIBO Client and the server is called VirtualAIBO Server. • - AIBO3D.DLL - 3D AIBO model • - CPCInfo.DLL - Get AIBO's hardware configuration (Robot design) • - VAIBOUPnP.DLL - Search for AIBO in the network using the UPnP feature • - VAIBOTTS.dll – TTS (Text to Speech) engine

20. Function categories in RFW: • - Connect/Disconnect the network to AIBO • Control services of AIBO – requested for FTP, Sensor, Image, and Audio transmission • Get semantics data – AIBO’s specific information • Get sensor data • Get/set camera parameters and retrieve image data • Send/receive audio data • Control motion - change posture, walk, head movement, kick • Control autonomous objects - face detection, motion detection, voice recognition, cliff detection, self recharging, visual pattern recognition • TTS (Text to Speech) – synthesize speech from text and send to PC’s loudspeakers.

21. How is RFW working? • Data transmission between VirtualAIBO Client and VirtualAIBO Server uses Windows Messaging. • VirtualAIBO Client (*.dll) creates Windows Messages and sends them to the application window • The following messages are generated: • WM_VAIBO_CONNECT / WM_VAIBO_DISCONNECT - WM_VAIBO_REMOTE_NOTIFY- WM_VAIBO_SEMANTICS / WM_VAIBO_INPUT_SEMANTICS- WM_VAIBO_SENSOR_DATA- WM_VAIBO_BATTERY_LVL - WM_IMAGETP_UPDATE / WM_AUDIOTP_UPDATE- WM_VAIBO_CLIENT_UPDATE

22. The Shared Memory system is used when data is sent and received between the VirtualAIBO Server and the user application. • Four shared memory areas are used: Image, Audio (Send), Audio (Receive), and server stored information (Windows handle, sensor data, posture data, AIBO's status, battery level, and semantics) • The shared memory areas are created upon the start-up of VirtualAIBO server • In total, 16 shared memory areas of each type are created (that’s why the maximum number of AIBO connections is limited to 16)

23. How is the image sent from AIBO to the user’s application? • VirtualAIBO Server stores image data to the shared memory • VirtualAIBO Server sends a Windows Message to VirtualAIBO Client • VirtualAIBO Client can get the image data from the shared memory • Virtual AIBO Client sends a Windows Message (WM_IMAGETP_UPDATE) to the user application. • - When user application receives the (WM_IMAGETP_UPDATE) from VirtualAIBO Client, it can retrieve the image data by using the Remote Framework API (VAIBO::ImageGetData). • Remark: The image data is sent in a JPEG format

25. Application: AIBOLive • The purpose of this application is to see how is possible to apply OpenCV functions on images acquired from AIBO’s camera • The received image is converted to an IplImage structure • OpenCV functions can be applied after that • We use Intel Jpeg library to decompress ‘on-the-fly’ the jpeg picture into memory • AIBO can send images at 3 levels of resolution: • high: 412x318 pixels (10 frames/s) • medium: 204x159 pixels (25 frames/s) • small: 102x79 pixels

26. CVAIBO *my_aibo; IplImage *aibo_image; IJLWrapper my_IJLWrapper; void CheckAIBOSemData (UINT msg, WPARAM wParam, LPARAM lParam ) { //Other messages are processed here if (msg==WM_IMAGETP_UPDATE) { //All needed variables are defined here my_aibo->ImageGetData((unsigned char*)jpg_buff,IMAGETP_RECVMAX); rgb_buff=(char*)my_IJLWrapper.DecompressImage(w, h, no_channels, im_size, (unsigned char*)jpg_buff, IMAGETP_RECVMAX); memcpy(aibo_image->imageData, rgb_buff, im_size); cvShowImage("AIBO EYES", aibo_image); } }

27. The result of applying face detector to the image sent by AIBO

28. AIBO Active tracking

29. RFW Structure • VAIBO.h • Contains main functions for AIBO programming (control, movement, services request, etc) • VAIBODef.h • Contains some macros definitions (windows messages, parameters for functions, etc,) • SemID.h • Contains macros definitions for semantic information (status of a command, detection/recognition of some objects/events) • CpcInfo.h • Contains definition of sensor data structure

30. Installation and configuration of MSVC++ • Remark: In what follows, it is assumed that you already have a wireless connection between your PC and AIBO • Recommended directory: C:\Archivos de programa\Sony\AIBO_RFW • - in the ‘c’ source file add the header files needed: ‘VAIBO.h’, ‘VAIBODef.h’. ‘Cpcinfo.h’, ‘SemID.h’ • from the ‘Project’ menu, choose ‘Settings’ and click on the ‘Link Tab’. In the field: ‘Object/library modules’ add the following: ‘VAIBOClient.lib’ • Remark: in order to be able to convert AIBO’s image to ‘IplImage’ structure, the ‘Intel JPEG’ library is needed (decompress jpeg file in memory). It is not included in the RFW installation toolkit. We assume that this library is installed in: • “C:\Archivos de programa\Intel\OpenCV\otherlibs\Intel JPEG”

31. Besides RFW’s specific header files, in the ‘c’ source file we have to add ‘IJLWrapper.h’ and ‘ijl15.lib’ to the library path, as shown below

32. From the menu ‘Tools’ choose ‘Options’ and then click on the ‘Directories’ tab. In the field ‘Show directories for’ choose ‘Include files’, then edit the following paths: • C:\ARCHIVOS DEPROGRAMA\Sony\AIBO_RFW\AIBORemoteFramework\INCLUDE • C:\ ARCHIVOS DEPROGRAMA\ \Intel\OpenCV\otherlibs\Intel JPEG • From the menu ‘Tools’ choose ‘Options’ and then click on the ‘Directories’ tab. In the • field ‘Show directories for’ choose ‘Library files’, then edit the following paths: • C:\Archivos de Programa\SONY\AIBO_RFW\AIBOREMOTEFRAMEWORK\LIB\DEBUG • C:\Archivos de programa\Intel\OpenCV\otherlibs\Intel JPEG

35. The last step consists in adding the paths for DLLs in the ‘Environment Variables’ • from ‘Control Panel’, choose ‘System’, then ‘Advanced’ and finally click on ‘Environment Variable’. From dialog box that appears, in the ‘System Variables’ section, click and edit the ‘PATH’ item. You have to add the following paths: • C:\Archivos de programa\Sony\AIBO Entertainment Player (cotains VAIBOServer.dll) • C:\Archivos de programa\Sony\AIBO_RFW\AIBORemoteFramework\LIB\Debug

37. A small kick for the robots, a giant score for science...

38. RoboCup final 2005 – Osaka Alemania - Australia

39. Appendix I – Sensor information

40. Appendix II Voice commands