Embodied Speech and Facial Expression Avatar Design Proposal

Embodied Speech and Facial Expression Avatar Design Proposal sss Dan Harbin - Evan Zoss - Jaclyn Tech - Brent Sicking sss February 9, 2004

Problem Background • Yano is capable of conveying many emotions through his facial expressions • We propose to reverse engineer this toy and create a puppet capable of showing happiness, sadness, anger, etc. • By adding an additional microphone input we can measure the intensity of the speech and adjust the facial expressions accordingly

Goals and Objectives • Gain knowledge of robotics systems, reverse engineering, and sensor/actuator interfacing • Experience with human computer interaction and multimodal interfaces • Develop system integration skills with low- level hardware control and graphical user interfaces • Practice and improve our design methodology, management process, and teamwork skills in order to insure a quality product

Technical Survey The Furby • In 1999, Peter van der Linden issued a challenge with a $250 reward for anyone who could hack Furby and make it reprogrammable • The first hacked Furby went to a mother of an autistic child who had written van der Linden early in the competition • Playing with the Furby was actually improving her son’s speech, however she was concerned about the gibberish the Furby speaks • These efforts at hacking Furby brought about the “Hack Furby Kit”

Technical Survey, Continued Yano, Robotic F.A.C.E. • The Media Lab at MIT created an output device for the computer, similar to a monitor or speakers, that outputs facial emotions • They took out all the internal electronics of Yano, and removed his body, so they were left with a head that had 3 motors and some gearing • The motors are controlled by three MIT developed microcontrollers, called iRX Boards • Used their hacked Yano with some MIT software called Listenin, a remote monitoring program that detects speech and other sounds remotely and sends them back to the local computer where Yano makes speaking movements accordingly

Technical Survey, Continued Yano, Robotic F.A.C.E.

Selecting a Toy: Furby vs. Yano • Yano • Movement in the eyelids, eyebrows, ears, mouth, cheeks, head, and hands • The mouth is opened and closed with one motor, the cheeks are moved up and down with a second motor, and the ears, eyes, and eyebrows all run off of the third motor • Speaker and infrared receiver Furby • Movement in the eyes, eyelids, mouth, ears, and feet – all controlled by one motor • Sensors for interaction in the head, stomach and tongue • Speaker, microphone, and infrared i/o • Internal electronics are packed in tight and sealed in a tough plastic case with resin

Selecting a Toy: Yano • Yano gives us more control over the face muscles (Furby only has a beak that can open and close) and has 3 motors that we will be able to independently configure

Selecting an Interface • USB - dependent on a master-slave type relationship that requires control from the slave side • Parallel Port - offers a simple way to control the motors and read the sensors all in parallel; however, it would require us to do the timing of the motor controls in software which would be unreliable • Serial Port - requires a microcontroller to go from the serial communication with the computer to a parallel i/o control for Yano; ability to time all communications to a certain baud rate

Selecting a Microcontroller OOPic • ability to program the PIC to do the serial to parallel conversion • object-oriented programming • we will need to decode serial protocols for serial to parallel conversion • unsure if the OOPic will be able to use the serial port as input in the running program

Selecting a Microcontroller, Continued SV203 “microcontroller based servo motor control” • It has a serial port connection, the SV203 processor, 8 servo control ports, and 5 input pins • You give it commands in ASCII text through the serial port, and it controls the servo ports, or reads the input pins based on the commands • Pulse Width Modulation • It may be more difficult to program than the OOPic, but the manufacturer has APIs for C and Visual Basic to make it much easier

Selecting a Microcontroller, Continued • Overall, the OOPic is more geared toward programming then letting it stand alone and control a device, and the SV203 is more geared toward direct control of the device through the serial port. • We chose to try the SV203 for our design.

Proposed Design: Hardware • Disassemble and hack Yano • locate all the motors, and test them using 6 volts of DC • Control motors with SV203 • use a transistor to enable or disable each motor, and to use an inverter to reverse direction • the transistors and the inverters will be connected to the logic and ground pins of the SV203 • digital input to the control circuit will be generated by sending commands to the SV203 through its serial interface • connect the end switches to the input pins of the SV203 • read the pins values using HyperTerminal

Proposed Design: Low-Level Software • Create rudimentary serial interface for SV203 • write function to generate the command to move a motor to a certain position in a clockwise or counterclockwise direction • read function to read the end signal of the specified motor • Calibration and per-motor control • all motors will be driven in one direction until they reach the end and a switch is triggered • then all motors will be driven in the opposite direction until they trigger the end switch, this time the number of pulses will be counted

Proposed Design: High-Level Software • Psychology of facial expressions and emotions • combine the move functions of various motors • create functions for many different moods that will move the motors to certain positions to express emotions • Graphical User Interface • design with the RAD tools included in Microsoft Visual Studio using C++ • matrix filled with pictures of different moods; when clicked, Yano will mimic the picture • sliders for each motor allowing for tweaking of the position of each motor to create custom expressions • voice interface to translate the intensity of the user's voice into mouth motor movements • calibration button

Approach for Design Validation • Use a power supply and a voltmeter to check the motors and end triggers of the disassembled Yano • Use HyperTerminal to drive the motors in both directions with the SV203, and to read the end conditions from the SV203 reliably • Redo the last test with the C++ interface • Execute the calibration test mentioned in the previous calibration section. Also, move the motors to many different positions, and see if the results can be reproduced reliably • Check that moving between emotions is reliable, and that the expressions are correct • Have a user test the GUI for intuitiveness, and see if the voice interface properly drives the Yano's mouth

Economic Analysis and Budget

Team Management • Brent – GUI layer manager; knows Visual Studio, C++, and .NET • Jaclyn – Software and API manager; psychology expert; develop algorithms for controlling the motors, and generating facial expressions • Evan – Motor control circuits; enable and disable the motors, and change their directions • Dan – Serial port interface and communication; provide a hardware/software interface for controlling the hardware

Schedule of Tasks

Societal, Safety, Environmental Analysis • Practical benefits to society, especially in the area of child therapy • Example: help autistic children communicate better and open up to adults • To ensure a safe project, all of circuits must be safely enclosed in the chest cavity of the Yano • Yano is an environmentally- friendly toy with the exception of the batteries it uses • Alternative 1: DC wall adapter • Alternative 2: rechargeable batteries

Thank You • Information on MIT’s Project http://web.media.mit.edu/~stefanm/yano/ • SV203 Motor Controller http://www.pontech.com/products/sv200/index.htm

Embodied Speech and Facial Expression Avatar Design Proposal