Experimental Robot Musicians

1. Experimental Robot Musicians Tarek Sobh, Kurt Coble and Bei Wang School of Engineering University of Bridgeport

2. Table of Content • Introduction • Motivation: music expressiveness • Robot Musician Architecture • Band overview • Architecture overview • Detailed module description • A Robot Musician Example: Jasche • Mechanical changes in real-time performance • Results • Robot Musician Instrument • Future Development • Future Discussions • Conclusion

3. Introduction • Robot musicians perform on real instruments through the usage of mechanical devices, such as servomotors and solenoids • Research innovations linking music, robotics and computer science

4. MotivationMusic Expressiveness • Offer the audience live-experience very similar to listening to a human musician. • Real instrument performance, such as the physical vibration of a violin string, provides a much stronger case in music expressiveness, versus electronic music synthesizers. • Mozart - eine kleine nacht musik: whole ensemble

5. MotivationMusic Expressiveness (cont.) • Bypass several technical difficulties that are typically encountered by human musicians • More degrees of freedom in real-time performances and reach a higher level of performance difficulty, flexibility and quality. • As an example, a violin is played by a robot musician with hands that have 12 fingers.

6. Robot Musicians ArchitectureRobot Musicians Band Overview • Each member of the band is a robot musician, which specializes in either a string or a percussion instrument

7. Robot Musicians ArchitectureRobot Musicians Band Overview (cont.) Robot musicians, “the P.A.M. band”, invented by Prof. Kurt Coble. The moth: features violin solo, composed by Prof. Kurt Coble, companied by percussion ensemble, electric base and electric guitar

8. Robot Musicians ArchitectureRobot Musicians Band Overview (cont.) Austin plays a Percussion Ensemble Dusty plays a red electric guitar

9. Robot Musicians ArchitectureRobot Musician Architecture Overview • A three-module architecture

10. Robot Musicians ArchitectureRobot Musician Architecture Overview(cont.) • Software module: interacts with users, provides the programming and composition environment and sends motion commands to the control module through a serial port. • Control module involves rules that govern application processing and connects the software module with the motion module. • Motion module - the hands of the robot musician; powered mechanics such as servos and solenoids, which manage and provide access to the actual musical instrument.

11. Robot Musicians ArchitectureSoftware Module Infrastructure • Implemented as a set of Turbo C++ programs • The software communicates with the control module through serial port I/O. • Consists of two major components: • composition component • performance component.

12. Robot Musicians ArchitectureSoftware Module: Composition Component • Direct programming environment: based on a set of pre-defined syntax, users can create C++ programs which control each detailed motion of the robot musician • Compose music scores: through the keyboard and mouse

13. Robot Musicians ArchitectureSoftware Module: Composition Component (Cont.) • Recording function: detects and records details of each key-press or mouse-movement event, and the time delay in between.

14. Robot Musicians ArchitectureSoftware Module: Performance Component • Read and play the musical scores recorded by the composition component or directly composed by a user • Single-line interpretation: produce a musical experience that is almost identical to the original instrumental performance • Tuning component: adjust sound quality of the instruments before composition and performance

15. Robot Musicians ArchitectureControl Module • An RSV Converter and a costume-manufactured motion control card • The motion control card is built on an Aesthetika Robotika Servo Control Rev 4 board, powered by 12V and 500 mA. It is custom manufactured for this project to support up to 4 servos (4 axes) and 12 solenoids • The converter interfaces between the serial port and motion control card

16. Robot Musicians ArchitectureControl Module (Cont.) • Receives motion control commands from the software module and computes joint angles and servo motor velocities for each axis • Deliver these values to the motion module, ensuring smooth servo motor movements.

17. Robot Musicians ArchitectureControl Module (Cont.) Control module with motion control card and converter, linked with the motion module of an incomplete keyboard robot musician hand

18. Robot Musicians ArchitectureMotion Module • The Motion module is provided by servos and solenoids • Two types of servos (Hitec Quarter Scale HS-700BB Servo and Hitec HS303 Servo) and three types of solenoids are used in its construction

19. Robot Musicians ArchitectureMotion Module (Cont.) • A violin is set up with two bows controlled by two servos and twelve fingers (solenoids). Servos are attached to bows of the violin. They move the bows back and forth across the violin strings. Solenoids, act as magnets to depress the strings when charged • Amazing grace: traditional American folk song.

20. Robot Musicians ArchitectureMotion Module (Cont.) Servo attached to one bow of Jasche Solenoid (with holding power of 1.5 pounds) attached to Jasche

21. Servomotor In Action

22. Robot Musicians ArchitectureMotion Module (Cont.) A coffee container’s plastic lid is connected with a servo so it flutters against the body of a drum when the servo receives control command from the control module Sample Motion Module Architecture: drumstick controlled by solenoid

23. Robot Musicians ArchitectureMotion Module (Cont.) Sample Motion Module Architecture: chimes wand controlled by servo

24. Motion Module In Action • Mozart - eine kleine nacht musik: whole ensemble

25. Robot Musicians ArchitectureA Robot Musician Example: Jasche • The Robot Musician Jasche plays a 2-bow 12-finger 2-string violin tuned with 3 octave chromatic range (From F below middle C to two octaves above middle C), which produces 24 pitches. • Happy Birthday

26. Robot Musicians ArchitectureA Robot Musician Example: Jasche (Cont.)

27. Jasche In Action Amazing grace: traditional American folk song

28. Robot Musicians ArchitectureA Robot Musician Example: Jasche (Cont.) • Software module: compose musical pieces through mouse movements • Keys on the third line of the keyboard, symbols “QWERTYUIOP” correspond to 10 speed levels of the bow movement. • Keys on the second line of the keyboard, symbols “1234567890-=” correspond to 12 solenoids used to press the strings, producing different pitches

29. Robot Musicians ArchitectureThe Robot Musician Band Architecture

30. Robot Musicians ArchitectureThe Robot Musician Band Architecture (Cont.) • 8 MCC boards are chained together, meaning, 8 robot musicians can be controlled by one single computer terminal simultaneously

31. The Mechanical Changes in Real Time Performance • The smooth transition between two musical notes (continuation of music score): the improvement of mobility, elasticity and sensitivity of the robot hands • High speed communication between the PC and the motion control card becomes critical

32. Real Time PerformanceMechanical Issues

33. Results • The recent work on robot musicians has some promising results. The existing system is found to be a robust method in controlling complex musical mechanics • A musical piece collection will follow • 7sg2: composed by Prof. Kurt Coble, the name of the musical piece stands for "7 solenoids guitar part 2", 6-string electric guitar, tuned in a parallel major 7th chord

34. Results: drum set in action

35. Results: whole ensemble • Botman: composed by Prof. Kurt Coble, featuring whole ensemble

36. Robot Musical InstrumentA New Generation and A New Art Form • Musical instruments that are more complex in nature for robot musicians • A combinational drum set: drums are specially positioned so that they are easily accessed by the robot hands. • A violin can be played with two bows simultaneously

37. Robot Musical InstrumentA New Generation and A New Art Form (Cont.) • Digitally-Manipulated Analogue Signal Systems (D-MASS) is proposed in this work as a description of this new art form. The following Table makes a comparison between MIDI (Multi Instrument Digital Interface) and D-MASS.

38. Future Development • Software/hardware tools: read and play traditional scores. • soft computing (fuzzy logic, neural networks, evolutionary and genetic algorithms) and artificial intelligence / expert systems techniques • Improving the emotional quality of the music performance • Having the robot musicians listen to various music pieces, recognize the tones, improve on them and then re-play them • Wireless control

39. Future Discussions • Appreciating the interaction of music, computer science, robotics and the new concepts that emerge from robot musicians. • The analysis of the degree of intelligence and sensitivity achieved by robot musicians

40. Conclusions • The experience of designing robot musicians is a breath-taking experiment. • A fully functional robot orchestra can be established in the near future, which will play not only simple tunes, but also masterpieces.

41. Acknowledgements • James Sedgwick • Dr. Jeffrey Johnson • Dung Hoang and Ying Guo • Interdisciplinary Robotics, Intelligent Sensing, and Control (RISC) laboratory • Arnold Bernhardt Center, Austin Mather Bubble Theatre

42. Current Project Status • 2002 Sigma Xi Grant-in-Aid of Research recipient • Paper titled “Experimental Robot Musicians” has been accepted by the Journal of Intelligent & Robotic Systems

43. Thank You