1 / 32

Musical Virtual Reality Applications

Musical Virtual Reality Applications. Michael Kriegel. Contents. My 2005 final MSc project at the Centre for Virtual Environments (University of Salford) A current Heriot Watt student project: Wii Piano by Thomas Methven. Motivation.

stian
Download Presentation

Musical Virtual Reality Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Musical Virtual Reality Applications Michael Kriegel

  2. Contents • My 2005 final MSc project at the Centre for Virtual Environments (University of Salford) • A current Heriot Watt student project: Wii Piano by Thomas Methven

  3. Motivation • Personal interest in electronic music technology (sampling, synthesizers, MIDI) • Use VR as a new interface to electronic musical instruments • Original idea: create a Virtual Reality drum kit interface with optical tracking input • Why do this? • Might provide interesting musical performance • good method of testing/measuring optical Vicon tracking system • Virtual interface doesn’t take up space (many people don’t have space for a real drum kit)

  4. Research Questions • Can the feeling of playing the drums be replicated with a VR interface? • Is optical tracking suited (fast and accurate enough) for this task ?

  5. Literature Review 1 • First initial literature review strived to find similar comparable projects • Not much on the academic sector: • University of Twente in Netherlands had a similar project, but used different technology – also no papers on that project • Axel Mulder, Phd thesis, Simon Fraser University, Canada – data glove / hand gestures to create sound • Interdisciplinary Centre for Scientific Research in Music (ICSRiM) at University of Leeds have a research framework “Music Via Motion” that tries similar things, however again with different technology

  6. Literature Review 1 • Related commercial and art projects: • Electronic Drum Kits • Theremin, Roland‘s D-Beam controller, Alesis Air FX • Eye Toy • Soundbeam • Myron W. Krueger‘s installations

  7. Technology - MIDI • Musical Instrument Digital Interface • A protocol that allows electronic musical instruments to communicate with each other (or with a computer) • A MIDI instrument (or computer) usually has a MIDI-in port to receive messages and a MIDI-out port to send messages • MIDI protocol defines messages: e.g. Note-On, Note-Off

  8. Technology - Creating drum sounds electronically • Can be easily done via Samplers, Drum Machines, Synthesizers, etc. (both hard- and software) • For my project: Using a free sampling software and a pre-defined drum kit (collection of drum sound samples) that can communicate with the application via MIDI

  9. Technology – optical tracking • Vicon tracking / motion capture system consists of • 5 infrared cameras • Dedicated computer for pattern recognition • Pattern recognition software

  10. Technology – optical tracking • Objects that need to be tracked are equipped with reflective markers • Objects need distinctive marker patterns • By observing the marker pattern from the 5 different perspectives, the objects position and orientation can be calculated • Software has training mode to learn to recognize new objects

  11. Tracked objects • Shutter glasses • Drum sticks • Feet

  12. Technology – Display System • Barco Trace Large Screen Display • Back Projection Screen with 120 Hz refresh rate for 1280 x 1024 resolution • Active Stereo with stereo shutter glasses

  13. System Design

  14. Implementation Steps • Create 3D Models of Drum Kit with 3DS Max • Import them scene graph • Create interface with tracking system (glasses, hand, feet) • Create MIDI connection between application and Software Sampler • Implement collision detection between virtual drum stick positions and drum skins (collision triggers Midi out note, velocity of impact determines the volume of the sound)

  15. Main Program Loop • Get current position of sticks • Calculate if a collision has occurred (also compare with previous position to avoid continuous sounds) • If collision has occurred • Calculate speed (again by comparing with the previous positions and their timestamps) • Send a midi message (Note On) with Note number relating to the object that was hit and volume relating to the speed • Update Scene (set new position of sticks) • Render Scene • Could be improved by seperating the collision detection/sound output and grpahics in different threads

  16. Screenshot

  17. Problems discovered • Missing haptic feedback seemed to be a major drawback • Some more research into haptic devices that could be used in this context, Spidar seems to be the only well suited device for drum sticks, but not available at our department

  18. Change of Research Direction • Since the haptic path was a dead end due to availability of hardware … • and I was ahead of schedule with the implementation.. • The implementation of two other virtual instruments was decided: • Virtual Piano – to compare both tracking performance and haptics with drums • Virtual Synth Control (Theremin like experimental instrument) – to compare replications of traditional instruments with experimental instrument

  19. Virtual Piano • Changes: • New controller: gloves • Different MIDI sound module:free software piano • Piano model generated by the application and not by modelling package • Different collision detection algorithm (need note-on and note-off events)

  20. Screenshot

  21. Virtual Synth Control (VSC) • Design objectives: • Easy to use • No haptics required • Psychedelic look and sound • Input mode: drum sticks • Sound module used: Free Software synthesizer JX 220

  22. Screenshot

  23. Evaluation • Main Purpose was to compare the user’s reception of the 3 instruments. • 12 subjects • Multiple choice questions: • Which instrument did you like most? • Which of the instruments, if any, could be best used for making real music? • Majority chose VSC for both questions (10/12 and 11/12) • No-one chose “none” so everyone could see some potential in the technology

  24. Evaluation • Another set of questions was aimed at evaluating all instruments individually on a scale of 1 to 7 • Some findings: • No depth perception problems with any of the instruments (question 1) • Lack of haptics is severe, even more so for the piano (question 2) • People found it hard to drum to a rhythm / easy to play VSC along with backing track (question 3)

  25. 3 ½ years later

  26. Wii Piano • Basic idea: use the WiiMote’s infrared camera for very cheap finger tracking • 2D • Compare with other input methods (e.g. mouse, touch screen) • Applications: Drawing, Piano

  27. WiiMote • Besides the usual buttons the WiiMote has 2 novel forms of input: • Accelerometers • Infrared Camera

  28. WiiMote Finger tracking • Normally the infrared camera is used to evaluate the WiiMote’s position relative to the Wii’s sensor bar • For WiiMote Finger tracking, the WiiMote’s camera is used other way round: • WiiMote doesn’t move • Infrared Light source moves

  29. WiiMote Finger tracking • No active light source on finger is used but reflection (see passive markers in mocap)

  30. WiiMote Finger tracking

  31. User Interface: Button Activation

More Related