1 / 21

SEMANTIC HIFI Browsing, listening, interacting, sharing on future HIFI systems

SEMANTIC HIFI Browsing, listening, interacting, sharing on future HIFI systems. Music Technology Group Universitat Pompeu Fabra (UPF) Barcelona. WP5. Performance Workpackage. Interaction & Performance.

Mia_John
Download Presentation

SEMANTIC HIFI Browsing, listening, interacting, sharing on future HIFI systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SEMANTIC HIFIBrowsing, listening, interacting, sharing on future HIFI systems Music Technology Group Universitat Pompeu Fabra (UPF) Barcelona

  2. WP5. Performance Workpackage

  3. Interaction & Performance “…it becomes possible for more people to make more satisfying music, more enjoyably and easily, regardless of physical coordination or theoretical study, of keyboard skills or fluency with notation. This doesn’t imply a dilution of musical quality. On the contrary, it frees us to go further and raises the base-level at which music making begins.” (Laurie Spiegel) “Let’s develop virtual instruments that do not just play back music for people, but become increasingly adept at making new and engaging music with people, at all levels of technical proficiency.” (Robert Rowe)

  4. Interaction Has to be: • natural & intuitive • easy And yet… • allow expression • enjoyable • rewarding

  5. Input devices • Feel natural • Maximize bandwidth • Profit from users’ knowledge

  6. Input devices • Feel natural • Maximize bandwidth • Profit from users’ knowledge We propose the use of • Mouth: microphone + small video camera • Hands & arm: remote command used as a baton

  7. Input devices • Feel natural • Maximize bandwidth • Profit from users’ knowledge We propose the use of • Mouth: microphone + small video camera • Hands & arm: remote command used as a baton

  8. Mouth control information will be reinforced by the two simultaneous input modes (sound + image)

  9. Mouth • Mouth interaction will not only allow karaoke • The system will be able to detect at least 4 different mouth input modes: • Singing (karaoke) • Scat (instrumental solos) • Beat boxing (drums) • Silent mouth movements (filters & timbre changes) • Voice transformations include • Voice Excitation based Transformations (pitch change, hoarseness, whisper…) • Vocal Tract based Transformations (timbre…)

  10. Music Context • The results of each of these interaction modes, will depend on the music being played • Use of metadata will provide increasing information

  11. Music Context • The results of each of these interaction modes, will depend on the music being played • Use of metadata will provide increasing information • Example: Scatting on different musical styles

  12. Music Context • This would correspond to a simplified context • More information can be obtained: • From the type of voiced sound (voice analysis - not mere pitch-2-MIDI – should profit all timbre information) • From additional metadata

  13. Music Context • This would correspond to a simplified context • More information can be obtained: • From the type of voiced sound (voice analysis - not mere pitch-2-MIDI – should profit all timbre information) • From additional metadata

  14. Additional Metadata* Time-stamped information: Music • Composition parts (A, B, chorus…) • Harmonic & rhythmic details • Score • Program changes • …. Audio Analysis • …. *Format and contents to be defined in WP1.2

  15. Editable Metadata • Advanced users will be able to edit and enrich the Metadata (in non real time), adding value to their contribution

  16. Hands Movements Will provide complementary information • e.g. crash cymbal on beat boxing Alternate functions • e.g. baton conduction • tempo changes • dynamic changes • groove & swing modification • …… • ……

  17. Hand & Body tracking • A camera fixed to the system could be used • For better tracking resolution (spatial & temporal) an additional device seems necessary • We propose to use the same command, fitted possibly with accelerometers (and wireless communication with the system)

  18. Score Following IRCAM: Instrument Score follower (for automatic performer accompaniment) To be defined: • Options • MIDI (or synthetic) accompaniment • Time-stretched prerecorded audio • Data formats • data resulting from the audio analysis (UPF), sent to the score follower module (IRCAM) (voice2MIDI?) • position data from the score follower to the time-stretching module

  19. Performing on a simple keyboard In this part Sony CSL will implement style and performing rules in a simple keyboard able to follow and continue the user play according to simple style constraints.

  20. Deliverables

  21. MTG Participants • Xavier Serra, local manager • Sergi Jordà, technical manager • Alex Loscos, voice processing • Martin Kaltenbrunner, interfaces • 1 additional programmer

More Related