Approaches to Overcoming Problems in Interactive Musical Performance Systems

1. Approaches to Overcoming Problems in Interactive Musical Performance Systems Cory McKay Music Technology Area Faculty of Music McGill University Cory McKay

2. Topics • Introduction to interactive performance systems • Computer that performs with human performer(s) • Problems that have limited the use of such systems • General approaches that address these problems • Proposed long-term solution: • Standardized control parameter extractor • Reliance only on parameters extracted from performers’ audio signal Cory McKay

3. Interactive performance systems • Software / hardware that reacts musically to input from performer(s) • Can perform with: • A human soloist • A human accompanist • An ensemble of human performers • Other performance systems Cory McKay

4. Possible roles • Use score-following to dynamically timewarp pre-set scores • Initiate pre-recorded events triggered by performer actions • Basic mappings involving pre-set heuristics • e.g. harmonizations, dynamic sound processing, etc. • Use of advanced expert systems or non-deterministic artificial intelligence to model sophisticated musical knowledge • e.g. “jamming systems” Cory McKay

5. Advantages • Can perform actions that are too technically difficult for humans • Take advantage of extended sonic capabilities of computers • e.g. precise control of sound synthesis • Capitalize on essential expressivity and live dynamism of human performer(s) • Useful in an educational context Cory McKay

6. Problems with interactive systems • Lack of reliability during performance • Pieces can be too difficult or unintuitive for performers • Software can be unstable or too strenuous on hardware • Limited longevity • Reliance on software and hardware with limited life span • Poor interface with little or no documentation • Few performances without composer’s direct involvement • Hardware expensive and difficult to acquire and transport • Systems difficult to set up • Proper documentation and instructions rarely provided • Lack of effort to make pieces attractive to performers and audiences Cory McKay

7. Solutions relating to performers • Performers should not be expected to perform tasks that are alien to their training • Limitations of performer bandwidth must be considered • A clear explanation must be made of specifically how the system will respond to their actions • Intuitive feedback must be provided • Auditory, visual, haptic • Performers must be given sufficient time to learn how to use the system • Includes significant rehearsal time with functional system Cory McKay

8. Solutions relating to system robustness • Software should be well tested • Creative ambition should be tempered by technical realism • Simple equipment should be used whenever possible • The smaller the amount of electronics the better • Systems must be error tolerant • Not fooled by minor mistakes and should not fail entirely if major mistakes occur • Software interface should be simple • Basic overrides should be available Cory McKay

9. Solutions relating to dissemination and longevity • Only commonly available, cheap and easy-to-transport equipment should be used • Well-known software that is likely to have a long life-span should be used whenever possible • Applies to operating system and implementation language • Any need for proprietary or unique hardware or software should be avoided unless absolutely necessary • Systems should require only minimal supervision during performance • What supervision is needed should involve a simple, intuitive and well-documented interface Cory McKay

10. Advantages of standardization • Standardized performance parameter extractor would be beneficial: • Composers would not need to implement their own systems • Professional design would cause system to be more robust • Concert venues could be pre-equipped with system • Technicians at venues would be accustomed to the system • Addresses problems with reliability and dissemination of compositions as well as robustness • Mapping system should not be standardized • Would compromise creative freedom • Could use a client-sever architecture to facilitate porting of parameter extraction system to mapping system Cory McKay

11. Advantages of exclusive use of parameters derived from audio • Performers do not need to spend time learning modified instruments or new performance gestures • Increased familiarity makes performers less likely to make mistakes • Miniaturization and wireless technology can make microphones essentially invisible to performers • Rich sound of original instruments is available as well as information previously provided only by MIDI instruments • Only hardware needed is microphones and computer • Unlikely to become obsolete anytime soon • Reliable technology • Cheap, portable and easy to set up • Could mic audio output of hyper-instruments Cory McKay

12. Parameters to extract I • Want a general-purpose system • A large number of parameters provide a diverse palette • Too many parameters can be problematic • Compromises simplicity • Increases latency • Implement only the most clearly useful parameters • Should be sufficient for most pieces • Additional parameters can be added as plug-ins Cory McKay

13. Parameters to extract II • Primary output is MIDI transcription: • Note Ons and Note Offs • Time, pitch, velocity • Stream of onsets detected • Frequencies take longer to detect than onsets • Note Ons therefore have a greater latency than onset detections • Stream of detected frequency values • Needed for microtonal pieces and recognition of techniques such as glissandi • Overall signal amplitude • Gives an indication of overall dynamics Cory McKay

14. Parameters to extract III • Additional features related to timbre • This potentially very rich source of information is unavailable from MIDI instruments • e.g. relative amounts of energy in different spectral regions • More sophisticated features can be extracted from these basic parameters • Chord classification, beat tracking, note density, etc. Cory McKay

15. Inaudible cues • Performers often communicate with each other inaudibly • e.g. visual gestures • Could also be useful to communicate with an interactive system inaudibly • e.g. manual state switching if a cue is missed • Should avoid sophisticated, expensive or unusual equipment or anything requiring specialized training • Cameras and a standardized image processing system could provide a good long-term solution • Basic footswitches Cory McKay

16. Difficulties with real-time transcription • Monophonic transcription is known to work well • Can still have problems with latency • Accurate polyphonic transcription still an unsolved problem • Latency an even greater problem here • Can use localized microphones in ensembles to help with polyphonic transcription • Turns hard polyphonic transcription problem into multiple easier monophonic transcription problems • Does not solve problem of polyphonic instruments • Piano, guitar, harp, etc. • Advances in transcription technology will make real-time transcription more feasible as time goes by Cory McKay

17. Conclusions • There are currently significant problems relating to longevity, dissemination and reliability • A variety of fundamental guidelines have been presented to address these issues • Promising long-term solution to these problems: • A standardized real-time control parameter acquisition system relying only on auditory input Cory McKay