1. Studying the Production Process

1. Analysis of Realtime ProcessesDifficulties for Modelling Electronic ModulesProduced with Graphical Open Signal Processing EnvironmentsAlain Bonardi - IrcamIntangible Heritage Workshop, april 19th, 2007

2. 1. Studying the Production Process

3. Studying the Production Processexample of the String Quartett by Florence Baschet • An « augmented » string quartett • Each instrument is enhanced with sensors (motion capture, pressure) • The goal is to detect « specific events » decided by composer Florence Baschet and to trigger the execution of predefined effects

4. Electronic Configuration of Baschet’s Quartett For each instrument : 2 sensors (motion capture and pressure) 7 different streams from sensors- three acceleration measures, - three rotation measures,- bow pressure. The problem: parameters don’t match musical concepts.

5. Towards Musical Gesture Recognition (IMTR team) • To create relationships between • these signals, • and the concepts musicians and composers generally handle, • so that they would not just map physical data to sound generation, but instead use a set of high level musical concepts • For instance, thanks to Hidden Markov Models, the patch is able to recognize some musical gestures • credits : Frederic Bevilacqua, Norbert Schnell

6. The production process 06 07 08 hardware setup capturing data training the HMM creation Current phase: capturing data in order to train the HMMs.

7. Events recognition • The events videos\gesture1_violin2.avi videos\gesture1_violin1.avi videos\gesture2_violon2.avi

8. Gesture modelling • videos\gestureModels.avi

9. Preservation issues • Performance preservation level • Using the processes to model performance • Identification of general and specific outputs from the processes • Sets of « test data » as sets of input data, coupled with output datas, intended to validate the correct setting of hardware/software (robustness / authenticity) • Documentation provided by the composer regarding the production process itself, containing an evaluation of the problems encountered • Identification of needed tools • Model of the relationship between the data streams and the different elements – sensors, instruments • Tools for analyzing data streams • Preservation of electronic tools • Software modules used in Baschet’s Quartett • Provide information for performance modelling • But need to be preserved…

10. 2. Preserving Electronic Tools

11. Graphical Environmentsfor Realtime Signal Processing… • Development of interactive modules without programmation thanks to a graphical language • Growing number of softwares and communities of users • Max/MSP/Jitter, PureData, Isadora, EyesWeb, etc. • From performing arts to robotics

12. … Provide Open Multimedia Environments • Easy interfacing with many devices(source: Jérôme Abel, http://impala.utopia.free.fr/) • Large possibilities of upgrading • Creation of new processings (Javascript) • Creation of new objects (SDK) • Creation of new libraries, implementing new paradigms • Classes in FTM library (developed by IMTR Team at Ircam)

13. Realtime Processes • Being able from inputs to compute required outputs within a non-perceivable duration • Examples (from IMTR Team, Ircam) • Score following • Input: live sound from an instrument • Output : position in the score • Gesture acknowledgment • Input: live data from captors (for instance getting acceleration and angle informations) • Output: most probable gesture within a series of gesture models

14. Authenticity in Preservation • Necessity of preserving processes and not only the output data • But these processes depend on hardware and software conditions • How can we guarantee reperformance respecting minimal authenticity ? • Technical incompatibilities: processors, operating systems, proprietary formats, software versions, etc. • Preservation requires adaptations • Example: Migration from one technical environment to another.Lichtung II by Emmanuel Nunes, from jMax to Max

15. Patch Abstract Description/Virtualization • Being able to describe in an abstract way a software electronic module (patch) • Requires extraction of artistic and technical knowledge from patches and documents

16. Towards AutomaticPatch Virtualization ? • Strong difficulties due to the lack of formalism of these open environments • The same processing, for instance a reverberation, may be implemented in different ways • Exchanges between modules or objects may be described in very different ways • Connections, send/receive objects, references (FTM), etc. • A global and comprehensive approach is impossible • Specific approaches • Design simple but helpful modules for musical assistants and realtime process developers ?

17. Tools for Patch Abstract Description • Patch parsing • Getting patch structure and synthetic information about its content • Patch comparison • Extracting differential information from two patches • Domain ontology • Building a thesaurus to set patch classes

18. Realtime Process Ontology • Towards an organology of realtime processes ? • Double structure • Mathematical description(ex. Faust system, by Grame) • Musical intention description

19. Conclusion • The introduction of electronic device and especially realtime softwares has provided • New possibilities of creation • Tools to model performance • But imposed the necessity to preserve them • Preserving electronic modules is quite difficult • Directions of work • Patch parsing • Patch comparison • Domain thesaurus