1 / 18

Musical Expressivity Description

Musical Expressivity Description. Research Proposal. Esteban Maestre. Music Technology Group Universitat Pompeu Fabra. OCTOBER 2004. Introduction. Musicians do not play exactly what is written in the score. Performers enrich the sound using expressive resources:. Time deviation.

chiquita-clayton
Download Presentation

Musical Expressivity Description

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Musical Expressivity Description Research Proposal Esteban Maestre Music Technology Group Universitat Pompeu Fabra OCTOBER 2004

  2. Introduction • Musicians do not play exactly what is written in the score • Performers enrich the sound using expressive resources: • Time deviation • Dynamics and articulation • Modulation • Ornamentations … • Many parameters related to the audio signal are relevant when analysing the musical expressivity of a particular performance • In this way, the extraction of key high-level descriptors allows the characterization of the expressivity • GOAL: • define a structured description scheme BEYOND NOTE-LEVEL • study possible methods for extracting expressivity descriptors • obtain a compact representation of the extracted expressive parameters

  3. State of the art • Some features studied so far: • TIMING • DYNAMICS • ARTICULATION • Much work on expressivity description from MIDI-like representations of real performances: • Widmer et al., rules for piano performance expressivity • Dillon, Camurri et al., mood recognition • Bressin et al., rules + ANN for automatic performance • Desain & Honing, score-performance matching • Gómez et al., expressive tempo transformations in saxophone jazz performances • Ramirez et al., rules for expressivity prediction in saxophone jazz performances • Some work on articulation description of isolated sounds: • Jenssen, K., timbre modelling …

  4. Context • Learn expressive PATTERNS from real performance monophonic recordings, more concretely saxophone jazz standards Description STEPS: • Global-Note Level Description (MIDI-like) • Intra-note/Inter-note Description (Expressive) • Expressivity ‘understanding’ performance recordings MIDI-like description ExpressivityDescripition Understanding Expressivity KNOWLEDGE SCORE AI Techniques

  5. Possible Applications • Characterization / Understanding / Classification of different performance MOODS or EMOTIONS • Audio expressive transformations: Add expressivity to a ‘non-expressive’ audio recording • Object audio processing respecting expressivity: For instance, tempo change • Improve synthesis: Translate expressivity into synthesis parameters • Expressive performance skills evaluation • Performer impersonating • Study of instrument expressive interaction in concrete performance contexts • …

  6. AUDIO Description Scheme MIDI-like description • Description at note-level has been used to implement expressive analysis and transformation of monophonic recordings (Gómez et al. 2003) analysis FRAME FRAME description windowing ONSET detection PITCH estimation MIDI-Note MIDI-like MELODIC description Onset, duration Intensity Global Description (mean Energy) • This depth of analysis lets limited expressivity modelling (timing deviations, note insertion/deletion, etc.), but there is much expressivity to be observed beyond note-level…

  7. AUDIO Description Scheme Description scheme extension • Description through different structural / temporal levels • Two main analysis parameters: FUNDAMENTAL FREQUENCY & ENERGY FRAME description analysis FRAME Note SEGMENTATION NOTE description NOTE MIDI-like description Intra-Note SEGMENTATION EXPRESSIVITY description Intra-Note SEGMENT Note TRANSITION • Modulation: tremolo, vibrato… • Dynamics: crescendo, decrescendo… • Articulation: legato, staccato… Intra-Note SEGMENT description TRANSITION description • …

  8. Intra-NOTE Segmentation Energy envelope approach • Based on studying ENERGY ENVELOPE of the notes played in a performance • CURVATURE of the smoothed envelope is analysed • Characteristic POINTS are extracted by finding 2nd DERIVATIVE EXTREMA FRAME description NOTE Segmentation EXX Instantaneous energy Note boundaries Envelope extraction t Energy ENVELOPE onset offset EXX’’ SMOOTHING t Smoothed ENVELOPE Find CHARACTERISTIC POINTS

  9. Intra-NOTE Segmentation Smoothing the envelope… • Smoothing means fewer 2nd derivative extrema but… loss of localization !!

  10. Intra-NOTE Segmentation Envelope model for performing segmentation • Notes are studied in their musical context (not isolated) • Observing general characterisitcs of noteenergy envelopes to be analysed (saxophone by now) lead to propose different approaches • Note energy envelope simplified model consisting of three linear segments: ATTACK, SUSTAIN, RELASE taken from the ADSR definition EXX t release attack sustain onset offset NOTE Segment

  11. Intra-NOTE Segmentation Algorithm description (1) • Characteristicpoints (2nd derivative extrema) help in finding segment limits • There are typically toomanyextrema of energy envelope 2nd derivative • Energy envelope has to be smoothed • Smoothing filter cutofff0is found in several steps by comparingoriginalandsmoothedenvelopes (error threshold) • The most predominant 2nd derivative extrema are selected and their projections on the envelope are joined by a line ENDof ATTACK • Right point of max slope  BEGINof RELEASE • Left point of min slope 

  12. em>eth? Intra-NOTE Segmentation Algorithm description (2) Extract Energy ENV F0initshould be sufficiently high Set F0init, eth , n Empirically… eth: less than 10% n: from 4 to 10 Smooth at F0 Increase F0 Compute em Error between original and smoothed envelope: YES NO Obtain ENV” Obtain 2nd derivative of the last smoothed envelope Find ENV’’ extrema Select n max ENV’’ extrema Set of n characterisitic points Take each pair of consecutive charcteristic points and calculate the slope of the line joining them Compute slopes Find max & min slope - ATTACK: from ONSET to ATTACKEND - RELEASE: from RELEASESTART to OFFSET - SUSTAIN: from ATTACKEND to RELEASESTART - ATTACKEND =2nd pt. of max slope - RELEASESTART=1ST pt. of min slope

  13. Intra-NOTE Segmentation Algorithm evaluation • Real notes from saxophone jazz standard performance recordings have been both manually and automatically segmented: • Number of segments detected automatically is not always the same as when it was manually marked !! • Mean duration error relative to note duration is computed for both the attack and release segments, revealing good algorithm performance • Instrument dependent? • Previous observation of energy envelopes will influence on some algorithm principles • Resolution selection • Efficient? • Can be performed locally? • Improvements… • Other possible approaches? • Wavelet analysis • Statistical approach • Multi-band energy envelope analysis …

  14. Intra-NOTE Segment Description • Once the segment limits have been obtained, some intra-note segment descriptors (linear model so far…) can be extracted from real energy shape • Duration • Level: initial and final energy envelope values • Slope: both mean and regression slopes • Integral: energy below the curve • LogAttackTime: following MPEG-7 definiton, only for attack segment • Intra-note segment similarities can be extracted for labelling intentions and more compact represetations by means of richer non-linear modelling Intra-NOTE Segment Description Intra-NOTE Segment Description Intra-NOTE Segment Description Intra-NOTE Segment Patterns CLUSTERING Non-linear PARAMETRIC modelling Intra-NOTE Segment Description Intra-NOTE Segment Description Complete NOTE description NOTE Patterns CLUSTERING • Fundamental frequency contour description !!

  15. TRANSITION Description • Using intra-note segmentation data, energy contour along note transitions can be analysed • Note transitions are considered including both release and attack segments of consecutive notes • Silences and release and attack segment shapes take part and characterize transitions… TRANSITION TRANSITION EXX • Fundamental frequency contour analysis during transitions !!  Further work… t release release silenci attack attack NOTE NOTE NOTE

  16. Fundamental Frequency Approach • The analysis of fundamental frequency plays an important role in expressivity description • Extend the curvature approach… • Not taken into account (so far) within intra-note segmentation: • Useful contribution… • Combine both energy envelope and fundamental frequency for a richer segmentation… • Vibrato descriptor in sustain segment • Glissando and other high level descriptors can be extracted analysing the contour along the transitions • Clustering & Parametric modelling…  Further work… TRANSITION EXX F0 t NOTE NOTE

  17. Further Work • Intra-note segmentation further work includes: • Algorithm performance improvement • Studying of other approaches • Combining both energy and fundamental frequency contour analysis • Intra-note segment and transition description includes: • Including fundamental frequency contour description • Enhancing the description using non-linear models and studying amplitude and fundamental frequency modulations • Studying possible existing context-related patterns for both energy and fundamental frequency contours • Defining a representative description for the transitions • Get a compact, usable representation and parameterisation of the expressive descriptors into a high level expressivity description • …

  18. Thank you…!

More Related