Functional Data Analysis: Techniques for Exploring Temporal Processes in Music

1. Bradley Vines: Good afternoon. I will be discussing (title here). This talk is intended as an introduction and an overview of FDA. Functional Data Analysis:Techniques for Exploring Temporal Processes in Music Bradley W. Vines McGill University

2. Collaborators Bradley Vines: I have collaborated with the following researchers to develop applications of FDA techniques for use in music cognition researche • Daniel Levitin (McGill University) • Carol Krumhansl (Cornell University) • Jim Ramsay (McGill University) • Regina Nuzzo (McGill University) • Stephen McAdams (IRCAM) ICMPC8

3. Talk Outline • What is Functional Data Analysis? • Steps of a typical FDA • Demonstrate some of the major FDA tools • Smoothing • Registration • General Linear Modeling (significance testing) ICMPC8

4. An Example of Functional Data The idea of “tension” has a shared meaning across participants, based upon extra-musical experiences like tension in physical objects, in social situations and in the body. • Solo clarinet performances • 3 Treatment Groups • Auditory only • Visual only • Auditory + Visual • Continuous Tension Judgments I will begin with an example of functional data. This data comes from a study that I presented earlier this week in which we explored the impact of seeing a musician perform. I will be using these data to demonstrate the Functional Data Analysis tools throughout the talk ICMPC8

5. An example of functional data The question is: How can this data be analyzed? Correlations are useful for identifying similarities between data sets as are multiple regressions, but they reduce all of the information to a summary statistic. Here we are also interested in how the relations between the different groups changes over time. This is where Functional Data Analysis is well suited. With functional data analysis software tools and analysis techniques, it is possible to explore changes over time and to understand WHEN important changes or relations are occurring. ICMPC8

6. The meaning of the music and its impact depends upon the relations between events over time and the way that those relations change. What is Functional Data Analysis?(Ramsay & Silverman, 1997) Bradley Vines: Temporal dynamics are an important aspect of music and they are the focus of much music cognition research Examples of time dependent measures in musical stimuli: • For data drawn from continuous processes • Growth curves, market value, movement, ERP’s • Model data as functions of time • Temporal dynamics in music (Vines, Nuzzo, & Levitin, under review) • continuous measurements of emotion • expressive timing profiles • physiological measurements • movement tracking • Software tools available in Matlab and in S-Plus Including measurements like growth curves… • To understand music, we need to know how changes in sound effect a listener and the performer - • Music necessarily occurs through time as the unfolding of related events. Bradley Vines: All of these data involve processes that evolve over time and therefore that may be intuitively thought of as functions of time, which makes FDA techniques a useful and meaningful way to analyze and explore such measures. Because we are interested not only in the current moment in music but also how events are changing over time and even how changes are changing over time (as in expressive timing profiles), it is meaningful and intuitive to think of continuous measurements as functions of time and to model them as such. It makes intuitive sense to think about this kind of data in terms of functions of time I will give the ftp site later in the talk. Tools for visualizing the data, revealing trends in variation and for significance testing ICMPC8

7. Bradley Vines: • Using basis-fitting techniques • Take a number of basic functions and add them together in such a way as to create the desired function (Add together basic functions) • Same process as in sound synthesis • Fourier analysis does just the opposite Modeling data as functions of time • Basis functions • Element functions that can be added together to approximate the data. W1*F1(t) + W2*F2(t) + W3*F3(t)… • A least squares algorithm is used to determine the weighting coefficients. ICMPC8

8. Bradley Vines: • Using basis-fitting techniques • Take a number of basic functions and add them together in such a way as to create the desired function (Add together basic functions) • Same process as in sound synthesis • Fourier analysis does just the opposite Two basis types • Fourier • B-spline • Polynomial functions • Knots Usually, however, data is messy, and non-periodic. B-spline bases are useful for data that do not have a simply periodicity. I will be concentrating on the use of B-spline bases to model functional data ICMPC8

9. Visualizing B-spline Bases ICMPC8

10. Visualizing B-spline Bases ICMPC8

11. Steps in a typical FDA • Representing the data in Matlab: Matrices • Each row: a sample point in time • Each column: an observation (participant/performer) • Third dimension for multivariate observations • As in Schubert’s multi-dimensional continuous interface • Valence • Arousal (Schubert, 1999) ICMPC8

12. Steps in a typical FDA • Modeling the data with functions • Two major considerations: • Order of the B-spline bases • The number of basis functions ICMPC8

13. Steps in a typical FDA • Modeling the data with functions • Two major considerations: • Order of the B-spline bases • The number of basis functions • The order of B-spline bases • Determines how many derivatives will be smooth. ICMPC8

14. Steps in a typical FDA • The number of basis functions • Affects the quality of fit to the data • The more B-splines, the smaller the error • Tradeoff: • Modeling data accurately • Excluding unimportant noise in the data ICMPC8

15. Remember to mention that there were 800 samples. “There are 800 samples shown here” Original Data ICMPC8

16. Modeled Data Correlation = .9975 ICMPC8

17. Modeled Data The choice of # of B-splines really depends upon the assumptions that the researcher can make about the data. If it is known that there is a single objective event that lead to two peaks, then it might be ideal to treat those two peaks with a single curve. Correlation = .97 ICMPC8

18. Bradley Vines: With the data all prepared and modeled with functions, it is possible to go on to use the tools available in FDA. Major FDA Tools

19. Controlling Unwanted Variability • Curvature (high frequency noise) • Smoothing • Amplitude • Scaling • Phase • Registration ICMPC8

20. Nine Tension Judgments ICMPC8

21. ICMPC8

22. ICMPC8

23. Nine Tension Judgments ICMPC8

24. Nine Tension Judgments ICMPC8

25. Nine Tension Judgments Bradley Vines: Note that some of the judgments were shifted ahead or backwards at the two points. ICMPC8

26. Time Warping ICMPC8

27. General Linear Modeling • Functional regression • Functional significance test (F-test) ICMPC8

28. The effect of adding video ICMPC8

29. Functional Linear Model Y(t) = U(t) + B1(t) [if video is added] ICMPC8

30. Results The question is: When is the difference from zero significant? ICMPC8

31. Significance Testing • Analogous components to traditional F-testing: • MSE(t) = SSE(t) / df(error) -with df(error) = N participants - P parameters • MSR(t) = [SSY(t) – SSE(t)]/df(model) -with df(model) = P parameters - 1 • FRATIO(t) = MSR(t)/MSE(t) ICMPC8

32. Significance Testing An F-value that is itself a function of time. ICMPC8

33. Other FDA techniques that are available • Analysis of covariance • Functional correlation analysis • Canonical correlation analysis • Principal Components Analysis ICMPC8

34. FUNCTIONAL DATA ANALYSIS: TECHNIQUES FOR EXPLORING TEMPORAL PROCESSES IN MUSIC • Prof. James Ramsay’s ftp site: http://www.psych.mcgill.ca/faculty/ramsay/fda.html bradley.vines@mail.mcgill.ca ICMPC8

35. Smoothing • The smoothing parameter, lambda, controls the curvature of a function. • Trade off between perfect fit to the original data and a best linear approximation for the data. Penalizes variance ICMPC8

36. Smoothing • Examples of curves before and after smoothing (try to find a good singly participant who is nice and dynamic for all of this, or a mean curve, I suppose) ICMPC8

37. Principal Components Analysis • Traditional statistics: • Identifying major modes of variation • Reducing the number of dimensions in the data • Determine which variables are related • Functional analogue: • Reveals major modes of variation • Can reveal trends in phase and in magnitude ICMPC8

38. Principal Components Analysis • Monthly temperature data (available on the ftp website) • Weather stations across Canada • Exploring trends in the data and grouping weather stations ICMPC8

39. Monthly Weather Data Bradley Vines: 32 weather stations ICMPC8

40. Eigenvalues, VARIMAX PCA ICMPC8

41. VARIMAX Principal Components ICMPC8

42. VARIMAX Principal Components ICMPC8

43. VARIMAX Principal Components ICMPC8

44. VARIMAX Principal Components ICMPC8

45. Component Scores ICMPC8