Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010

1. Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance Rebecca FiebrinkPerry Cook, AdvisorFPO, 12/13/2010

2. Source: googleisagiantrobot.com

5. function [x flag histdt] = pagerank(A,optionsu) [mn] = size(A); if (m ~= n) error('pagerank:invalidParameter', 'the matrix A must be square'); end; options = struct('tol', 1e-7, 'maxiter', 500, 'v', ones(n,1)./n, … 'c', 0.85, 'verbose', 0, 'alg', 'arnoldi', … 'linsys_solver', @(f,v,tol,its) bicgstab(f,v,tol,its), … 'arnoldi_k', 8, 'approx_bp', 1e-3, 'approx_boundary', inf,… 'approx_subiter', 5); if (nargin > 1) options = merge_structs(optionsu, options); end; if (size(options.v) ~= size(A,1)) error('pagerank:invalidParameter', … 'the vector v must have the same size as A'); end; if (~issparse(A)) A = sparse(A); end; % normalize the matrix P = normout(A); switch (options.alg) case 'dense’ [x flag histdt] = pagerank_dense(P, options); case 'linsys’ [x flag histdt] = pagerank_linsys(P, options) case 'gs’ [x flag histdt] = pagerank_gs(P, options); case 'power’ [x flag histdt] = pagerank_power(P, options); case 'arnoldi’ [x flag histdt] = pagerank_arnoldi(P, options); case 'approx’ [x flag histdt] = pagerank_approx(P, options); case 'eval’ [x flag histdt] = pagerank_eval(P, options); otherwise error('pagerank:invalidParameter', ... 'invalid computation mode specified.'); end; function [x flag histdt] = pagerank(A,optionsu)

9. useful usable effective, efficient, satisfying

10. ? Machine learning algorithms

11. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

12. interactive computer music

13. Interactive computer music sensed action interpretation response (music, visuals, etc.) human with microphone, sensors, control interface, etc. audio synthesis or processing, visuals, etc. computer

14. Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 1” response computer

15. Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 2” Hi-hat response computer

16. Model of sensed action to meaning sensed action model meaning response computer

17. Example 2: Continuous gesture-to-sound mappings

18. Example 2: Continuous gesture-to-sound mappings sensed action interpretation mapping sound generation human + control interface computer

19. A composed system sensed action mapping/model/interpretation mapping/model/interpretation response

20. supervised learning

21. Supervised learning inputs training data algorithm model Training outputs

22. Supervised learning inputs “Gesture 1” “Gesture 2” “Gesture 3” training data algorithm model Training outputs “Gesture 1” Running

23. Supervised learning is useful • Models capture complex relationships from the data and generalize to new inputs. (accurate) • Supervised learning circumvents the need to explicitly define mapping functions or models. (efficient) So why isn’t it used more often in computer music? Useful but not necessarily usable

24. Criteria for a usable supervised learning tool for composers Weka and similar general-purpose tools Existing computer music tools ??? 1. General-purpose: many algorithms & applications ✗ ✓ ✓ ✗ ✓ ✓ 2. Runs on real-time signals ✗ ✓(some) ✓ 3. Compatible with other compositional tools ✓ ✓(some) ✓ 4. Supports interaction via a GUI 5. Supports appropriate end-user interactions with the supervised learning process ✗ ✗ ✓

25. Interactive machine learning (IML) • Fails and Olsen (2003): training set editing for computer vision systems • Also: handwriting analysis web image classification document classification (Shilman et al. 2006; Fogarty et al. 2008, Amershi et al. 2009; Baker et al. 2009) • Other research investigates other interactions, e.g. classifier ensemble manipulation (Talbot et al. 2009), confusion matrix manipulation (Kapoor et al. 2010) • Relevant research questions explored in my work: • Which interactions are possible and useful? • What are the essential differences between IML and conventional ML? • How can IML be used in real-time and creative contexts?

26. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

27. The Wekinator 1. General-purpose: many algorithms & applications • Built on Weka API • Downloadable at http://code.google.com/p/wekinator/(Fiebrink, Cook, and Trueman 2009; Fiebrink, Trueman, and Cook 2009; Fiebrink et al. 2010) 2. Runs on real-time signals ✓ 3. Compatible with other compositional tools ✓ ✓ 4. Supports interaction via a GUI 5. Supports appropriate end-user interactions with the supervised learning process ✓ ✓

28. A tool for running models in real-time Feature extractor(s) .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... time model(s) 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … time Parameterizable process

29. A tool for real-time, interactive design Wekinator supports user interaction with all stages of the model creation process.

30. Under the hood joystick_x joystick_y webcam_1 … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM volume pitch 3.3098 Class24

31. Under the hood Learning algorithms: Classification: AdaBoost.M1 J48 Decision Tree Support vector machine K-nearest neighbor Regression: MultilayerPerceptron … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM 3.3098 Class24

32. Tailored but not limited to music The Wekinator • Built-in feature extractors for music & gesture • ChucK API for feature extractors and synthesis classes Open Sound Control (UDP) Control messages Other feature extraction modules Other modules for sound synthesis, animation, …?

33. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

34. Wekinator in performance

35. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

36. Study 1: Participatory design process with composers • Method: • 7 composers • 10 weeks, 3 hours / week • Group discussion, experimentation, and evaluation • Iterative design • Final questionnaire • Outcomes: • A focus on instrument-building • Much-improved software and lots of feedback(Fiebrink, Trueman, et al., 2010)

37. Study 2: Teaching interactive systems building in an undergraduate course • COS/MUS 314 Spring 2010 (PLOrk) • Focus on building interactive music systems • Method: • Wekinator midterm assignment • 1 continuous + 1 discrete system • Limited choice of controller + synthesis, balanced task order • Logging + questionnaire data • Observations of midterm and final performances • Outcomes: • Successful project completion • Logs from 21 students

38. Study 3: Bow gesture recognition • Work with a composer/cellist to build gesture classifiers for a commercial sensor bow, the “K-Bow”

39. Study 3: Bow gesture recognition • Work with a composer/cellist to build gesture classifiers for a commercial sensor bow • Method: • Work with cellist to build 8 classifiers for standard bow gestures:Up/down bow, on/off string, grip engaged/disengaged, bow rolled/not rolled, horizontal position, vertical position, speed, articulation • First session: all 8 classification tasks • Supplemented with second session: 5 “hard” tasks repeated • Logging (data + actions), observations, final questionnaire • Cellist assigned each iteration’s model a rating of quality (1 to 10) • Outcomes: • Successful models created for all 8 tasks (rated “9” or “10”)

40. Study 4: Composer case studies • CMMV by Dan Trueman, faculty

41. Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student

42. Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student • G by Raymond Weitekamp, undergraduate

43. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

44. Discussion of Findings Users’ interactions with the Wekinator(i.e., first reasons why interaction is useful) Users’ goals for the trained models The Wekinator’s influence on users’ actions and goals Usability and usefulness of the Wekinator The Wekinator as a tool for supporting creativity and embodiment

45. [interaction summary]

46. Users’ interactions with the Wekinator • Users in all studies iteratively modified the learning problem, retrained, and re-evaluated. • Editing the training dataset was used more frequently than changing the learning algorithm or features.

47. PLOrk students’ actions (Study 2)

48. K-Bow actions (Study 3)

49. Composers’ actions (Study 1)

50. Key questions • How and why did users edit the training data? • What methods and criteria did users employ to evaluate the trained models?