Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010

1. Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance Rebecca FiebrinkPerry Cook, AdvisorFPO, 12/13/2010

5. function [x flag histdt] = pagerank(A,optionsu) [mn] = size(A); if (m ~= n) error('pagerank:invalidParameter', 'the matrix A must be square'); end; options = struct('tol', 1e-7, 'maxiter', 500, 'v', ones(n,1)./n, … 'c', 0.85, 'verbose', 0, 'alg', 'arnoldi', … 'linsys_solver', @(f,v,tol,its) bicgstab(f,v,tol,its), … 'arnoldi_k', 8, 'approx_bp', 1e-3, 'approx_boundary', inf,… 'approx_subiter', 5); if (nargin > 1) options = merge_structs(optionsu, options); end; if (size(options.v) ~= size(A,1)) error('pagerank:invalidParameter', … 'the vector v must have the same size as A'); end; if (~issparse(A)) A = sparse(A); end; % normalize the matrix P = normout(A); switch (options.alg) case 'dense’ [x flag histdt] = pagerank_dense(P, options); case 'linsys’ [x flag histdt] = pagerank_linsys(P, options) case 'gs’ [x flag histdt] = pagerank_gs(P, options); case 'power’ [x flag histdt] = pagerank_power(P, options); case 'arnoldi’ [x flag histdt] = pagerank_arnoldi(P, options); case 'approx’ [x flag histdt] = pagerank_approx(P, options); case 'eval’ [x flag histdt] = pagerank_eval(P, options); otherwise error('pagerank:invalidParameter', ... 'invalid computation mode specified.'); end; function [x flag histdt] = pagerank(A,optionsu)

8. useful usable effective, efficient, satisfying

9. ? Machine learning algorithms

10. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

11. interactive computer music

12. Interactive computer music sensed action interpretation response (music, visuals, etc.) human with microphone, sensors, control interface, etc. audio synthesis or processing, visuals, etc. computer

13. Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 1” response computer

14. Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 2” Hi-hat response computer

15. Model of sensed action to meaning sensed action model meaning response computer

16. Example 2: Continuous gesture-to-sound mappings computer

17. Example 2: Continuous gesture-to-sound mappings sensed action interpretation mapping sound generation human + control interface computer

18. A composed system sensed action mapping/model/interpretation mapping/model/interpretation response

19. supervised learning

20. Supervised learning inputs training data algorithm model Training outputs

21. Supervised learning inputs “Gesture 1” “Gesture 2” “Gesture 3” training data algorithm model Training outputs “Gesture 1” Running

22. Supervised learning is useful • Models capture complex relationships from the data and generalize to new inputs. (accurate) • Supervised learning circumvents the need to explicitly define mapping functions or models. (efficient) Has been demonstrated to be useful in musical applications, but no usable, general-purpose tools exist for composers to apply algorithms in their work.

23. Criteria for a usable supervised learning tool for composers Weka and similar general-purpose tools Existing computer music tools ??? 1. General-purpose: many algorithms & applications ✗ ✓ ✓ ✓ ✓ ✗ 2. Runs on real-time signals ✓(some) ✓ 3. Compatible with other compositional tools ✗ ✓ ✓(some) ✓ 4. Supports interaction via a GUI 5. Supports appropriate end-user interactions with the supervised learning process ✗ ✗ ✓

24. Criteria for a usable supervised learning tool for composers Weka and similar general-purpose tools Existing computer music tools ??? Interactively… • create training data • evaluate models • modify training data 1. General-purpose: many algorithms & applications ✗ ✓ ✓ ✓ ✓ ✗ 2. Runs on real-time signals ✓(some) ✓ 3. Compatible with other compositional tools ✗ ✓ ✓(some) ✓ 4. Supports interaction via a GUI 5. Supports appropriate end-user interactions with the supervised learning process ✗ ✗ ✓

25. Interactive machine learning (IML) • Fails and Olsen (2003): training set editing for computer vision systems • More recent work: • application to other domains (Shilman et al. 2006; Fogarty et al. 2008; Amershi et al. 2009; Baker et al. 2009) • other types of interactions, e.g. manipulating confusion matrices(e.g., Talbot et al. 2009, Kapoor et al. 2010)

26. Research questions • Which interactions are possible and useful? • What are the essential differences between IML and conventional ML? • How can IML be used in real-time and creative contexts?

27. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

28. The Wekinator 1. General-purpose: many algorithms & applications • Built on Weka API • Downloadable at http://code.google.com/p/wekinator/(Fiebrink, Cook, and Trueman 2009; Fiebrink, Trueman, and Cook 2009; Fiebrink et al. 2010) 2. Runs on real-time signals ✓ 3. Compatible with other compositional tools ✓ ✓ 4. Supports interaction via a GUI 5. Supports appropriate end-user interactions with the supervised learning process ✓ ✓

29. Running models in real-time Feature extractor(s) .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... time model(s) 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … time Parameterizable process

30. Real-time, interactive design The Wekinator supports user interaction with all stages of the model creation process.

31. Under the hood joystick_x joystick_y webcam_1 … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM volume pitch 3.3098 Class24

32. Under the hood Learning algorithms: Classification: AdaBoost.M1 J48 Decision Tree Support vector machine K-nearest neighbor Regression: MultilayerPerceptron … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM 3.3098 Class24

33. Tailored but not limited to music The Wekinator • Built-in feature extractors for music & gesture • ChucK API for feature extractors and synthesis classes Open Sound Control (UDP) Other modules for sound synthesis, animation, …? Other feature extraction modules

34. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

35. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

36. Study 1: Participatory design process with composers • Method: • 7 composers • 10 weeks, 3 hours / week • Group discussion, experimentation, and evaluation • Iterative design • Final questionnaire • Outcomes: • Composers focused on instrument-building, 2 ultimately used in performance • Much-improved software and lots of feedback(Fiebrink, Trueman, et al., 2010)

37. Study 2: Teaching interactive systems building in an undergraduate course • COS/MUS 314 Spring 2010 (PLOrk) • Focus on building interactive music systems • Method: • One component of the midterm assignment:students building interactive systems using supervised learning • 1 continuous + 1 discrete system • Logging + short answer questions • Outcomes: • Successful project completion, use in midterm and final performances • Logs from 21 students

38. Study 3: Bow gesture recognition • Work with a composer/cellist to build gesture classifiers for a commercial sensor bow, the “K-Bow”

39. Study 3: Bow gesture recognition • Work with a composer/cellist to build gesture classifiers for a commercial sensor bow • 8 classifiers for standard bow gestures: e.g., up/down bow (2 classes), articulation (7 classes) • Method: • All classification tasks defined by cellist and addressed in order of her choice • All 8 tasks in 1st session, 5 tasks in 2nd session • Logging, observations, final questionnaire • Cellist assigned each iteration’s classifier a quality rating (1 to 10) • Outcome: • Successful classifiers created for all 8 tasks (rated “9” or “10”)

40. Study 4: Composer case studies • Clapping Music Machine Variations (CMMV) by Dan Trueman, faculty

41. Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student

42. Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student • G by Raymond Weitekamp, undergraduate

43. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions

44. Discussion of Findings Users took advantage of interaction in their work with the Wekinator. Users employed a variety of model evaluation criteria, and subjective evaluation did not always correlate with cross-validation accuracy. Feedback from the Wekinator influenced users’ actions and goals. The Wekinator was a useful and usable tool. Interactive supervised learning can be a tool for supporting creativity and embodiment.

45. An iterative approach to model-building

46. An iterative approach to model-building (mean per classification task) (mean per student for each task)

47. Frequent modifications to training data in-between re-trainings KBowper-task average PLOrk per-student average

48. Interaction and the training dataset • Training data is an interface for key tasks: • defining the learning problem • clarifying the learning problem to fix errors • communicating changes in problem over time • User incrementally makes model more complex • User changes his mind about how a model should work • providing a “sketch” that the computer fills in

49. Thoughts: Data editing makes sense • The training data is the most effective and appropriate interface for accomplishing these tasks. • Potentially appropriate for many problem domains where: • User has knowledge of learning problem • User is capable of creating examples • The interfaces for creating and editing the training data are crucial to the user’s ability to effectively and efficiently accomplish these tasks.

50. Playalong data recording • Employed by • 3 of 8 composers in Studies 1 and 4 • 8 of 21 PLOrk students • These users found playalong highly useful • Allowed training data to represent more fine-grained information • Enabled some composers to engage their musical and physical expertise • Allowed practice and attention to “feel”