1.05k likes | 1.19k Views
Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. Rebecca Fiebrink Perry Cook, Advisor FPO, 12/13/2010. Source: googleisagiantrobot.com. function [ x flag hist dt ] = pagerank(A,optionsu ) [ m n ] = size(A ); if ( m ~= n )
E N D
Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance Rebecca FiebrinkPerry Cook, AdvisorFPO, 12/13/2010
function [x flag histdt] = pagerank(A,optionsu) [mn] = size(A); if (m ~= n) error('pagerank:invalidParameter', 'the matrix A must be square'); end; options = struct('tol', 1e-7, 'maxiter', 500, 'v', ones(n,1)./n, … 'c', 0.85, 'verbose', 0, 'alg', 'arnoldi', … 'linsys_solver', @(f,v,tol,its) bicgstab(f,v,tol,its), … 'arnoldi_k', 8, 'approx_bp', 1e-3, 'approx_boundary', inf,… 'approx_subiter', 5); if (nargin > 1) options = merge_structs(optionsu, options); end; if (size(options.v) ~= size(A,1)) error('pagerank:invalidParameter', … 'the vector v must have the same size as A'); end; if (~issparse(A)) A = sparse(A); end; % normalize the matrix P = normout(A); switch (options.alg) case 'dense’ [x flag histdt] = pagerank_dense(P, options); case 'linsys’ [x flag histdt] = pagerank_linsys(P, options) case 'gs’ [x flag histdt] = pagerank_gs(P, options); case 'power’ [x flag histdt] = pagerank_power(P, options); case 'arnoldi’ [x flag histdt] = pagerank_arnoldi(P, options); case 'approx’ [x flag histdt] = pagerank_approx(P, options); case 'eval’ [x flag histdt] = pagerank_eval(P, options); otherwise error('pagerank:invalidParameter', ... 'invalid computation mode specified.'); end; function [x flag histdt] = pagerank(A,optionsu)
useful usable effective, efficient, satisfying
? Machine learning algorithms
Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions
Interactive computer music sensed action interpretation response (music, visuals, etc.) human with microphone, sensors, control interface, etc. audio synthesis or processing, visuals, etc. computer
Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 1” response computer
Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 2” Hi-hat response computer
Model of sensed action to meaning sensed action model meaning response computer
Example 2: Continuous gesture-to-sound mappings sensed action interpretation mapping sound generation human + control interface computer
A composed system sensed action mapping/model/interpretation mapping/model/interpretation response
Supervised learning inputs training data algorithm model Training outputs
Supervised learning inputs “Gesture 1” “Gesture 2” “Gesture 3” training data algorithm model Training outputs “Gesture 1” Running
Supervised learning is useful • Models capture complex relationships from the data and generalize to new inputs. (accurate) • Supervised learning circumvents the need to explicitly define mapping functions or models. (efficient) So why isn’t it used more often in computer music? Useful but not necessarily usable
Criteria for a usable supervised learning tool for composers Weka and similar general-purpose tools Existing computer music tools ??? 1. General-purpose: many algorithms & applications ✗ ✓ ✓ ✗ ✓ ✓ 2. Runs on real-time signals ✗ ✓(some) ✓ 3. Compatible with other compositional tools ✓ ✓(some) ✓ 4. Supports interaction via a GUI 5. Supports appropriate end-user interactions with the supervised learning process ✗ ✗ ✓
Interactive machine learning (IML) • Fails and Olsen (2003): training set editing for computer vision systems • Also: handwriting analysis web image classification document classification (Shilman et al. 2006; Fogarty et al. 2008, Amershi et al. 2009; Baker et al. 2009) • Other research investigates other interactions, e.g. classifier ensemble manipulation (Talbot et al. 2009), confusion matrix manipulation (Kapoor et al. 2010) • Relevant research questions explored in my work: • Which interactions are possible and useful? • What are the essential differences between IML and conventional ML? • How can IML be used in real-time and creative contexts?
Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions
The Wekinator 1. General-purpose: many algorithms & applications • Built on Weka API • Downloadable at http://code.google.com/p/wekinator/(Fiebrink, Cook, and Trueman 2009; Fiebrink, Trueman, and Cook 2009; Fiebrink et al. 2010) 2. Runs on real-time signals ✓ 3. Compatible with other compositional tools ✓ ✓ 4. Supports interaction via a GUI 5. Supports appropriate end-user interactions with the supervised learning process ✓ ✓
A tool for running models in real-time Feature extractor(s) .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... time model(s) 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … time Parameterizable process
A tool for real-time, interactive design Wekinator supports user interaction with all stages of the model creation process.
Under the hood joystick_x joystick_y webcam_1 … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM volume pitch 3.3098 Class24
Under the hood Learning algorithms: Classification: AdaBoost.M1 J48 Decision Tree Support vector machine K-nearest neighbor Regression: MultilayerPerceptron … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM 3.3098 Class24
Tailored but not limited to music The Wekinator • Built-in feature extractors for music & gesture • ChucK API for feature extractors and synthesis classes Open Sound Control (UDP) Control messages Other feature extraction modules Other modules for sound synthesis, animation, …?
Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions
Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions
Study 1: Participatory design process with composers • Method: • 7 composers • 10 weeks, 3 hours / week • Group discussion, experimentation, and evaluation • Iterative design • Final questionnaire • Outcomes: • A focus on instrument-building • Much-improved software and lots of feedback(Fiebrink, Trueman, et al., 2010)
Study 2: Teaching interactive systems building in an undergraduate course • COS/MUS 314 Spring 2010 (PLOrk) • Focus on building interactive music systems • Method: • Wekinator midterm assignment • 1 continuous + 1 discrete system • Limited choice of controller + synthesis, balanced task order • Logging + questionnaire data • Observations of midterm and final performances • Outcomes: • Successful project completion • Logs from 21 students
Study 3: Bow gesture recognition • Work with a composer/cellist to build gesture classifiers for a commercial sensor bow, the “K-Bow”
Study 3: Bow gesture recognition • Work with a composer/cellist to build gesture classifiers for a commercial sensor bow • Method: • Work with cellist to build 8 classifiers for standard bow gestures:Up/down bow, on/off string, grip engaged/disengaged, bow rolled/not rolled, horizontal position, vertical position, speed, articulation • First session: all 8 classification tasks • Supplemented with second session: 5 “hard” tasks repeated • Logging (data + actions), observations, final questionnaire • Cellist assigned each iteration’s model a rating of quality (1 to 10) • Outcomes: • Successful models created for all 8 tasks (rated “9” or “10”)
Study 4: Composer case studies • CMMV by Dan Trueman, faculty
Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student
Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student • G by Raymond Weitekamp, undergraduate
Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo and video • User studies • Findings and Discussion • Contributions, Future Work, and Conclusions
Discussion of Findings Users’ interactions with the Wekinator(i.e., first reasons why interaction is useful) Users’ goals for the trained models The Wekinator’s influence on users’ actions and goals Usability and usefulness of the Wekinator The Wekinator as a tool for supporting creativity and embodiment
Users’ interactions with the Wekinator • Users in all studies iteratively modified the learning problem, retrained, and re-evaluated. • Editing the training dataset was used more frequently than changing the learning algorithm or features.
Key questions • How and why did users edit the training data? • What methods and criteria did users employ to evaluate the trained models?