Bayesian Cognition Winter School at Chamonix, France 9.1.2008

1. Bayesian Cognition Winter School at Chamonix, France 9.1.2008 Bayesian Models for Computational Laban Movement Analysis Jörg Rett and Jorge Dias

2. Intro: Bayesian Models for Computational Laban Movement Analysis Laban Movement Analysis: Model for human behaviour Bayesian Model: Probabilistic model to analyse human interaction

3. Applications Intro: Human Movement Analysis • Analysis • Studies on patients • Rehabilitation • Socially assistive robotics • Social robots Mataric et al., Socially assistive robotics for post-stroke rehabilitation Journal of NeuroEngineering and Rehabilitation, 2007

4. Applications Intro: Applications • Analysis • Studies on patients • Surveillance • Public spaces Datasets and videos of the european project caviarhttp://homepages.inf.ed.ac.uk/rbf/CAVIAR/, 2003

5. Applications Intro: Applications • Analysis • Studies on patients • Surveillance • Public spaces • Virtual Reality • Interactive virtual worlds Enguerran BoissierCharacter animation using Maya software LAAS/ISRReport 05,2005

6. Applications Intro: Applications • Analysis • Studies on patients • Surveillance • Public spaces • Virtual Reality • Interactive virtual worlds • Control Interfaces • Gesture driven control C. Eberst et al., Towards Programming Robots by Gestures, Test-case: Programming Bore Inspection for Small Lotsizes, ICRA, 2006

7. Applications Intro: Skills Skills • Analysis • Studies on patients Human Motion Capture • Tracking • Model based • 3-D vs. 2-D • Surveillance • Public spaces • Virtual Reality • Interactive virtual worlds • Control Interfaces • Gesture driven control R. Urtasun and P. Fua, 3D Tracking for Gait Characterization and Recognition, FGR,2004

8. Applications Intro: Skills Skills • Analysis • Studies on patients Human Motion Capture Face Recognition • Surveillance • Public spaces • Virtual Reality • Interactive virtual worlds • Control Interfaces • Gesture driven control P. Viola and M.J. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, CVPR, 2001

9. Applications Intro: Skills Skills • Analysis • Studies on patients Human Motion Capture Face Recognition • Surveillance • Public spaces Hand Gesture Recognition • Virtual Reality • Interactive virtual worlds • Online Behaviour • Anticipatory Behavior • Control Interfaces • Gesture driven control J. Rett and J. Dias: Gesture Recognition Using a Marionette Model and Dynamic Bayesian Networks (DBNs), ICIAR,2006

10. Applications Intro: Skills Skills • Analysis • Studies on patients Human Motion Capture Face Recognition • Surveillance • Public spaces Hand Gesture Recognition • Virtual Reality • Interactive virtual worlds Laban Movement Analysis • Expressiveness • Semantic descriptor • Control Interfaces • Gesture driven control J. Rett and J. Dias, Human-robot interface with anticipatory characteristics based on Laban Movement Analysis and Bayesian models, ICORR,2007

11. Applications Intro: Methods Skills Methods • Analysis • Studies on patients Human Motion Capture Bayesian Face Recognition • Surveillance • Public spaces SVD Hand Gesture Recognition • Virtual Reality • Interactive virtual worlds Neural Networks Laban Movement Analysis • Control Interfaces • Gesture driven control

12. Laban Movement Analysis (LMA) • Five major components • Set of semantic descriptors (labels) for movements Laban: Major components of LMA Relation- ship Body Effort Space Shape observe describenotateinterprete Method to human movements.

13. Laban: Body Body Relation- ship Effort Space Shape • Body component • Which body parts are moving • How is their movement is related to the body centre (~navel). • Locomotion • Kinematics

14. Space component • Spatial pathways of human movements inside a frame of reference • Three Axes, • Three Planes • Vector Symbols Laban: Space Relation- ship Body Effort Shape Space

15. Effort component • Dynamic qualities of the movement • Inner attitude towards using energy • Four bipolar Effort qualities Laban: Effort Effort Space {Direct, Neutral, Indirect} Weight {Strong, Neutral, Light} Time {Sudden, Neutral, Sustained} Flow {Free, Neutral, Bound} Relation- ship Body Space Shape • Neutral qualities • Single Effort: Rare, difficult to perform • Four Effort: Rare; extreme movements • Three Effort: Most natural • Two Effort: Transitions, failure

16. Drives • One Effort quality is neutral Laban: Effort Effort Relation- ship Body Space Shape

17. Shape component • Emerging from the Body and Space components. • Focused on the body or towards a goal in space Laban: Shape Relation- ship Body Effort Space Shape

18. Modes of interaction with oneself • … with others • … with the external environment. Relationship Laban: Relationship Body Effort Space Shape

19. Assigning semantic descriptors to the movement ‘Punch’ Laban: Summary Body Effort Hands/Head Relation- ship Space Shape Forward High

20. Example: Human–Robot Interaction based on gestures ? Design: Process of designing a Bayesian Model Affirmative, now I am going to perform this action.

21. Design: Phenomenon-description Movements can be distinguished through their expressiveness • Describing the Phenomenon • What is the phenomenon?

22. Design: Phenomenon-description Interesting objects are hands and head • Describing the Phenomenon • What is the phenomenon? • Which features can be observed?

23. Design: Phenomenon-description • Using: • Commercial 3-D motion capture device • Camera based colour tracker • Describing the Phenomenon • What is the phenomenon? • Which features can be observed? • How can the features be extracted?

24. Design: Phenomenon-description 3-D trajectories are represented through three principal planes. • Describing the Phenomenon • What is the phenomenon? • Which features can be observed? • How can the features be extracted? • How can the features be represented? 3-D Vertical plane

25. Design: Phenomenon-description Low-level variables and their sample space. • Describing the Phenomenon Vector symbolsA {O, U, UR, R, DR, D, DL, L, UL} CurvatureK {180, 135, 90, 45, 0, -45, -90, -135} SpeedVel {Zero, Low, Medium, High} Speed GainAcc {Zero, Low, Medium, High} • What is the phenomenon? • Which features can be observed? • How can the features be extracted? • How can the features be represented?

26. Design: Phenomenon-description LMA variables and their relation to the movements • Describing the Phenomenon Movement Punching Effort.Space Direct Effort.Weight Strong Effort.Time Sudden Effort.Flow Neutral Movement Threading a needle Effort.Space Direct Effort.Weight Light Effort.Time Sustained Effort.Flow Bound • What is the phenomenon? • Which features can be observed? • How can the features be extracted? • How can the features be represented? • How do the features relate to the phenomenon

27. Design: Phenomenon-description Relation of low-level variables to LMA variables • Describing the Phenomenon • What is the phenomenon? • Which features can be observed? • How can the features be extracted? • How can the features be represented? • How do the features relate to the phenomenon

28. Design: Bayesian model Bayes-net • Describing the Phenomenon • Building the probabilistic models movement frame I M • Bayes model for Space A B C vector symbols (atoms)

29. Design: Bayesian model Joint distribution • Describing the Phenomenon • Building the probabilistic models P(MIABC) = P(M) P(I) P(A| M I) P(B| M I) P(C| M I) • Bayes model for Space

30. Design: Bayesian model Random variables and their sample space • Describing the Phenomenon • Building the probabilistic models M {punching, ..., pointing}<n> I {1, ..., Imax}<Imax> A {O, F, FR, R, BR, B, BL, L, LF}<9> B {O, U, UR, R, DR, D, DL, L, UL}<9> C {O, U, UF, F, DF, D, DB, B , UB}<9> • Bayes model for Space

31. Design: Bayesian model Bayes-net (upper part) • Describing the Phenomenon • Building the probabilistic models M Ph Movement Phase • Bayes model for Space • Bayes model for Effort E.Sp E.Ti E.We E.Fl Time Space Weight Flow

32. Design: Bayesian model Bayes-net (lower part) • Describing the Phenomenon • Building the probabilistic models E.Sp E.Ti E.We E.Fl Time Space Weight Flow • Bayes model for Space • Bayes model for Effort curva-ture K Vel Acc acce-leration velocity

33. Design: Bayesian model Full model using Space, Effort and Phase • Describing the Phenomenon • Building the probabilistic models • Bayes model for Space • Bayes model for Effort • Bayes model for Phase • Bayes model for Geometry • Bayes model for Shape • Connecting the sub-models • Uncertain (Soft) evidence

34. Design: Learning Question for learning • Describing the Phenomenon. • Building the probabilistic model. • Learning of the probabilities What is the probability of a vector symbol for a movement M=m at frame I=i? movement frame I M • What needs to be learned? P(A| M I) A atoms

35. Design: Learning Conditional Probability Table Asking the question for all movements M and all frames I • Describing the Phenomenon. • Building the probabilistic model. • Learning of the probabilities movement frame I M • What needs to be learned? Learning P(A | M=m1 ... mnI=1 ... imax) A atoms

36. Design: Learning Histogram learning • Describing the Phenomenon. • Building the probabilistic model. • Learning of the probabilities M= pointing I= 1 Example: Davim, trial 3 • What needs to be learned? • How can we learn? O F FR R BR B BL L FL

37. Design: Learning Zero probability problem Some events (Atoms) have not been observed. => Zero probability is assigned => Problem for later classification • Describing the Phenomenon. • Building the probabilistic model. • Learning of the probabilities • What needs to be learned? • How can we learn? O F FR R BR B BL L FL

38. Design: Learning Solution: Learning based on the ‚Laplace Sucession Law‘. • Describing the Phenomenon. • Building the probabilistic model. • Learning of the probabilities na number of occurences of event A=a n number of sets |_A_| possible values of A • What needs to be learned? • How can we learn?

39. Design: Classification Question in 2-D: What is the probability distribution of movements m given the frame i and direction symbols of the vertical plane A? • Describing the Phenomenon • Building the probabilistic model • Learning of the probabilities • Defining the question for classification

40. Design: Classification Question in 3-D: What is the probability distribution of movements m given the frame i and direction symbols of all planes A, B, C? • Describing the Phenomenon • Building the probabilistic model • Learning of the probabilities • Defining the question for classification

41. Design: Continuous Update Likelihood computation For a sequence of n observations of a. • Describing the Phenomenon • Building the probabilistic model • Learning of the probabilities • Defining the question for classification • Continuous update of the results

42. Design: Continuous Update Update in 2-D: • Describing the Phenomenon • Building the probabilistic model • Learning of the probabilities • Defining the question for classification • Continuous update of the results

43. Performance evaluation using confusion tables Experiment 1 Using one 2-D projection (fronto-parallel view), (B atoms) Results: Confusion tables 2 maestro 5 pointing 1 lunging 6 byebye 3 stretch 8 nthrow movements 7 shake Movement 6 Testing in 13 trials of movement byebye 4 ok 1 lunging 2 maestro 3 stretch 4 ok 5 pointing 6 byebye 13 7 shake Horizontal waving Bye-bye sign 8 nthrow

44. Performance evaluation using confusion tables Experiment 1 Using one 2-D projection (fronto-parallel view), (B atoms) Results: Confusion tables 2 maestro 5 pointing 1 lunging 6 byebye 3 stretch 8 nthrow movements 7 shake Movement 8 Testing in 5 trials of movement nthrow 4 ok 1 lunging 2 maestro 3 stretch 4 ok 5 pointing 6 byebye 7 shake Sagittal waving Approach sign 8 nthrow 1 4

45. Performance evaluation using confusion tables Experiment 1 Using one 2-D projection (fronto-parallel view), (B atoms) Results: Confusion tables 2 maestro 5 pointing 1 lunging 6 byebye 3 stretch 8 nthrow movements 7 shake Results Trajectories of nthrow and ok in the vertical plane 4 ok 1 lunging 2 maestro 3 stretch 4 ok 5 pointing 6 byebye 7 shake nthrow ok 8 nthrow 1 4

46. Performance evaluation using confusion tables Experiment 1 Using one 2-D projection (fronto-parallel view), (B atoms) Results: Confusion tables 2 maestro 5 pointing 1 lunging 6 byebye 3 stretch 8 nthrow movements 7 shake Final Results 2-D 95 trials 31 Wrong classifications => Reconition rate 67% 4 ok 1 lunging 2 maestro 3 stretch 4 ok 5 pointing 6 byebye 7 shake 8 nthrow

47. Performance evaluation using confusion tables Experiment 2 Using the three principal planes (horizontal, vertical, sagittal), (A, B, C atoms) Results: Confusion tables 2 maestro 5 pointing 1 lunging 6 byebye 3 stretch 8 nthrow movements 7 shake Movement 8 Testing in 5 trials of movement nthrow 4 ok 1 lunging 2 maestro 3 stretch 4 ok 5 pointing 6 byebye 7 shake Sagittal waving Approach sign 8 nthrow 5

48. Performance evaluation using confusion tables Experiment 2 Using the three principal planes (horizontal, vertical, sagittal), (A, B, C atoms) Results: Confusion tables 2 maestro 5 pointing 1 lunging 6 byebye 3 stretch 8 nthrow movements 7 shake Final Results 3-D 95 trials 21 Wrong classifications => Reconition rate 78% 4 ok 1 lunging 2 maestro 3 stretch Final Results 2-D => Reconition rate 67% 4 ok 5 pointing 6 byebye 7 shake 8 nthrow

49. Performance evaluation using confusion tables Experiment 3 Using one 2-D projection but from a 22° rotated perspective, (B atoms) Results: Confusion tables 2 maestro 5 pointing 1 lunging 6 byebye 3 stretch 8 nthrow movements 7 shake Final Results 2-D perspective 95 trials 48 Wrong classifications => Reconition rate 48% 4 ok 1 lunging 2 maestro Final Results 2-D => Reconition rate 67% 3 stretch 4 ok Final Results 3-D => Reconition rate 78% 5 pointing 6 byebye 7 shake 8 nthrow

50. Anticipation and Certainty Results: Continuous classification i = 0 i = 4 i = 5 i = 6 i = 8 i = 11 certain P(G) 0.91 quite certain 0.63 uncertain i G