A system for automatic animation of piano performances

1. A system for automatic animation of piano performances Alysha Bogaers Joel Morisset

2. Overview • Generates 3D animations for piano performance from MIDI file. • Combines Graph Theory, Geometric Constraints, and Piano Theory. • Generates realistic finger placement, including: • finger crossover • Arpeggio

3. Problem Statement • Purpose: Automatically create 3D animations of piano playing • Why? • Hard for amateur players (actors?) to play "properly" considering finger placements • Hard to animate: • high dimension poses with many joints • exact musical timing constraints which have to hold • occlusions make motion capture hard • even if motion capture worked, piece must be played exactly like recording to have correct velocity/volume/timing etc

4. Related Work

5. String Instruments • Use of statistical approaches • Learning method that evaluate a cost function, given a playing style. • Use of genetic algorithm to create fingering. (not elegant, but playable) • Interactive program that models various performance possibilities for the same music for different instruments • The Handrix system that generates guitar fingering using procedural algorithm.

6. Piano • Rule based expert systems that generates piano fingering. (Cases where the rules are not applicable) • 2D piano key animation given any input music. • Use of a Greedy algorithm to generate the optimal fingering for Piano or Guitar

7. Human Modeling • Anatomical approach to building a hand model • Kinematic approach • Effective physics-based approach

8. In Depth Explanation

9. In Depth Explanation Basic steps: • Find placement of the “used” fingers for chords • Find sequence of chord fingerings with a minimum loss of energy • Find positions of the “unused” fingers such that these are in natural positions or “ready to play” the next chords • Find natural hand position from constraints on the fingers, wrist and contact surface

10. Instructed Fingers • Compute cost of maintaining a hand pose for a specific chord • Compute cost for moving between different fingerings of subsequent chords • Find shortest path between chord fingerings

11. Non-Instructed Fingers • Consider next 1-4 chords for positioning of “unused” fingers • If a finger is used in the next chord, it should be positioned such that it has to move little from the current chord fingering • Fingers should not be too far apart for comfortable playing

12. Hand Model • 16 Joints • 27 DOFs • 6 DOFs for the wrist • 1 DOFs for DIP • 1 DOFs for PIP • 2 DOFs for MCP • 3 DOFs for thumb base • Each finger has IK from the base to the tip • IK used for pressing/releasing the keys • FK is used for the in between movements

13. Fingers, Wrist, and Hand Position And Rotation • The generated fingering is used to decide the position of the fingertips along the Z axis • The finger base is decided by the wrist position • The fingertip, wrist position and orientation have to be calculated

14. Fingers, Wrist, and Hand Position And Rotation • Fingertips: • Z is determined by which keys are pressed • X is determined by the wrist position and the piano key range • Y is the height of a black or white key • Wrist Position: • Determined by the fact that X will move forward and the Y down the more spread the fingers are. • Fingers relative position along X in a standard pose with influence prioritized 1,5,2,4,3 • The Z will move further from the thumb and closer to the little finger • Weighted sum of all fingertip position with larger weights for the thumb and little finger

15. Fingers, Wrist, and Hand Position And Rotation Largest wrist orientation Wrist Orientation: • Along Y: • Along Z: The wider the range the lower the wrist • Along X: keeps hand parallel with the piano Uses the ring and index finger to define a line in 3D which projection is perpendicular to X and can be evaluated to get the rotation along X Orientation of ray from wrist to Fingertip i for each finger i (1-5) Largest Angle Rotation weight influenced by the finger distribution Precomputed based on Motion Capture Data Standard pose pressing 5 neighboring keys

16. Optimized Hand Poses • Optimize by finding a wrist pose sequence which minimizes the overall cost of the wrist movements between chords

17. Hand Motion Animation • Generate a natural up- and downwards motion of the wrist from motion capture of performances • Introduce dependencies between instructed and non-instructed fingers • Generate realistic “feedback” on the wrist after pressing keys

18. DEMO

19. Critical Analysis Negative: • Result seems Stiff • There where a few Erratas Positive: • Good Paper • Well Structured • Explained in detail

20. Future Work • Extension to other key pressing instruments • Method does not resolve interpenetration of fingers and the sides of black keys • Generate more emotional piano playing • Could be beneficial to apply machine learning to determine standard fingering sequences • Implement various hand size models • Playing music that require the two hand to play together • Consider rotation of joints within fingers

21. Questions