Accelerometer-based User Interfaces for the Control of a Physically Simulated Character

1. Accelerometer-based User Interfaces forthe Control of a Physically Simulated Character Takaaki Shiratori Jessica K. Hodgins Carnegie Mellon University

2. Physical Simulation in Games Little Big Planet, 20081 Simulation ONLY for passive objects. - Effective! Jurassic Park: Trespasser, 19982 Simulation for everything. - Very limited. Controllable interface for physically simulated character 1 http://www.littlebigplanet.com/ 2 http://www.mobygames.com/game/trespasser-jurassic-park

3. Physically Simulated Character • Natural-looking motion. • Natural responses to environment/disturbance. Half speed

4. Physically Simulated Character • Natural-looking motion. • Natural responses to environment/disturbance. • Hard to control due to delayed response (anticipation). Speed [m/s] Anticipation Walk Jump 5 4 3 2 1 Time [sec] Walk  Jump 0 0 1.0 2.0 3.0 – Our Hypothesis – Performing similar actions might make the delay seem intuitive.

5. Our Approach • Performance interface: User Imitates character’s motion with Wiimotes. • Wrist interface • Arm interface • Leg interface • Test our hypothesis about delay with user study. Leg interface

6. WiimoteTM • Accelerometer • IR sensor • $40 / unit • 30 million copies 1 2 1 http://www.gizmag.com/go/6773/ 2 http://www.cs.cmu.edu/~johnny/projects/wii/

7. Related Work • Interface for controlling a character • Physically simulated character in 2D. • Virtual navigation • Control “view point”. [Chai and Hodgins 2005] [van de Panne and Lee 2003] [Slyper and Hodgins 2008] • Wii games • Either dynamic or static measurement. [Johnson et al. 1999] [Nintendo 2006] [Nintendo 2007] [Ubisoft 2007] [Templeman et al. 1999] [Razzaque et al. 2002]

8. Overview Acceleration Analysis Moving or not, Freq., Phase Amp., Mean, Inclination 2-3 Wiimotes Controller selection Parameter change Mapping Motion controller (walk, run, jump, step) Physical Simulation Physically simulated motion

9. Overview Acceleration Analysis Moving or not, Freq., Phase Amp., Mean, Inclination 2-3 Wiimotes Controller selection Parameter change Mapping Motion controller (walk, run, jump, step) Physical Simulation Physically simulated motion

10. User Input • Focus on periodicity of character’s motion. Walking Running Jumping Out-of-phase In-phase Basic command: swing Wiimotes

11. Acceleration Analysis Variance Kalman filter Raw acceleration Moving? Amplitude No Yes Inclination Frequency Mean Phase diff. Features L R

12. Frequency • Auto-correlation function a: acceleration data T: current time tp: window size

13. Acceleration Analysis Kalman filter Raw acceleration Moving? Amplitude No Yes Inclination Frequency Mean Phase diff. Features L R

14. Phase Difference • Cross-correlation function a: acceleration data a: mean acceleration T: current time tp: window size

15. Acceleration Analysis Kalman filter Raw acceleration Moving? Amplitude No Yes Inclination Frequency Mean Phase diff. Features L R

16. Inclination Estimation • If Wiimote is not moving, • If Wiimote is moving,  Wiimote’s local coordinate. g y x z t t acc. x acc. x mean

17. Overview Acceleration Analysis Moving or not, Freq., Phase Amp., Mean, Inclination 2-3 Wiimotes Controller selection Parameter change Mapping Motion controller (walk, run, jump, step) Physical Simulation Physically simulated motion

18. Hopper Model Rendering Simulation Hip: Ball joint (3 DoFs) Knee: Slider joint (1 DoF)

19. Basic Motions Stepping in place (Stopping) Walking Running Jumping

20. Walking Controller • Consists of 3 contact states with Proportional-Derivative (PD) controller. Double Support Swing leg contacts ground. Rear leg leaves ground. Fall Rise Support foot passes under hip. Stepping-in-place controller: target velocity = 0 [Raibert and Hodgins, 1991]

21. Running Controller Vertical hip velocity less than zero. • Consists of 4 contact states with PD controller. Upward Flight Downward Flight Front swing leg contacts ground. Front swing leg contacts ground. Support leg leaves ground. Support leg leaves ground. Support foot passes under hip. Support foot passes under hip. Compression Extension Jumping controller: both legs in phase. [Raibert and Hodgins, 1991]

22. Gait Transition • Based on the robustness of motion. Slow Fast Walking Running Walking Running Stepping in place Jumping

23. Overview Acceleration Analysis Moving or not, Freq., Phase Amp., Mean, Inclination 2-3 Wiimotes Controller selection Parameter change Mapping Motion controller (walk, run, jump, step) Physical Simulation Physically simulated motion

24. Mapping Wiimotes to Physical Simulation Slow Fast Walking Running Wii frequency Walking Running Wiimotes in phase. Stepping in place Jumping Wiimotes not moving. Wiimotes out of phase. Height: Wiimote amplitude

25. Wrist Interface • Imitate character’s leg motions with user’s wrists.

26. Wrist Interface • Imitate character’s leg motions with user’s wrists. Walking (Slow swing) Running (Fast swing) Jumping Turning

27. Arm Interface • Imitate arm motion of human’s biped motion. (though the character doesn’t have upper body)

28. Arm Interface • Imitate arm motion of human’s biped motion. (though the character doesn’t have upper body) Walking Running Jumping Turning

29. Leg Interface • Imitate character’s leg motion with user’s legs.

30. Leg Interface • Imitate character’s leg motion with user’s legs. Walking (Slow step) Running (Fast step) Jumping Turning

31. Joystick Interface • Typical usage. Forward Left Right Run Walk Turn in place Step in place Locomotion Jumping

32. User Study • 15 subjects. • Tasks • Straight track completion. • Test track completion. • Questionnaire • Fun, ease of use, stress, familiarity, immersion, how much they liked it? • Free-form questions. Courses:

33. Straight Track Completion • Task • Motion transition at line. • Keep straight walking/running until line. Average failure count • Straight walking: All of our interfaces < Joystick

34. Failure for Straight Walking Wrist Joystick Arm Leg Forward Left Right Run Walk Turn in place Step in place Straight walk Approximate motions Precise manipulation

35. Test Track Completion Simulation failure Jump failure Curve failure + Time to completion Time to failure

36. Result of Test Track Completion Average count Time [sec] Curve failure: Leg interface < joystick Our interfaces are easier to control than joystick.

37. Result of Questionnaire Score “Fun” “Ease of use” “Stress” “Familiarity” “Immersion” “Like”

38. Questionnaire Rating score: • “Immersion”: Wrist, Leg > Joystick • “Like”: Wrist, Leg > Joystick • Free-form questions: • Most subjects did not complain about the delay. • A few subjects complained about the delay of all interfaces (including joystick).

39. Insights from User Study • Delay factors: • Acceleration analysis: 100 – 500 ms (Not included in joystick interface) • Physical simulation (anticipation): 200 – 500 ms • Task completion: Easy to control: our interfaces > joystick. • Questionnaire: “Immersive” and “Like”: our interfaces > joystick. Immersive performance interface might help with the delay issue.

40. Competitive Game

41. Conclusion Summary • Performance interfaces for controlling a physically simulated character. • User study mainly focusing on delay issues. • Performance interface might be able to help with delay issue. Future Work • Reduce delay. • Improve acceleration analysis. • Further user study. • Delay issue: Compare with data-driven control. • Other scenarios (e.g. fighting game, FPS).