Project Hands.DVI - KickOff Meeting Italian Institute of Technology, IIT, Genova, 24.01.2011

1. HANDS.DVI Centro di Ricerca “E. Piaggio” – University of Pisa Project Hands.DVI - KickOff Meeting Italian Institute of Technology, IIT, Genova, 24.01.2011

2. Introduction to project HANDS.DVI • The project HANDS.DVI will contribute to the development • of a unified structure for programming and controlling robotic hands based on a number of fundamental primitives, and • abstracting, to the extent possible, from the specifics of their kinematics, mechanical construction, sensor equipment • The project will propose a new DeVice-Independent programming and control framework for robotic HANDS • This project builds upon well-founded results on sensorimotor synergies developed in neuroscience and preliminary used in some special robotic applications. The project HANDS.DVI will exploit initial results of the research activity developed in the European Project The Hand Embodied (THE)

3. Motivation and objective of project THE • The project THE is focused on the study of how the embodied characteristics of the human hand and its sensors, the sensorimotor transformations, affect and determine the learning and control strategies we use for such fundamental cognitive functions as exploring, grasping and manipulating • The project hinges about the conceptual structure and the geometry of such enabling constraints, or synergies: correlations in redundant hand mobility (motor synergies), correlations in redundant cutaneous and kinaesthetic receptors readings (multi-cue integration), and overall sensorimotor system synergies • These key ideas aim at advancing the state of the art in artificial systems for robotic manipulation, haptic, and neuroprosthetic interfaces

4. Motivation and objective of project HANDS.DVI • The device independent mapping will be mediated by a model of an anthropomorphic robotic hand (paradigmatic hand) able to capture the idea of synergies in human hands. This model will be used to design the new control strategies for both anthropomorphic and non–anthropomorphic robotic hands • The mapping between the paradigmatic hand and the robotic hands can be obtained as the result of an optimal control strategy. A possible solution is to organize this function as two hierarchical levels: • in the high level stage only a few elements, the synergies, are controlled • in the low level the synergies are mapped to the actual kinematic variables of the robotic hand we want to control

5. Contents of the presentation • Force control of robotic grasping: the role of hand synergies in the optimal choice of grasping forces • Extending the paradigmatic hand model: modelling and control of biomorphic fingers using tendon actuation and rolling contacts • Mapping synergies between the paradigmatic hand and robotic hands with different kinematics: “DLR/HIT Hand II” (DLR) and “THE First Hand” (UNIPI) - direct kinematic mapping of the contact points or definition of an optimal function based on grasping forces • Synergies definition: the Motion Capture system and statistical methods for data analysis

6. On the Role of Hand Synergies in the Optimal Choice of Grasping Forces

7. Objective of the present study • Investigate analytically and numerically the role that limiting the DoFs of an hand via synergistic underactuation have on: • making possible precision/whole-hand grasps; • grasping force distribution; • grasp optimality evaluated with respect to grasp quality indices; • robustness with respect to physical characteristics (e.g., compliance).

8. Fundamental Equations Equilibrium Equations (object) (hand) Congruence Equations (object) (hand) where: (Grasp matrix: maps contact forces to object wrench ) (Hand Jacobian: maps joint velocities to fingertip twists ) (maps contact forces to joint torques ) (maps the EE twist to twists of the object contact points )

9. Contact compliance model j-th limb object Penetration of the fingertip point into the object surface Contact force (penalty formulation) By properly juxtaposing the expression for all the contact forces (contact stiffness matrix) (contact compliance matrix)

10. Joint compliance model joint actual positions joint reference positions Joint torque costitutive equation By properly juxtaposing the expression for all the joints (joint stiffness matrix) (joint compliance matrix)

11. Soft Synergy Model • We introduce a synergy vector whose components can be interpreted as “high level knobs” controlling joint coordination patterns; • In the classical synergistic model: • the hand configuration is rigidly constrained to a dimensional sub-manifold of the joint configuration manifold. Therefore the virtual variations allowed are • Synergy matrix found through Principal Component Analysis (PCA) of recorded grasping experiments [Santello et al. 1998]; • In the soft synergy model (proposed) synergistic displacements command joint reference positions • The actual hand configuration depends on hand-object interaction (hand adaptability to different grasping conditions). (synergy matrix)

12. Soft-Synergy Concept actual joint position coupling defines the synergy vector on references synergistic displacements (actual DoF)

13. Solution of the Grasping Problem with Synergies • For a general grasping system applying a wrench to the object, i.e., • the contact force distribution is given by • where: = particular solution (minimizes the strain energy) ( weighted right inverse of ) = active internal forces that can be commanded via synergistic displacements If no underactuation takes place (independent joint control): = passive (preload) internal contact forces

14. Internal Controllable Forces Internal controllable forces directly dependent on synergistic variations Defining it is possible to write also (coefficient of the linear combination) The above form turns out to be very convenient in optimizing the grasping forces. Direct employment of is not efficient since it entails the explicit calculation of . A more efficient algorithm can be employed based on the intersection of subspaces.

15. Qualitative analysis of Hand Embodiment (increases) 1) Assume each has non-null projection onto , then (a plateau is reached) 2) Consider a grasp for which indep. joint control bears • 3) For the underactuated case, increase the no. of synergies engaged one by one (increase the number of columns in ): • For • For • Therefore if is “small” (2, 3), embodiment with an equal number of synergies does not endanger ability to exert the same grasping forces with great control simplification.

16. Grasping Force Optimization Problem • Given a grasp characterized by • weighted right inverse of the Grasp Matrix ; • basis for the subspace of internal controllable forces by employing a given set of synergies; • basis for the subspace on internal passive (preload) forces by employing a given set of synergies; • External wrench to be resisted • Grasping Force Optimization Problem (GFOP) In the tests performed: , [N] Problem feasibility force-closure (normal components)

17. Paradigmatic Hand Model [M] • Paradigmatic Hand Model features: • model of the Cyberglove employed in the work of Santello et al. (1998) for which high quality experimental data were recorder and made available to us; • the only “plant” for which postural synergies are rigorously defined; • 15 DoFs • 4 DoFs[T]: Rot., Abd., Metac., Interph.; • 3 DoFs[I, R, L]: Abd., Metac., Prox. Interph.; • 2 DoFs[M]: Metac., Prox. Interph. • limbs represented as cylinders; • anthropometric data relative to one subject employed in the experimental tests. [R] [I] [L] [T]

18. Numerical tests – Precision grasp • As a representative of precision grasp layout we consider the grasp of a cherry. • Grasp analysis (independent joint control): • 3 contact points: [T] Interph., [I] & [M] Prox. Interph.; • contact constraints: PCWF; • ; • ; (internal) • ; (internal, controllable) • ; (internal, passive) • force-closure: yes! • Questions: • What is the effect of hand embodiment for this grasp? • Is the grasp still force-closure? For which synergy/synergies? • What is the effect of synergies on grasp optimality criteria? active limbs cherry.avi

19. Precision grasp – Contact Stiffness Var.ns • Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp. • The grasp is force-closure already for the 1-st synergy. • Limited effect of Contact Stiffness variation, units [N/mm].

20. Precision grasp – Joint Stiffness Var.ns • Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp. • The grasp is force-closure already for the 1-st synergy. • Limited effect of Joint Stiffness variation, units [N/mm].

21. Numerical tests – Power grasp • As a representative of power grasp layout we consider the grasp of an ashtray. • Grasp analysis (independent joint control): • 11 contact points: [T] 3 cps; [I], [M], [R], [L] 2cps each; • contact constraints: PCWF; • ; • ; (internal) • ; (internal, controllable) • ; (internal, passive) • force-closure: yes! • Questions: • What is the effect of hand embodiment for this grasp? • Is the grasp still force-closure? For which synergy/synergies? • What is the effect of synergies on grasp optimality criteria? ashtray.avi

22. Power grasp – Contact Stiffness Var.ns • Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp. • The grasp is not always force-closure for the 1-st synergy. • Some effects of Contact Stiffness variation, units [N/mm].

23. Power grasp – Joint Stiffness Var.ns • Variation of the cost function (2-norm of contact force vector) as more synergies are engaged into the grasp. • The grasp is not always force-closure for the 1-st synergy. • Some effects of Joint Stiffness variation, units [N/mm].

24. Conclusions • Theoretical results: • a way has been proposed to solve the force decomposition and optimization problem for hands with synergies; • a way to map synergies from the kinematic domain (where have been observed) to the force domain (soft-synergies). • Numerical results: • force-closure properties of grasps strongly depends on which synergies are used to control the hand; • if the first few synergies (PCs) are not actively controlled, force-closure can be obtained only in many more DoFs; • no improvement beyond the first three synergies for precision grasp, continuous but small improvements in the whole-hand grasp case; • results are consistently robust with respect to different values of the stiffness parameters (uncertainty in their knowledge/control).

25. Modelling and Control of a Tendon Actuated Biomorphic Hand

26. Biomorphic reference scheme

27. Contact points move over the surfaces Finger Object Contact frames Convective coords Palm Object/finger contact [M. Gabiccini, A. Bicchi, GRSSP Workshop, 2008]

28. Total wrench on each limb Resultant wrench on the -th body: where : is the wrench applied by the tendons expressed in body frame is the wrench applied by the contact forces expressed in body frame

29. Solution of the problem If we consider the complete system, we obtain: the system must be in equilibrium under an external wrench , therefore: We search a vector such that the system is in equilibrium and all the friction/tendon constraints are satisfied (tendon tension must be positive): where is the particular solution; is the homogeneous solution that not affect the overall system eq. The vector must be chosen by means of an optimization routine to fulfill all the constraints. This can be formulated as a convex optimization problem.

30. Control scheme TRAJECTORY GENERATION COMPUTED TORQUE OPTIMIZATION ROUTINE INITIAL CONDITIONS SYSTEM DYNAMICS SOLUTION (TENDON TENSIONS) CONTACT KINEMATICS

31. System layout The hand/object system consists of: • Three-fingered hand grasping a ball; • Each finger is a 4-d.o.f. serial manipulator with unilateral joints; • The first joint (MCP) is a 2-d.o.f. hyperboloidic (saddle joint); • The 2-nd and 3-rd joints are hinge joints with 1-d.o.f. • There are 8 tendons which guaranteed the integrity of the system

32. Task • Task for the hand/object system: • the object has to track a prescribed trajectory (e.g., G follows ellipse in vert. plane); • integrity of the hand/object contacts and integrity of the finger limbs have to be • guaranteed by proper co-contraction forces in the tendons; • integrity meaning: • contact forces within friction cone boundaries; • max/min contact force below/above prescribed limits; • max/min tendon tensions below/above prescribed limits (e.g., tendons • cannot pull)

33. Videos of simulations • Move the Center of Gravity of the ball along an ellipse in the vertical plane • Rotate the ball around a vertical axis • Move the Center of Gravity of the ball up and down

34. 16 DoFs Robotic Hand Prototype

35. Hand Features Kinematic model of the “First Hand” is very similar to the paradigmatic hand model • 16 DoFs • Five fingered hand • Tendon driven • One rotoidalspring for each joint (only one tendon is needed) • Low cost • Human sized • Human-like movements • Servo controlled

36. Hand Schematics PHALANGES ABDUCTIONS Index, Middle, Ring and Little • Middle finger doesn’t have the abduction DOF • Thumb and Little abductions are independently controlled • Proximal phalangesare independently controlled w.r.t intermediate and distal phalanges • Intermediate and distal phalanges are dependently controlled • THUMB • Phalanges are independently controlled PALM • Two rotoidal joints to close palm on itself

37. Grasping Trials - 1

38. Grasping Trials - 2

39. Hand Synergies First Synergy Second Synergy first.mp4 second.mp4

40. Hand Posture Tracking: Motion Capture System

41. Experimental Setup • Motion capture system: • active leds, impulse cameras, server computer, led controllers, led base station • Easy to reconfigure and calibrate • Fast and reliable data • transmission (up to 480 Hz) • Possibility to connect several clients and integrate measurements of grasping forces • Problems to overcome: • Optimal marker positioning (phalanges or joints) • Kinematic hand model that best fits experimental data • Observability, model parameters estimation and posture reconstruction using nonlinear filters (EKF) • Statistical methods for data analysis

42. Statistical Methods • Modelling grasping movements using statistical methods: Hidden Markov Models and Gaussian Mixture Models • Two Objectives: • Task replication using regression techniques: Gaussian Mixture Regression • Tasks classification from continuous movements while grasping different objects • Problem of data dimension reduction (15 dofs for the paradigmatic hand) using linear and nonlinear PCA • Hand posture reconstruction with inaccurate data using a-priori information and its application to a low cost glove (sensory synergies).

43. Thanks for your attention