VisHap:

1. Augmented Reality Combining Haptics and Vision VisHap: Guangqi Ye, Jason J. Corso, Gregory D. Hager, Allison M. Okamura Presented By: Adelle C. Knight

2. Agenda • Introduction • Design & Implementation • Vision Subsystem • World Subsystem • Haptic Subsystem • Experimental Results • Future Work • Conclusions

3. Design & Implementation

4. Design & Implementation • Pentium III PC • Linux OS • SRI Small Vision System (SVS) • SVS with STH-MDCS stereo head • PHANToM Premium 1.0A (SensAble Technologies)

5. Vision Subsystem

6. Vision Subsystem • Purpose: • track user’s finger • provide 3D information & video to world • Appearance-based Hand Segmentation • Fingertip Detection and Tracking • VisHap Implementation: • Assume user is interacting using a single finger • Perform finger tracking on the left color image • Compute 3D position of the finger in the coordinate system of the left camera

7. Vision Subsystem:Appearance-based hand segmentation • Basic idea • split image into small tiles & build hue histogram • Start images • on-line learning procedure • Future images • build histogram and carry out pair wise histogram comparison with background model

8. background image foreground image segmentation result Vision Subsystem:Appearance-based hand segmentation • Colour appearance model of human skins • Collect training data • Convert pixels from RGB to HSV colour space • Learn single Gaussian model of hue distribution • Perform check on foreground pixels (filter out non-skin points) • Remove noise with median filter operation

9. Vision Subsystem:Fingertip Detection & Tracking • Detect finger by exploiting geometrical property • Use cylinder with hemispherical cap to approximate shape of finger • Radius of sphere corresponding to fingertip (r) is approximately proportional to reciprocal of depth of fingertip with respect to the camera (z) • r = K/z • Series of criteria checked on candidate fingertips to filter out false fingertips

10. Vision Subsystem:Fingertip Detection & Tracking • Algorithm outputs multiple candidates around true location • Select candidate with highest score to be the fingertip • Kalman Filter to predict position of fingertip (real time tracking)

11. WorldSubsystem

12. World Subsystem • Purpose: • Perform 3D vision/haptics registration • Scene rendering • Notify haptic device about imminent interaction • System calibration (SVS and PHANToM 1.0) • Move haptic device around in field of view of camera • Record more than 3 pairs of coordinates in camera and haptic frames • Calculate optimal absolute orientation solution

13. World Subsystem • Implement Interaction Properties • Database of various interaction modes and object surface properties • Example: • Interaction with virtual wall • Interaction property: slide along • Interaction with button • Interaction property: click

14. HapticSubsystem

15. Haptic Subsystem • Purpose: • Simulate touching experience • Presents suitable force feedback to fingertip • Control scheme: • Control Law • Gravity Compensation for PHANToM • Interaction with Objects

16. Haptic Subsystem:Control Law for Haptic Device • Control Law based on error space to guide haptic device to target position • Low pass filter to achieve smooth control and remove high frequency noise or in time space

17. Haptic Subsystem:Gravity Compensation for PHANToM • Motor torques needed to counteract the wrench applied to the manipulator • Total torque caused by gravity of all parts of device • Smooth and stable trajectory tracking

18. Haptic Subsystem:Interaction with Objects • Simulate interaction forces by adjusting force gain according to object properties and interaction mode. • Transform converts object’s gain matrix to that of haptic device: • VisHap Implementation: • Defined OΛgain as a diagonal matrix with λx, λy , λz its diagonal elements • Z-axis of objects frame is along normal of object’s surface

19. Haptic Subsystem:Interaction with Objects • Example: • Object: button or keyboard key • Destination: center of button’s surface at initial contact • Enter object: user pushes button down • increase λz to proper value to simulate button resistance • Adjust destination point of haptic device to surface of bottom board of the button & increase λz to larger value Relationship of force gain λz and depth d of fingertip under the surface of a button.

20. Experimental Results • Foreground segmentation: used 1st 10 frames to learn appearance model of background • Hue histograms of 8 bins for each 5 x 5 image patch • Test algorithm: record image pairs of background and foreground • Evaluate scheme: compare segmentation result and ground truth classification image • Tested 26 pairs of images: • Average correct ratio: 98.16% • Average false positive ratio: 1.55% • False negative ratio: 0.29%

21. Experimental Results • Virtual Environment • virtual plane in space • Interactions • user moves finger to interact with plane • user moves finger to press fixed button • VisHap is capable of automatically switching interaction objects according to the scene configuration and current fingertip position. Haptic force feedback along normal of object surface and the distance of the fingertip to the object.

22. Future Work • Head mounted display (HMD) • to achieve higher immersive ness and fidelity • Extend virtual environment • incorporate richer sets of objects and interaction modes

23. Conclusions • Generate “complete” haptic experience • Modular framework: • computer vision • haptic device • augmented environment model • Experimental results justify design • Experimental results show flexibility and extensibility of framework