Free-hand Pointing for Identification and Interaction with Distant Objects

1. Free-hand Pointing for Identification and Interaction with Distant Objects Sonja Rümelin | Chadly Marouane | Andreas Butz

2. Agenda user-centered design Goal definition: Identify and interact with objects by pointing First explorational data collection:Let people point Second user study: Develop & test algorithms & gather info on use cases Third user study: Test use cases & collect experiences in the field

3. Motivation Problem with selection of distant objects • Complicated with navigation systems Complex mapping of real objects and virtual representations • AR solutions not hands-free www.bmw.de www.lgworld.com www.bmw.de

4. Motivation Potential of pointing • A common and quick way to identify • an object • or a direction • Less ambiguous than swiping gestures www.bmw.de http://www.wuff.at/cms/uploads/pics/Joachim_05_4c.jpg

5. Related work Bolt (´80): Put that there Althoff et al. (´05): In-car use Mahr et al. (´11): Microgestures Park et al. (´11): Robotics use

6. IDEA Pointing to identify and interact with distant objects

7. 1st (exploring) evaluation Goal:Identify common behavior and first use cases Procedure: Let people point towards different directions

8. 1st (exploring) evaluation Results: searchglance confirmglance confirmglance pointing interaction

9. Pointing recognition Precondition + Approach Remember the specific context of the car: • Restricted movements (only hand, arm, maybe shoulder are moved) • Restricted pointing (windows and cockpit) Recognition approach based on separated head and hand tracking: • Head: based on known characteristics of a head (20 x 15 x25 cm). • Hand: 1) identify all user pixels 2) take the most distant one

10. 2nd (Pointing) evaluation Goal:Evaluate recognition approach, further specify use cases (in a semi-realistic setting), evaluate tolerated pointing durations. Procedure: Let people point towards objects in a projected street scene and on different positions on the dashboard. Split-plot design (2 environments, 3 street scene, 5 POIs each)

11. 2nd (Pointing) evaluation Results • Kinect problems with cluttered foreground • hit rates > 95% (in the lab) • 800 ms tolerable for holding phase • Interior pointing works only for large targets • Usefulness of audio feedback Use outside: get information on POIs, mark street signs/cars/accidents, sky = weather Use inside: storage for rare or later use on passengers‘ side

12. 3rD (Qualitative) evaluation USE CASES 3 || ImmoScouter 3 || ImmoScouter Looking for a new appartment, drive though your favorite district, collect properties of appartments, get offers directly 1 || RestaurantFinder 1 || RestaurantFinder Search for a well rated restaurant, identify and call to make a reservation 2 || SightCollecter 2 || SightCollecter Prepare a tour by selecting interesting sights along your way

13. 3rD (Qualitative) evaluation Driving study Goal:Evaluate pointing in a real driving environment, focus on sujective feedback and effect on driving behavior. Procedure: 1 ½ route with stops for questionnaires. 3 given use cases.Wizard-of-Oz setup.

14. 3rD (Qualitative) evaluation Driving study Results:

15. 3RD (Qualitative) evaluation RESULTS Looking for a POI more distracting than the pointing process. Slowing down to cope with distraction. Seamless pointing gestures preferred to mix of modalities. Pointing as an additional feature for location-based services

16. TAKE AWAY … perceived as natural to interact with distant objects … is easy to understand and unambiguous We didn‘t experience negative effects on driving behavior but found people compensating for it Do: Make use of restrictions in the car! Next: Robust depth tracking & pointing plus speech.

17. Thank you!