Consistent Visual Information Processing

1. Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University of Technology pinz@emt.tu-graz.ac.at http://www.emt.tu-graz.ac.at/~pinz

2. “Consistency” • Active vision systems / 4D data streams • Multiple visual information • Imprecision • Ambiguity • Contradiction

3. This Talk: Consistency in • Active vision systems: • Active fusion • Active object recognition • Immersive 3D HCI: • Augmented reality • Tracking in VR/AR

4. AR as Testbed Consistent perception in 4D: • Space • Registration • Tracking • Time • Lag-free • Prediction

5. Agenda • Active fusion • Consistency • Applications • Active object recognition • Tracking in VR/AR • Conclusions

6. Active Fusion Simple top level decision-action-fusion loop:

7. Active Fusion (2) • Fusion schemes • Probabilistic • Possibilistic (fuzzy) • Evidence theoretic (Dempster & Shafer)

8. Probabilistic Active Fusion Nmeasurements, sensor inputs: mi Mhypotheses: oj , O = {o1, …, oM } Bayes formula: Use entropy H(O)to measure the quality ofP(O)

9. Probabilistic Active Fusion (2) Flat distribution: P(oj )=const. Hmax Pronounced distribution: P(oc ) = 1; P(oj ) = 0, j  c H = 0 • Measurements can be: • difficult, • expensive, • N can be prohibitively large, … •  Find iterative strategy to minimizeH(O)

10. Probabilistic Active Fusion (3) Start with A  1 measurements: P(oj|m1, … ,mA), HA Iteratively take more measurements: mA+1, … ,mB Until: P(oj|m1, … ,mB), HB  Threshold

11. Summary: Active Fusion • Multiple (visual) information, many sensors, measurements,… • Selection of information sources • Maximize information content / quality • Optimize effort (number / cost of measurements, …) Information gain by entropy reduction

12. Summary: Active Fusion (2) • Active systems (robots, mobile cameras) • Sensor planning • Control • Interaction with the scene • “Passive” systems (video, wearable,…) • Filtering • Selection of sensors / measurements

13. Consistency • Consistency vs. Ambiguity • Unimodal subsets Ok • Representations • Distance measures

14. Consistent Subsets Hypotheses O = {o1 ,…, oM } Ambiguity: P(O) is multimodal Consistent unimodal subsets Ok  O Benefits: • Application domains • Support of hypotheses • Outlier rejection

15. Distance Measures Depend on representations, e.g.: • Pixel-level SSD, correlation, rank • Eigenspace Euclidean • 3D models Euclidean • Feature-based Mahalanobis, … • Symbolic Mutual information • Graphs Subgraph isomorphism

16. Mutual Information Shannon´s measure of mutual information: O = {o1 ,…, oM } A  O, B  O I(A,B) = H(A) + H(B) – H(A,B)

17. Applications • Active object recognition • Videos • Details • Tracking in VR / AR • Landmark definition / acquisition • Real-time tracking

18. Active vision laboratory

19. Active Object Recognition

20. Active Object Recognitionin Parametric Eigenspace • Classifier for a single view • Pose estimation per view • Fusion formalism • View planning formalism • Estimation of object appearance at unexplored viewing positions

28. Applications  Active object recognition • Videos • Details  Control of active vision systems • Tracking in VR / AR • Landmark definition / acquisition • Real-time tracking Selection, combination, evaluation  Constraining of huge spaces

29. Landmark Definition / Acquisition What is a “landmark” ? corners blobs natural landmarks

30. Automatic Landmark Acquisition • Capture a dataset of the scene: • calibrated stereo rig • trajectory (by magnetic tracking) • n stereo pairs • Process this dataset • visually salient landmarks for tracking

31. Automatic Landmark Acquisition visually salient landmarks for tracking • salient points in 2D image • 3D reconstruction • clusters in 3D: • compact, many points • consistent feature descriptions • cluster centers  landmarks

32. Processing Scheme

33. Office Scene

34. Office Scene - Reconstruction

35. Office Scene - Reconstruction

36. Unknown Scene Landmark Acquisition Real-Time Tracking

37. Real-Time Tracking • Measure position and orientation of object(s) • Obtain trajectories of object(s) • Stationary observer – “outside-in” • Vision-based • Moving observer, egomotion – “inside-out” • Hybrid • Degrees of Freedom – DoF • 3 DoF (mobile robot) • 6 DoF (head and device tracking in AR)

38. Outside-in Tracking (1) stereo-rig IR-illumination • wireless • 1 marker/device: • 3 DoF • 2 markers: 5 DoF • 3 markers: 6 DoF devices

39. Outside-inTracking (2)

40. Consistent Tracking (1) • Complexity • Many targets • Exhaustive search vs. Real-time • Occlusion • Redundancy (targets | cameras) • Ambiguity in 3D • Constraints

41. Consistent Tracking (2) • Dynamic interpretation tree • Geometric / spatial consistency • Local constraints • Multiple interpretations can happen • Global consistency is impossible • Temporal consistency • Filtering, prediction

42. Consistent Tracking (3)

45. Hybrid Inside-Out Tracking (1) Inertial Tracker • 3 accelerometers • 3 gyroscopes • signal processing • interface

46. Hybrid Inside-Out Tracking (2) • complementary sensors • fusion

47. Summary: Consistency in • Active vision systems: • Active fusion • Active object recognition • Immersive 3D HCI: • Augmented reality • Tracking in VR/AR

48. Conclusion Consistent processing of visual information can significantly improve the performance of active and real-time vision systems

49. Acknowledgement Thomas Auer, Hermann Borotschnig, Markus Brandner, Harald Ganster, Peter Lang, Lucas Paletta, Manfred Prantl, Miguel Ribo, David Sinclair Christian Doppler Gesellschaft, FFF, FWF, Kplus VRVis, EU TMR Virgo