Sensor Fusion Using Dempster-Shafer Theory

1. Sensor Fusion Using Dempster-Shafer Theory Huadong Wu, Mel Siegel The Robotics Institute, Carnegie Mellon University Rainer Stiefelhagen Interactive System Labs, University of Karlsruhe Jie Yang Interactive System Labs, Carnegie Mellon University IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

2. agenda • describe sensor fusion requirement for context-aware computing applications • describe application case study: meeting participants’ focus-of-attention analysis • review alternative sensor fusion approaches • introduce Dempster-Shafer theory of evidence • develop “weighted D-S evidence combination” • demonstrate its effectiveness in the case study • consider extensions of scale and scope IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

3. outline • context aware computing • sensor fusion • system architecture • case: focus-of-attention sensor fusion • Dempster-Shafer approach • implementation details • experiments & analysis • conclusions and future work IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

4. context aware computing • best algorithm for human-computer interaction tasks depends on context • context can be difficult to discern • multiple sensors give complementary (and sometime contradictory) clues • sensor fusion techniques needed • (but best algorithm for sensor fusion tasks may depend on context!) IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

5. sensor fusion • how to combine outputs of multiple sensor perspectives on an observable? • modalities may be “complementary”,“competitive”, or “cooperative” • technologies may demand registration • variety of historical approaches, e.g.: • statistical (error and confidence measures) • voting • Bayesian (probability inference) • neural network, fuzzy logic, etc IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

6. generalizable SF architecture • for context-aware computing applications • cartoon (next) illustrates typical configuration • low level sensor fusion done locally • sensor clusters deliver local perceptions • implemented via LAN communication and shared global database technologies IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

7. user context database sensor Internet smart sensor node sensor gateway smart sensor node sensors appliance applications sensor fusion embedded OS Intranet appliance embedded OS sensor applications lower-level sensor fusion context server higher-level sensor fusion appliance smart sensor node database sever embedded OS site context database sensors system architecture to support sensor fusion for context-aware computing IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

8. dynamic database • example: user identification and tracking • tables (next) list basic information about environment (room) and parameters, e.g., • temperature, noise, lighting, available devices, number of people, segmentation of area, etc • detail: analyze focus-of-attention of each user in a meeting at a small conference table IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

9. context information architecture:dynamic context information database IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

10. system architecture • create a “sensor fusion mediator” for each context • mediator collects and processes sensor data • in this test, sensor fusion implemented using Dempster-Shafer theory of evidence • need to develop differential weighting scheme for sensors in which we have differing confidence • allows combination of individual sensors’ observation with specified confidence level • admits measures of ignorance as well as knowledge IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

11. Other AI algorithms Other AI algorithms Interpreter Interpreter application application context data context data Resource Registry Resource Registry AI algorithms Dempster-Shafer rule Dynamic Context Database Discoverer Widget Widget Widget Widget sensor sensor sensor sensor SF mediator SF mediator user - mobile site context site context Aggregator Aggregator computer server database server system configuration IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

12. focus of attention • recorded meeting activities of four people around a conference table • “whose focus-of-attention is on whom” ground-truth found manually at 10 Hz • microphones detect who is talking • video imagery analyzed for each talker/listener pair IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

13. Perspective View Camera View focus of attention analysis: equipment Panoramic View IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

14. video image analysis • neural network estimates head poses • focus of attention estimate based on head pose probability distribution analysis • audio reports speaker, assumed to be focus of other participants’ attention • situation is not easy to analyze due to, e.g., dependence of behavior on discussion topic • suggests we need more general fusion approach than provided by Bayesian IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

15. focus of attention:estimation from video and audio sensors IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

16. comparison of sensor fusion alternatives #1. complementary #3. cooperative Parametric template, Figures of merit, Syntactic pattern recognition … … Logical template AI rule-based reasoning, Heuristic inference Neural network … … #2. competitive IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

17. add Dempster-Shafer approach • generalization of Bayesian approach • “theory of evidence” • implement via quantitative definitions of “belief” and “plausibility” IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

18. sensor fusion with Dempster-Shafer theory of evidence algorithm • Frame of discernment Θ: { {L}, {O}, {R}, {L | O}, {O | R}, {L | R}, {L | O | R} } • Updated Belief: IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

19. we add: weighted Dempster-Shafer evidence combination rule IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

20. weight == prior probability IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

21. person validframes audiocorrect videocorrect linearsum DScorrect weightedDS Experiment Set2 #0 1229 55.6% 70.5% 70.1% 70.0% 71.4% #1 1075 61.5% 66.2% 69.8% 70.0% 69.4% #2 1098 66.1% 78.3% 80.2% 80.8% 80.2% #3 991 68.8% 60.3% 65.6% 66.6% 70.0% Experiment Set5 #0 768 73.8% 74.4% 76.8% 77.0% 77.0% #1 956 67.6% 68.5% 72.0% 72.3% 72.1% #2 1006 73.2% 83.0% 84.1% 84.2% 83.9% #3 929 53.3% 67.9% 75.7% 76.9% 73.2% Experiment Set6 #0 799 59.1% 70.6% 71.2% 71.5% 71.0% #1 751 63.3% 84.6% 85.5% 85.8% 85.2% #2 827 57.4% 82.2% 83.3% 84.3% 83.4% #3 851 60.8% 80.6% 81.9% 82.3% 81.7% Experiment Aufnahme2 #0 653 73.2% 84.2% 85.0% 85.0% 84.2% #1 653 57.3% 53.5% 54.2% 54.2% 54.5% #2 681 72.7% 65.6% 69.5% 69.3% 70.3% #6 435 85.3% 75.6% 78.2% 78.4% 79.8% summary 13702 64.6% 72.8% 75.8% 75.4% 75.4% Sensor Fusion Results of Focus-Of-Attention Experiments IMTC-2002-1076 Dempster-Shafer mws@cmu.edu

22. conclusions and future research • Dempster-Shafer evidence combination rule and weighted Dempster-Shafer evidence combination rule are generalized forms of the classic Bayesian inference method • Dempster-Shafer method, especially the weighted Dempster-Shafer method, is suitable for sensor fusion tasks in context-sensing architectures with highly dynamic sensor configurations • we expect the weighted Dempster-Shafer evidence combination rule will outperform linear summation and other sensor fusion methods in planned research that will involve additional sensors IMTC-2002-1076 Dempster-Shafer mws@cmu.edu