PRIMA P erception R ecognition and I ntegration for Observing and M odeling A ctivity

1. PRIMAPerception Recognition and Integration for Observing and Modeling Activity James L. Crowley, Prof. I.N.P. Grenoble Augustin Lux, Prof. I.N.P. Grenoble Patrick Reignier, MdC. Univ. Joseph Fourier Dominique Vaufreydaz, MdC UPMF

2. The PRIMA Group Leaders Doms, Jim, Patrick and Augustin

3. The PRIMA Group Members Trombinoscope

4. The PRIMA Group, May 2006 • Permanents : • James L. Crowley, Prof. I.N.P. Grenoble • Augustin Lux, Prof. I.N.P. Grenoble • Patrick Reignier, MdC. U.J.F. • Dominique Vaufreydaz, MdC. UPMF. • Assistante : • Caroline Ouari (INPG) • Contractual Engineers • Alba Ferrer, IE INRIA • Mathieu Langet, IE INPG

5. The PRIMA Group, May 2006 • Doctoral Students : • Stan Borkowski (Bourse EGIDE) • Chunwiphat, Suphot (Bourse Thailand) • Thi-Thanh-Hai Tran (Bourse EGIDE) • Matthieu Anne (Bourse CIFRE - France Telecom) • Olivier Bertrand (Bourse ENS Cachan) • Nicolas Gourier (Bourse INRIA) • Julien Letessier (Bourse INRIA) • Sonia Zaidenberg (Bourse CNRS - BDI) • Oliver Brdiczka (Bourse INRIA) • Remi Emonet (Bourse MENSR)

6. Plan for the Review • 1) Presentation of Scientific Project • Objectives • Research Problems and Results • Bilan 2003 - 2006 • Evolutions for 2007-2010

7. Objective of Project PRIMA • Develop the scientific and technological foundations for context aware, interactive environments • Interactive Environment: • An environment capable of perceiving, acting, communicating, and interacting with users.

8. Experimental Platforme : FAMEAugmented Meeting Environment • 8 Cameras • 7 Steerable • 1 fixed, wide angle • 8 Microphones • (acoustic Sensors) • 6 Biprocessors (3 Ghz) • 3 Video Interaction Devices • (Camera-projector pairs) January 06: Inauguration of new Smart Environments Lab (J 104)

9. Augmented Meeting Environment

10. Research Problems • Context-aware interactive environments • New forms of man-machine interaction (using perception) • Real Time, View Invariant, Computer Vision • Autonomic Architectures for Multi-Modal Perception

11. Research Problems • Context-aware interactive environments • New forms of man-machine interaction (using perception) • Real Time, View Invariant, Computer Vision • Autonomic Architectures for Multi-Modal Perception

12. User Services Situation Modeling Perceptual Components Ontology Server, Utilities Logical Sensors, Logical Actuators Sensors, Actuators, Communications Software Architecture for Observing Activity • Sensors and actuators: Interface to the physical world. • Perception and action: Perceives entities, Assigns entities to roles. • Situation: Filter events, Describes relevant actors and props for services. • (User) Services: Implicit or explicit. Event driven.

13. Situation Graph Situation-3 Situation-5 Situation-1 Situation-6 Situation-2 Situation-4 Situation Graph • Situation: An configuration of entities playing roles • Configuration: Set of Relations (Predicates) over entities. • Entity: Actors or Objects • Roles: Abstract descriptions of Persons or objects • A situation graph describes a state space of situations • and the actions of the system for each situation

14. Situation and Context • Basic Concepts: • Property: Any value observed by a process • Entity: A “correlated” set of properties • Composite entity: A composition of entities • Relation: A predicate defined over entities • Actor: An entity that can act. • Role: Interpretation assigned to an entity or actor • Situation: A configuration of roles and relations.

15. Situation and Context • Role: Interpretation assigned to an entity or actor • Relation: A predicate over entities and actors • Situation: An configuration of roles and relations. • A situation graph describes the state space of situations • and the actions of the system for each situation • Approach: Compile a federation of processes to observe the roles (actors and entities) and relations that define situations.

16. Acquiring Situation Models • Objective: • Automatic acquisition of situation models. • Approach: • Start with simple sterotypical model for scenario • Develop using Supervised Incremental Learning • Recognition: • Detect Roles with Linear Classifiers • Recognize Situation using probablisitic model

17. Camera Camera Video Acquisition System V2.0 Process Supervisor Situation Modeling Event Bus Face Detection Audience Camera Audio-Visual Composition Streaming Video MPEG Speaker Tracker Steerable Camera 1 Vocal Activity Detector New Slide Detection New Person Detection M i c M i c Projector Wide Angle Camera

18. Audio-Visual Acquisition System Version 1.0 - January 2004

19. Research Problems • Context-aware interactive environments • New forms of man-machine interaction (using perception) • Real Time, View Invariant, Computer Vision • Autonomic Architectures for Multi-Modal Perception

20. Steerable Camera Projector Pair

22. Portable Display Surface

23. Rectification by Homography • For each rectified pixel (x,y), project to original pixel and compute interpolated intensity

24. Real Time Rectification for the PDS

25. Luminance-based button widget S. Borkowski, J. Letessier, and J. L. Crowley. Spatial Control of Interactive Surfaces in an Augmented Environment. In Proceedings of the EHCI’04. Springer, 2004.

26. y Gain x Striplet – the occlusion detector x

27. Striplet – the occlusion detector x y

28. Striplet – the occlusion detector x y

29. Striplet-based SPOD SPOD – Simple-Pattern Occlusion Detector

30. Projected Calculator

31. Research Problems • Context-aware interactive environments • New forms of man-machine interaction (using perception) • Real Time, View Invariant Computer Vision • Autonomic Architectures for Multi-Modal Perception

32. Chromatic Gaussian Basis Normalized in Scale and Orientation to Local Neighborhood

33. Real Time, View Invariant Computer Vision • Results • Scale and orientation normalised Receptive Fields computed at video rate. (BrandDetect system, IST CAVIAR) • Real time indexing and recognition (Thesis F. Pelisson) • Robust Visual Features for Face Detection • (Thesis N. Gourier) • Direct Computation of Time to Crash • (Masters A. Negre) • Natural Interest "Ridges" • (Thesis Hai Tranh)

34. Scale and Orientation Normalised Gaussian RF's Intrinisic Scale: Peak in Laplacian as a function of Scale. • Oriented Response can be obtained as a weighted sum of cardinal derivatives • <A(i,j) G()> = <A(i,j) Gx()> Cos() + <A(i,j) Gy()> Sin() • Normalisation of scale and orientation provides invariance to distance and • camera rotation.

35. Natural Interest Points(Scale Invariant "Salient" image features) • Local extrema of < 2G(i,j,)•A(i,j)> • over i, j,  • Problems with Points • Elongated shapes • Lack of discrimination power • No orientation information • Proposal: Natural Interest Ridges • Maximal ridges in Laplacian Scale Space:

36. Natural Ridge Detection [Tran04] • Compute Derivatives at different Scales. • For each point (x,y,scale) • Compute second derivatives: fxx,fyy,fxy • Compute eigenvalues and eigenvectors of Hessian matrix • Detect local extremum in the direction corresponding to the largest eigenvalue. • Assemble Ridge points, Laplacian Hessian

38. Real Time, View Invariant Computer Vision • Current activity • Robust Visual Features for Face Detection • Direct Computation of Time to Crash • Natural Interest "Ridges" for perceptual organisation.

39. Research Problems • Context-aware interactive environments • New forms of man-machine interaction (using perception) • Real Time, View Invariant, Computer Vision • Autonomic Architectures for Multi-Modal Perception

40. Supervised Perceptual Process • Supervisor Provides: • Execution Scheduler • Command Interpreter • Parameter Regulator •Description of State and Capabilities

41. Detection and Tracking of Entities • Entities: Correlated sets of blobs • Blob Detectors: Backgrnd difference, motion,color, receptive fields histograms • Entity Grouper: Assigns roles to blobs as body, hands, face or eyes

42. Autonomic Properties provided by process supervisor • Auto-regulatory: The process controller can adapt parameters to maintain a desired process state. • Auto-descriptive: The process controller provides descriptions of the capabilities and the current state of the process. • Auto-critical: Process estimates confidence for all properties and events. • Self Monitoring: Maintaining a description of process state and quality of service

43. Self-monitoring Perceptual Process • Process monitors likelihood of output • When an performance degrades, process adapts processing (modules, parameters, and data) Error Classification Error Recovery Process Model Error? Model Learning Perceptual Process Video Unknown Errors

44. Autonomic Parameter Regulation • Parameter regulation provides robust adaptation to • Changes in operating conditions. Parameter Regulator System Parameters Operator Pixel-level Detection Entities Recognition Video Stream Tracked Entities Entity Database Operator Input Training

45. Research Contracts (2003-2006) • National and Industrial: • ROBEA HR+ : Human-Robot Interaction (with LAAS and ICP) • ROBEA ParkNav: Perception and action dynamic environments • RNTL ContAct: Context Aware Perception (with XRCE) • Contract HARP (Context aware Services - France Telecom) • IST - FP VI: • Projet IST IP - CHIL : Multi-modal perception for Meeting Services • IST - FP V: • Project IST - CAVIAR: Context Aware Vision for Surveillance • Project IST - FAME: Multi-modal perception for services • Project IST - DETECT : Publicity Detection in Broadcast Video • Project FET - DC GLOSS : Global Smart Spaces • Thematic Network: FGNet (« Face and Gesture ») • Thematic Network: ECVision- Cognitive Vision

46. Collaborations • INRIA Projects • EMOTION (INRIA RA): Vision for Autonomous Robots; ParkNav, ROBEA (CNRS), Theses of C. Braillon and A. Negre • ORION (Sophia): Cognitive Vision (ECVision), Modeling Human Activity • Academic: • IIHM, Laboratoire CLIPS: Human-Computer Interaction, Smart Spaces; Mapping Project, IST Projects GLOSS, FAME, Thesis: J. Letissier • Univ. of Karlsruhe (Multimodal interaction): IST FAME and CHIL. • Industry • France Telecom: (Lannion and Meylan) Project HARP, Thesis of M. Anne. • Xerox Research Centre Europe: Project RNTL/Proact Cont'Act • IBM Research (Prague,New York): Situation Modeling, Autonomic Software Archictures, Projet CHIL

47. Knowledge Dissemination

48. Conferences and Workshops Organised • General Chairman (or co-chairman) • Conference: SoC-EuSAI 2005 • Workshops: Pointing 2004, PETS 2004, Harem 2005 • Program Co-Chairman • International Conference on Vision Systems, ICVS 2003, • European Symposium on Ambient Intelligence, EuSAI 2004, • International Conference on Multimodal Interaction, ICMI 2005. • Program Committee/Reviewer: UBICOMP 2003, ScaleSpace 2003, sOc 03, ICIP 03, ICCV 03 AMFG 04, ICMI 03, RFIA 2004, IAS 2004, ECCV 2004,FG 2004, ICPR 2004, CVPR 2004, ICMI 2004, EUSAI 2004, CVPR 2005, ICRA 2005, IROS 2005, Interact 2005, ICCV05, ICVS 06, PETS 05, FG06, ECCV06, CVPR06, ICPR06, IROS06…

49. APP Registered Software • 1) CAR : Robust Real-Time Detection and Tracking • APP IDDN.FR.001.350009.000.R.P.2002.0000.00000 • Commercial License to BlueEyeVideo • 2) BrandDetect: Detection, tracking and recognition of commercial • trademarks in broadcast video • APP IDDN.FR.450046.000.S.P.2003.000.21000 • Commercial License to HSArt • 3) ImaLab: Vision Software Development Tool. • Shareware, APP under preparation. • Distributed to 11 Research Laboratories in 7 EU Countries • 4) Robust Tracker v3.3 (stand-alone) • 5) Robust Tracker v3.4 (Autonomic) • 6) Apte: Monitoring, regulation and repair of perceptual systems. • 7) O3MiCID: Middleware for Intelligent Environments

50. Start-up: Blue Eye Video PDG: Pierre de la Salle Marketing : Jean Viscomte Engineers : Stephane Richetto Pierre-Jean Riviere Fabien Pelisson Dominique de Mont (HP) Sebastien Pesnel Councelor : James L. Crowley Incubation: INRIA Transfer, GRAIN, UJF Industrie. Region Rhône Alpes Lauréat de concours création d'enterprise Creation : 1 June 2003 Market: Observation of human activity Sectors: Commercial services, Security, and traffic monitoring Status: 386 K Euros in sales in 2005, >100 Systems installed