WP 4: Context Aware Video Acquisition

1. WP 4: Context Aware Video Acquisition • James L. Crowley • Professeur I.N.P.Grenoble • Project PRIMA, Laboratoire GRAVIR • INRIA Rhône Alpes

2. WP 4: Context Aware Video Acquisition • Plan Situations and Actions for the Lecture Scenario Assembling a Process Federation Version 1.0 of the Intelligent Camera Man Current Work

3. FAME Augmented Meeting Environment • 5 Sony Steerable Cameras • Wide Angle Camera • Microphone Array • 3 Video Interaction Devices: • Vertical • Horizontal • Steerable

4. FAME Augmented Meeting Environment

5. WP 4: Context Aware Video Acquisition • Plan Situations and Actions for the Lecture Scenario • Process Federation for CA Video Acquistion v1.0 • Task, Situation and Context • Compiling the Situation Graph Assembling a Process Federation Version 1.0 of the Intelligent Camera Man Current Work

6. Supervisory Process Event Bus Agent Tracker Agent Tracker Agent Tracker Speech Detection Speech Location Camera 1 Camera 1 Camera 1 Microphone Array Camera Control Federation

7. Task, Situation and Context • Situation: An configuration of roles and relations. • Role: Interpretation of an entity or agent • Relation: A predicate over entities and agents • A task model describes the state space of situations • and the actions of the system for each situation • Approach: Compile a federation of processes to observe the agents and entities that define situations.

8. Task, Situation and Context • Basic Concepts: • Property: Any value observed by a process • Entity: A “correlated” set of properties • Composite entity: A composition of entities • Relation: A predicate defined over entities • Agent: An entity that can act. • Situation: A configuration of roles and relations. • Context: A network of situations

9. Context Aware Video Acquisition • Design Method: 1) Define actions to be taken by system 2) Define situations for each action 3) Define roles and relations 4) Define observation processs 5) Compile situation graph into supervisor rules.

10. Actions to be taken by Context Aware Video Acquisition System v1.0 • Record Shots: • A1 Record wide angle view of the scene • A2 Record the speaker • A3 Record the audience

11. Camera 3 Camera 1 Camera 2 Camera Angles

12. Situations for the Context Aware Video Acquisition System • Situations: S0 empty room Æ A1 S1 Actor enters the room Æ A1 S2 Speaker (actor) speaks Æ A2 S3 Audience (actor) asks a question Æ A3

13. JESS (CLIPS in Java) Events Events Process1 Process 2 Process 3 Data Properties Process Federation Tool • JESS (CLIPS in Java) Environment sends messages to processes

14. Define the roles and relations for Context Aware Video Acquisition System • Roles and relations for camera controller • R1: Agent in audience asks a question • (Agent in audience is speaking) • R2: Lecturer (Agent at lecture-position) • R3: Arriver (Agent at door) • R4: Audience (Agents in audience region) • R5: Speaker (Agent currently speaking)

15. Compiling Situations to Rules XML: <role name="lecturer" arity="1"> <description> The person giving a lecture </description> </role> SituationsPetri NetXML Description->Rules

16. Compiling Situations to Rules CLIPS: (defrule t2EventTransition ?tr <- (transitionTrigger (name "t2") (lock ?l&:(neq ?l 0))) ?pre_S1 <- (situation (name "S1") (entities ?newComer)) (situationPlace (name "S1") (mark ?m_S1&:(neq ?m_S1 0))) (or (not (newComer (isPlayedBy ?newComer))) ) (lecturer (isPlayedBy ?new_lecturer)) (speaker (isPlayedBy ?new_speaker)) (isSameAs (isVerifiedBy ?new_speaker ?new_lecturer)) ?post_S2 <- (situation (name "S2")) => (modify ?tr (lock (- ?l 1))) (assert (event (name "t2"))) (modify ?pre_S1 (entities)) (modify ?post_S2 (entities ?new_lecturer ?new_speaker)) ) SituationsPetri NetXML Description->Rules

17. WP 4: Context Aware Video Acquisition • Plan Situations and Actions for the Lecture Scenario Assembling a Process Federation • Perceptual Processes • Tracking Bodies Hands and Faces • Recognizing and Locating Speech Sounds Version 1.0 of the Intelligent Camera Man Current Work

18. Supervisory Process Event Bus Agent Tracker Agent Tracker Agent Tracker Speech Detection Speech Location Camera 1 Camera 1 Camera 1 Microphone Array Video Acquisition Process Federation

19. Processs to observe agents, entities and relations • P1: Supervisory Controller • P2: Visual Tracking Process for agents with Camera 1 (Wide Angle camera) • P3: Visual Tracking Process for agents with Camera 2 (Audience region) • P4: Visual Tracking Process for agents with Camera 3 (lecturer region) • P5: Speech preprocessing and detection • P6: Speech position estimation.

20. Agent Detection and Tracking Process • Observation Modules: • Color Histogram Ratio • Background Difference • Receptive Field Histograms • Motion History Image

21. Agent Detection and Tracking Process • Process Phases: • While True Do • Acquire next image • Calculate ROI for targets • Verify and update targets • Detect new targets • Regulate module parameters • Interpret entities • Process messages

22. Agent Detection and Tracking • Actors: Composite Entities. • Entity Tracker: Background difference, motion and color • Entity Grouper: Assigns roles to blobs as body, hands, face or eyes

23. Software bus Audio Localisation Audio Router Synchronized audio channels Channel 1 Time Difference Of Arrivals (TDOA) Geometric coordinates evaluation (4 microphones => 3D localisation) Channel 2 Voice activity detection : - energy analysis - speech signal Recognition Channel 3 TCP/IP Client Channel 4 TCP/IP server Speech recognition Channel n • Audio preprocessing : • hardware offset • echo cancellation TCP/IP Client Acoustic Perception Processes

24. supervisory Controller Receive : - configuration messages - commands Send : - position of video targets Software bus Send : - speech activity messages - position of audio targets Receive : - configuration messages - position of video targets Send : - configuration messages Receive : - position of video targets - position of audio localisation targets - Speech activity Send : speech activity messages Receive : configuration messages Audio Router Audio Localisation Context tracker Acoustic Process Federation

25. Multi channel Acoustic Server Process Channel selection Speech Waveform and Spectrogram Remove soundcard offset Usage Adaptive cepstral echo cancellation Final voice activity detection (using energy and neural net results) Temporal energy analysis Voice detection

26. Microphones Room map Current target Software bus status connection status Processing flag Configuration Confidence Acoustic Position Estimation

27. Speech Detection Process

28. WP 4: Context Aware Video Acquisition • Plan Situations and Actions for the Lecture Scenario Assembling a Process Federation Version 1.0 of the Intelligent Camera Man Current Work

29. Context Aware Video Acquisition v1.0

30. Context Aware Video Acquisition v1.0

31. WP 4: Context Aware Video Acquisition • Plan Situations and Actions for the Lecture Scenario Assembling a Process Federation Version 1.0 of the Intelligent Camera Man Current Work • Adding Camera Pan-Tilt Control • Adding New cameras • Estimating Face Orientation • Integration of Topic Spotter

32. Camera Control

33. Face Orientation Estimation

34. WP 4: Context Aware Video Acquisition • Plan Situations and Actions for the Lecture Scenario Assembling a Process Federation Version 1.0 of the Intelligent Camera Man Current Work

35. WP 4: Context Aware Video Acquisition • Plan Situations and Actions for the Lecture Scenario Assembling a Process Federation Version 1.0 of the Intelligent Camera Man Current Work

36. WP 4: Context Aware Video Acquisition • PRIMA Group, Laboratoire GRAVIR (UMR) • INPG (P2), INRIA (P8), UJF (P9), CNRS(P10) • Personnel Contributing during period: • James L. Crowley (Prof. INPG) • Augustin Lux (Prof INPG) • Patrick Reignier (MdC UJF) • Dominique Vaufreydaz (Post Doc INPG) • Alban Caparossi (Engineer UJF) • Stan Borkowski (Doctoral Student, INPG) • Hai Tranh (Doctoral Student, INPG) • Nicolas Gourier (Doctoral Student, INRIA)