Multi-Person Multi-Camera Tracking for EasyLiving

1. Multi-Person Multi-Camera Tracking for EasyLiving Brian Meyers Steve Harris John Krumm Vision Technology Research Group Microsoft Research Redmond, WA USA Barry Brumitt Steve Shafer Michael Hale

2. What Is EasyLiving? EasyLiving is a prototype architecture and technologies for building intelligent environments that facilitate the unencumbered interaction of people with other people, with computers, and with devices.

3. Adjust lights as you move around a space • Route video to best display • Move your Windows session as you move • Deliver e-messages to where you are • Monitor a young child or old person Example Behaviors

4. EasyLiving Demo (7 min.)

5. Self-Aware Space EasyLiving must know about people, computers, software, devices, and geometry to work right.

6. Who’s Where?

7. Person-TrackingSystem 5 Triclops stereo cameras 5 PCs running “Stereo Module” (and Microsoft Windows 2000) (only U.S. $319) (includes Internet Explorer) (as part of the OS) (for a limited time only?) 1 PC running “Person Tracker”

8. Triclops Cameras Now superceded by “Digiclops” digital IEEE-1394 version

9. Typical Images Color image from Triclops Disparity image from Triclops

10. Requirements To work in a real-life intelligent environment, our tracking system must … • Maintain location & identity of people • Work with multiple cameras (we’re up to five) • Run at reasonable speeds (we get 3.5 Hz) • Use cameras in the room • Work with multiple people (we handle up to three) • Work for extended periods • Create and delete people instances • Tolerate partial occlusions and variable postures

11. Other Systems • Non-Vision • Olivetti Research (’92) & Xerox PARC (’93) – IR badges • AT&T Laboratories (Cambridge) (’97) – Ultrasonic badges • PinPoint, Ascension, Polhemus – commercial RF badges • Vision (for multiple people) • Haritaoglu & Davis (’98-’99) • Darrell et al. (’98) • Orwell et al. (’99) • Collins et al. (’99) • Rosales & Sclaroff (’99) • Kettnaker & Zabih (’99) • Intille et al. (’95, ’97) • Rehg et al. (’97) • Boult et al. (‘99) • Stiefelhagen et al. (’99) • MacCormick & Blake (’99) • Cai & Aggarwal (’98) • Halevi & Weinshall (’97) • Gavrila & Davis (’96) “I see by the current issue of ‘Lab News,’ Ridgeway, that you’ve been working for the last twenty years on the same problem I’ve been working on for the last twenty years.”

12. Why Use Vision? • Alternative sensors: • Active badges • Pressure-sensitive floors • Motion sensors • Localized sensors, e.g. on door, chair • But … • Cameras are getting cheap • Cameras are easy to install • Cameras give location and identity • Cameras can find other objects, e.g. video screens • Cameras can be use to model room geometry (active badge)

13. Person Detection Steps Camera calibration Background modeling • Background subtraction • Blob clustering • Histogram identification

14. Camera Calibration Paths before calibration Paths after calibration • All tracking done in ground plane • Record path of single person walking around room • Compute (x,y,) that best aligns paths • Requires robust alignment to deal with outliers

15. Background Modeling View of space Combined color & disparity background image

16. Background Subtraction • Foreground if: • valid depth over invalid depth • - OR - • depth difference > Td • - OR - • any (R,G,B) difference > Tc • Color takes over when person sinks into couch cushions • Potential problem when person walks in front of moving video • (thus turn on moving video when acquiring background)

17. Person Detection Region-growing on foreground pixels gives fragmented blobs Group blobs into people-shaped clusters

18. Blob Clustering • Minimum spanning tree • Break really long links • Find five remaining longest links • Break all combinations of these five: • Covariance matrices of 3D coordinates of linked blobs • Eigenvalues of covariance matrices • Compare eigenvalues to person model

19. Color Histograms • Identify people with RGB color histograms, 16x16x16 • Each camera PC maintains its own histograms • Space-variant histograms built as person moves around room • Person tracker uses histogram to resolve ambiguities window Bluish tint window Regular color

20. calibration, background • background subtraction (color & depth) • blob clustering • histogram maintenance So Far

21. Person Tracking • Takes reports from stereo modules • Transforms to common coordinate frame (common coordinate frame)

22. Resolve with color histograms Person Tracking – Steady State One “track” for each person Predicted location Feed back results to stereo modules for histogram updating

23. Person Tracking – Bad Data • Measurement Noise: • Computed position based on predicted position from many reports • Occlusions: • Multiple cameras • Long timeout on unsupported tracks

24. Person Creation Zone • Tracks begin and end here • Initial tracks are provisional • Makes remainder of room more robust

25. Summary • Live demos, 20 minutes long • Person tracker runs at 3.5 Hz • Up to three people in room • People can: • enter • leave • walk around • stop moving • sit • collide

26. Recent Efforts • Stop breaking the vision system! • Moved chairs & changing lights  bad background model • Special behavior, e.g. slow through person creation zone • Lots of people, e.g. around conference table • Find other objects to enable interesting behaviors, e.g. “Where’s that book?” • Easier method to model room geometry

27. Workshop on Multi-Object Tracking