Animal Tracking for Behavior Modeling

1. Animal Tracking for Behavior Modeling James M. Rehg Georgia Tech (standing in for Tucker Balch)

2. People • PI • Tucker Balch • Faculty • Jim Rehg, Computer Science • Collaborator on animal tracking • Aaron Bobick, Computer Science • Collaborator on animal behavior modeling • Bruce Walker, Psychology • Collaborator on aquarium project • Atsushi Nakazawa, Computer Science (Osaka Univ., Japan) • Collaborator on animal tracking • PhD Students • Michael Novitzky • Jin Lee • Matthew Flagg

3. Goals • Track social animals reliably under natural conditions (in vivo) from video • Estimate behavioral parameters from tracking data • Construct executable models of social animal behavior • Develop a biologically-inspired protocol for dynamic team formation

4. Year 1 Overview • Goal • Develop reliable multi-target tracking algorithms for animals in video • Approach • Jointly estimate segmentation and motion of a nonrigid, deformable target • Key requirements • Reliably estimate target shape over time, to support behavioral analysis • Minimize the amount of human effort required

5. Challenges • Accurate segmentation of target • Set-point tracking is not sufficient • Track reliably with significant camera motion • Background subtraction is not sufficient • Handle a wide range of animals • Scalable solution for the animal kingdom  • Track multiple interacting targets • Multiple instances of target type (e.g. wolf pack) • Occlusions (with self, other targets, and background) • Track long sequences, track across cuts

6. Approach • Basic research in video object segmentation and tracking • Modular software architecture • Easily change features, models, and solver • Testing in two scenarios • Aquarium Monitoring • Long sequences, static camera, some modeling effort • Tracking Animal Planet videos • Shorter sequences, moving camera, minimal human effort • Preliminary application • Accessible Aquarium Project

7. Tracking App • Basic Trackers: • Contour feature w/ Iterated Closest Point • Color feature w/ Mean Shift

8. Color Histogram Tracker • Fish Model: • Appearance model • For each species of fish, multiple HSV Color Histograms on Image patches • Off-line Model Selection by Human • Detection and Tracking: • Histogram-based mean shift approach • Maximization of Bhattacharyya Coefficient between Observation and Model • Selection of a model with the highest coefficient and update track

9. Histogram-based Model Selection 0.6 0.5 0.7 * 0.8 0.3 Input Image Model Image Patches (to build histogram) Best Model Image Patch after Mean-shifting Similarity between shifted & model patches

10. Accessible Aquarium Project • Provide a meaningful and informative aquarium experience for visually-impaired or blind visitors • Approach • Track the movement of individual fish within tank • Sonify the fish movement • Example: 65 gallon marine aquarium • Track yellow tangs and blue chromis • Music structure is Bach chorale • Each fish type is same instrument, different registers • Movement speed mapped to note density (tempo) • Horizontal dimension is stereo, vertical is timbre

11. Example • Play video

12. Tracking in the Video Volume

13. Graphcut Tracking

14. Tracking Results

15. Social Game Retrieval

16. Overview

17. Summary of Progress • Robust long-life tracking of multiple targets (fish) • Tracking under controlled (but realistic) conditions • Virtual Aquarium Project • Novel experience of animal behavior via sonification • New state-of-art motion segmentation algorithm • Accurate segmentation of (shorter) video sequences under a wide range conditions

18. Year 2 Plans • Make general tracking method work for longer sequences • Develop integrated tracking application tool for biologists • Connect to biologists more effectively (what to say about this?) • Start developing behavior models, how?