Activity Recognition Journal Club

1. Activity RecognitionJournal Club “Neural Mechanisms for the Recognition of Biological Movements” Martin Giese, Tomaso Poggio (Nature Neuroscience Review, 2003)

2. Objective • Recognition of complex movements and actions using a neurophysiologically plausible and quantitative model • Biology has already generated a system that is robust and has high selectivity - let’s mimic it.

3. Biological Intuitions • Separate dorsal and ventral streams http://upload.wikimedia.org/wikipedia/commons/thumb/c/c0/Gray722.png/300px-Gray722.png http://hebb.mit.edu/courses/8.515/lecture3/sld007.htm

4. Biological Intuitions/Assumptions • Hierarchical architecture with increasing complexity along the hierarchy • Mainly feedforward • Activities and views are learned • Interaction between two streams not necessary for some recognition

5. Biological Intuitions

6. Artificial Neural Networks • Feedforward tr dv/dt = -v + F(W·u) • Recurrent tr dv/dt = -v + F(W·u + M·u)

7. Form Pathway • Object Recognition Riesenhuber, Poggio 2002

8. Form Pathway • Simple Cells • Modeled by Gabor Filters • Output via Linear threshold function • Complex Cells • ‘MAX’ function • Output via linear threshold function • Invariant bar detectors Giese, Poggio, AIM-2002-012, 2002.

9. Form Pathway • View/Object Tuned Units • Radial Basis Function • u is the vector of the responses of the significant invariant bar detector • u0 signifies the preferred input pattern • C is a diagonal matrix with the elements Cu that are inversely proportional to the variance of the l-th component of u in the training set Giese, Poggio, AIM-2002-012, 2002.

10. Form Pathway • Motion pattern neurons • Leaky integrator • ty dy/dt + y(t) = S f(un(t)) • Most active when input follows synaptically encoded sequence

11. Motion Pathway • Represents dorsal stream • Similar hierarchy • Increasing complexity and invariance

12. Motion Pathway • Lowest level • Optical flow, computed directly • Second Level • First Class • Translational flow (four tuned directions) • Second Class • Motion edges (horizontal and vertical) • MAX operator • Third Level • “Snapshot Neurons” • Modelled by RBF’s • Fourth Level • Motion Pattern Neurons

13. Motion Pathway

14. Model Parameters • Simple Cells • 8 orientations, 2 scales • s1=10, s2=7, k=0.35 • Motion Pattern & Motion Snapshot Neurons • ts=150ms Giese, Poggio, AIM-2002-012, 2002.

15. Results

16. Results

17. Results

18. Limitations • Does not address ‘attentional’ effects • No model for computing optical flow • Does not address interaction between form and motion streams • Form stream does not recognize point-light motion as per experimental data

20. Recap

21. Recap

22. Example • Pattern (sn) • a • b • u(:,t+1) = sn(:,t) + w * tanh(u(:,t)); • v(:,t+1) = sum( tanh(u(1:3,t)) );

23. Example