University of Rochester

1. University of Rochester • Activities • Abductive Inference of Multi-Agent Interaction • Capture the Flag Data Collection • Representing Beliefs & Goals of Multiple Agents • Modal Markov Logic • Recognizing Indoor Activities using Multi-Modal Data • Fusing RFID and Machine Vision

2. Multi-Agent Interaction • Many agent behaviors can only be understood in the context of the actions of other agents • Exercising? • Being chased? • Chasing someone? • Location alone provides a surprisingly rich source of information about behavior • GPS data can be used to learn a probabilistic model of a individual’s common activities (Liao, Fox, & Kautz 2007) • Goal: learn models of groups and interactive activities • Relational learning problem: ideal for ML • Needed: dataset of competitive & cooperative interaction

3. Capture the Flag • Capture the Flag Data Collection • UR campus • Up to 150 x 300 m area • Complex topology • 14 players, 8 games • GPS loggers • Accuracy varies 1-9 m • Average game 12 m

4. Start of Game

5. End of Game red & orange guarded by green green leaves prisoners violet releases red & orange red captures flag

6. Ground Truth • For supervised learning methods, need to create a labeled training set • First attempt: record voice annotations from players • Failed: players too involved to accurately comment on their actions • Second attempt: post-hoc annotation • Created general annotation tool for relations over GPS streams

7. Supervised Weight Learning • Discrete features calculated from GPS streams • Supervised learning applied to simple 2-slice model • Precision: 46% (second by second) • Recall 64% • 12 hours to label 1 hour of training data

8. Observations • Humans can accurately perceive interactive behaviors • High agreement between annotators • GPS noise often obscures geometric details • Reasoning about intention over extended temporal context disambiguates action

9. Year 2 Goals • Improve quality of data (features) using physical constraints • Hard constraints: walls • Soft constraints: paths • CRF “snapping” tool • Model long temporal dependencies • Unsupervised learning: discover behaviors, tactics, strategies

10. Representing Beliefs & Goals of Multiple Agents • Abduction often requires reasoning about the establishment of “propositional attitudes” • Belief, desire, intention, commitment, … • Example: principles of communication: • If A tells B that P, then A believes P. • If A tells B that P, then B will believe that A wants B to believe P. • If A is cooperative with B, and B wants P, then A will want P. • Such principles are defeasible

11. Modal Operators • In logic • Predicates relate one object to another • Modal operators relate objects (agents) to propositions (sentences) • Different modalities can be axiomatically characterized • Deductive closure: • Transitivity:

12. Modal Operators in Markov Logic • ML defines a probability distribution over propositional truth assignments • Idea: define probability distribution over assignments that are modally consistent Non-modal atoms Modal atoms Modal consistency check

13. Inference • Complexity of consistency check • Depends on target modal logic • Belief (KD45): • Unbounded nesting: PSPACE-complete • Bounded nesting: NP-complete • Modal Markov Logic Inference • Rejection sampling • Optimizations • Cache g(M) • Compute g(M) incrementally

14. Year 2 Goals • Implement MML in Alchemy • Applications • Understanding indirect speech acts • Capture the flag • Establishing knowledge by perception • Representing degrees of belief • Functional modal operators