1 / 45

Change Detection in Shape Dynamical Models & Application to Activity Recognition

Change Detection in Shape Dynamical Models & Application to Activity Recognition. Namrata Vaswani Dept. of Electrical & Computer Engineering University of Maryland, College Park http://www.cfar.umd.edu/~namrata. Overview. The Group Activity Recognition Problem

afia
Download Presentation

Change Detection in Shape Dynamical Models & Application to Activity Recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Change Detection in Shape Dynamical Models & Application to Activity Recognition Namrata Vaswani Dept. of Electrical & Computer Engineering University of Maryland, College Park http://www.cfar.umd.edu/~namrata Namrata Vaswani, Univ. of Maryland, College Park

  2. Overview • The Group Activity Recognition Problem • Slow and Drastic Change Detection • Landmark Shape Dynamical Models • Experiments and Results Namrata Vaswani, Univ. of Maryland, College Park

  3. The Group Activity Recognition Problem Namrata Vaswani, Univ. of Maryland, College Park

  4. Problem Formulation • The Problem: • Model activities performed by a group of moving and interacting objects (which can be people or vehicles or robots or diff. parts of human body). Use the models for abnormal activity detection andtracking • Our Approach: • Treat objects as point objects: “landmarks”. • Changing configuration of objects: deforming shape • ‘Abnormality’: change from learnt shape dynamics • Related Approaches for Group Activity: • Co-occurrence statistics, Dynamic Bayes Nets Namrata Vaswani, Univ. of Maryland, College Park

  5. Bayesian Approach • Define aStochastic State-Space Model (a continuous state HMM) for shape deformations in a given activity, with shape & motion forming the hidden state vector and configuration of objects forming the observation. • Use a particle filter to track a given observation sequence, i.e. estimate the hidden state given observations. • Define Abnormality as a slow or drastic change in the shape dynamics with unknown change parameters.We propose statistics for slow & drastic change detection. Namrata Vaswani, Univ. of Maryland, College Park

  6. Human Action Tracking Cyan: Observed Green: Ground Truth Red: SSA Blue: NSSA Namrata Vaswani, Univ. of Maryland, College Park

  7. Slow and Drastic Change Detection in Continuous State HMMs Namrata Vaswani, Univ. of Maryland, College Park

  8. The Problem • General Hidden Markov Model (HMM): State seq. {Xt}, Observation seq. {Yt} • Finite duration change in system model which causes a permanent change in probability distribution of state • Change parameters unknown:Log LRT(Xt)  LL(Xt) • Noisy Observations: LL(Xt)  E[LL(Xt)|Y1:t)] = ELL • Nonlinear dynamics: Particle filtered estimate • Slow or drastic change:ELL for slow, OL/TE for drastic Namrata Vaswani, Univ. of Maryland, College Park

  9. Related Work • Change Detection in Nonlinear Systems using PF • Known change parameters, sudden change • Log-LRT of current observation given past observations • Multimode system detect change in mode • Unknown change parameters, sudden change • Generalized LRT • Tracking Error (TE) • Neg. Log likelihood of current observation given past (OL) • Avg. Log Likelihood of i.i.d. observations used often • But ELL =E[LL(Xt)|Y1:t] (MMSE of LL given observations) in context of HMMs is new Namrata Vaswani, Univ. of Maryland, College Park

  10. Particle Filtering Namrata Vaswani, Univ. of Maryland, College Park

  11. Change Detection Statistics • Drastic Change: • Tracking Error (TE). If Gaussian noise, TE ≈ OL • Neg. of Current Observation Likelihood given past (OL) OL = -log [Pr(Yt|Y0:t-1,H0) ] = -log[<Qt pt-1 , ψt>] • Slow Change: Propose Expected Log Likelihood (ELL) • ELL = Kerridge Inaccuracy b/w πt and pt0 ELL(Y1:t )=E[-log pt0(Xt)|Y1:t]=Eπ[-log pt0(Xt)]=K(πt: pt0) • “Detectable changes” using ELL: • E[ELL(Y1:t0)] = K(pt0:pt0)=H(pt0), E[ELL(Y1:tc)]= K(ptc:pt0) • Chebyshev Ineq: With false alarm & miss probabilities of 1/9, ELL detects all changes s.t. K(ptc:pt0) -H(pt0)>3 [√Var{ELL(Y1:tc)} +√Var{ELL(Y1:t0)}] Namrata Vaswani, Univ. of Maryland, College Park

  12. OL & ELL: Slow & Drastic Change • Problem of TE or OL: Fail to detect slow changes • Particle Filter tracks slow changes “correctly” • Assuming change till t-1 was tracked correctly (error in posterior small), OL only uses change introduced at t • ELL works because it uses total change in posterior till time t & since PF tracks the posterior “correctly” for a slow change, ELL is approximated “correctly” • Problem of ELL: Fails to detect drastic changes • Approximating posterior for changed system using a PF for unchanged system: error large for drastic changes • OL relies on the error introduced due to change to detect it, so works for drastic changes • ELL detects change before loss of track, OL/TE: after Namrata Vaswani, Univ. of Maryland, College Park

  13. A Simulated Example Tracking Error (or OL) • Change introduced in system model from t=5 to t=15 ELL Namrata Vaswani, Univ. of Maryland, College Park

  14. Practical Issues • Defining pt0(x): • Either use part of state vector which has linear Gaussian dynamics: can define pt(x), in closed form. • Assume a parametric family for pt(x), learn parameters using training data (assume pt(x) piecewise constant over time) • Declare a change when either ELL or OL/TE exceed their respective thresholds. • Set ELL threshold to H(pt0) +3√Var{ELL(Y1:t0)} • Set OL threshold to a little above E[OL0,0]=H(Yt|Y1:t-1) • Single frame estimates of ELL or OL/TE may be noisy • Average the statistic or average no. of detects or modify CUSUM Namrata Vaswani, Univ. of Maryland, College Park

  15. Approximation Errors • Total error < Bounding error + Model error + PF error • Bounding error: Stability results hold only for bounded fns but LL is unbounded. BE=|ELLtc,c-ELLtc,c,M| • Model error: Error b/w exact filtering with original system model & with changed model, ME=|ELLtc,c,M-ELLtc,0,M| • PF Error: Error b/w exact filtering with changed model & particle filtering with changed model, PE=|ELLtc,0,M-ELLtc,0,M,N| Namrata Vaswani, Univ. of Maryland, College Park

  16. Asymptotic Stability, Stability with t • The error in ELL estimation averaged over observation sequences & PF realizations is asymptotically stable if • Change lasts for a finite time • “Unnormalized filter kernels are uniformly “mixing” • Certain boundedness assumptions hold • Stability & monotonic decrease of error if the kernels are only “mixing” • Analysis generalizes to errors in MMSE estimate of any function of state evaluated using a PF with system model error Namrata Vaswani, Univ. of Maryland, College Park

  17. “Unnormalized filter kernel” “mixing” • “Unnormalized filter kernel”,Rt, is state transition kernel,Qt,weighted by observation likelihood given state • “Mixing”: measures the rate at which the transition kernel “forgets” its initial condition or eqvtly. how quickly the state sequence becomes ergodic. Mathematically, • Example: State transition Xt =Xt-1+nt + a : not mixing. But if Yt=h(Xt)+wt, wtis truncated noise, then Rt is mixing Namrata Vaswani, Univ. of Maryland, College Park

  18. Complementary Behavior of ELL/OL • We have shown thatetc,0=|ELLtc,c- ELLtc,0,N| is upper bounded by an increasing function of OLkc,0, tc<k<t • Implication: • Assume “detectable” change i.e. ELLc,clarge • OL fails => OLkc,0,tc<k<t small => ELL error, etc,0 small=> ELLc,0 large =>ELL detects • ELL fails => ELLc,0 small =>ELL error, etc,0 large => at least one of OLkc,0,tc<k<t large => OL detects Namrata Vaswani, Univ. of Maryland, College Park

  19. “Rate of Change” Bound • The total error in ELL estimation is upper bounded by increasing functions of the “rate of change” (or “system model error per time step”) with all increasing derivatives. • OLc,0 is upper bounded by increasing function of “rate of change”. • Metric for “rate of change” (or equivalently “system model error per time step”) for a given observation Yt: DQ,t is Namrata Vaswani, Univ. of Maryland, College Park

  20. Applications/ Possible Applications • Surveillance: abnormal activity detection • Medical applications: Detect motion disorders by modeling normal human actions using “Shape Activity” models • ELL + “PSSA model” for activity segmentation • Neural signal processing: detecting changes in stimuli • Congestion Detection • System model change detection in target tracking problems without the tracker loses track Namrata Vaswani, Univ. of Maryland, College Park

  21. Landmark Shape Dynamical Models Namrata Vaswani, Univ. of Maryland, College Park

  22. What is Shape? • Shapeis the geometric information that remains when location, scale and rotation effects are filtered out [Kendall] • Shape of k landmarks in 2D • Represent the X & Y coordinates of the k points as a k-dimensional complex vector: Configuration • Translation Normalization: Centered Configuration • Scale Normalization: Pre-shape • Rotation Normalization: Shape Namrata Vaswani, Univ. of Maryland, College Park

  23. Normal Activity Shape space Tangent space (hyper-plane) μ Observation vector: k-dim complex vector representing locations of k point objects at time t Abnormal Activity “Activities” on the Shape Sphere in Ck-1 Namrata Vaswani, Univ. of Maryland, College Park

  24. Related Work • Related Approaches for Group Activity • Co-occurrence Statistics • Dynamic Bayesian Networks • Shape Analysis/Deformation: • PDMs,Thin plate splines, Principal & Partial warps • Active Shape Models: affine deformation in configuration space • ‘Deformotion’: Euclidean motion of avg. shape + deformation • Piecewise geodesic models for tracking on Grassmann manifolds • Particle Filters for Multiple Moving Objects: • JPDAF (Joint Probability Data Association Filter): difficult to define complicated interactions b/w objects Namrata Vaswani, Univ. of Maryland, College Park

  25. Motivation • Obtain a generic & sensor invariant approach for “activities” performed by multiple moving objects. Easy to fuse sensors • Why “shape”: invariant to translation, zoom & in-plane rotation of camera • Single global framework for modeling motion & interactions, co-occurrence statistics requires individual & joint histograms. • New framework to track a group of interacting & moving objects: know that the group is “constrained” to move in a certain fashion defined by the “activity”. • Active Shape Models good for approximately rigid objects (small nonrigidity introduced by camera motion) Namrata Vaswani, Univ. of Maryland, College Park

  26. The HMM • Observation, Yt: Centered configurations • State,Xt=[γt, ct, st, θt ] • Current Shape (γt), • Shape “Velocity” (ct): Tangent coordinate w.r.t. γt • Scale (st), • Rotation angle (θt) • Use complex vector notation to simplify equations • Use a particle filter to approximate the optimal non-linear filter, pt(dx) = Pr(Xtєdx|Y0:t) = posterior state distribution conditioned on observations upto time t, by an N-particle empirical estimate of pt Namrata Vaswani, Univ. of Maryland, College Park

  27. State Dynamics Shape Dynamics: • Define shape “velocity” at time t in the tangent space w.r.t. the current shape, γt • Tangent space is a vector space: define a linear Gaussian-Markov model for shape “speed”, ct • “Move” γtby an amount ct on shape manifold to getγt+1 Motion Dynamics: • Linear Gauss-Markov dynamics for log st, unwrapped θt Namrata Vaswani, Univ. of Maryland, College Park

  28. HMM Equations Observation Model: Map [Shape,Motion]Centered Config. System Model : Shape and Motion Dynamics Shape Dynamics: Motion Dynamics: • Linear Gauss-Markov models for log st and θt • Can be stationary or non-stationary Namrata Vaswani, Univ. of Maryland, College Park

  29. Special Cases • Stationary Shape Activity (SSA):γt = μ, constant • Models shape variation in a single tangent space w.r.t mean shape. • Track normal behavior, detect abnormality • Non-Stationary Shape Activity:γt changes for all t • Tangent space changes at every time instant • Most flexible: Detect abnormality and also track it • Piecewise Stationary Shape Activity:γt p.w. constant • Change time can be fixed or decided on the fly using ELL • PSSA + ELL: Activity Segmentation Namrata Vaswani, Univ. of Maryland, College Park

  30. Stationary Shape Activity • Mean shape is constant, so set γt= μ (Procrustes mean), for all t, γtnot part of state vector, learn mean shape using training data. • Define a single tangent space w.r.t. μ:shape dynamics simplifies to linear Gauss-Markov model in tangent space • Since shape space is not a vector space, “data mean” may not lie in shape space, evaluate “Procrustes mean”: an intrinsic mean on the shape manifold. Namrata Vaswani, Univ. of Maryland, College Park

  31. What is Procustes Mean? • “Proc mean” μ, minimizes sum of squares of “Proc distances” of the set of pre-shapes from itself • “Proc distance” is Euclidean distance between “Proc fit” of one pre-shape onto another • “Proc fit”: scale or rotate a pre-shape to “optimally” align it with another pre-shape • “Optimally” : minimum Euclidean distance between the two pre-shapes after alignment Namrata Vaswani, Univ. of Maryland, College Park

  32. Abnormal Activity Detection • Define abnormal activity as • Slow or drastic change in shape statistics with change parameters unknown. • System is a nonlinear HMM, tracked using a PF • This motivated research on slow & drastic change detection in general HMMs • Tracking Error detects drastic changes. Weproposed a statistic called ELL for slow change. • Use a combination of ELL & Tracking Error and declare change if either exceeds its threshold. Namrata Vaswani, Univ. of Maryland, College Park

  33. Applications/Possible Applications • Modeling group activity to detect suspicious behavior • Airport example • Lane change detection in traffic • Model human actions: track a given sequence of actions, detect abnormal actions (medical application to detect motion disorders) • Activity sequence segmentation: unsupervised training • Sensor independent: IR/Radar/Seismic • Robotics, Medical Image Processing Namrata Vaswani, Univ. of Maryland, College Park

  34. Experiments and Results Namrata Vaswani, Univ. of Maryland, College Park

  35. Experiments • Group Activity: • Normal activity: Group of people deplaning & walking towards airport terminal: used SSA model • Abnormality: A person walks away in an un-allowed direction: distorts the normal shape • Simulated walking speeds of 1,2,4,16,32 pixels per time step (slow to drastic distortion in shape) • Compared detection delays using TE and ELL • Plotted ROC curves to compare performance • Human actions: • Defined NSSA modelfor tracking a figure skater • Abnormality: abnormal motion of one body part • Able todetect as well as track slow abnormality Namrata Vaswani, Univ. of Maryland, College Park

  36. Normal/Abnormal Activity Normal Activity Abnormal Activity Namrata Vaswani, Univ. of Maryland, College Park

  37. Abnormality • Abnormality introduced at t=5 • Observation noise variance = 9 • OL plot very similar to TE plot (both same to first order) Tracking Error (TE) ELL Namrata Vaswani, Univ. of Maryland, College Park

  38. ROC: ELL • Plot of Detection delay against Mean time b/w False Alarms (MTBFA) for varying detection thresholds • Plots for increasing observation noise Drastic Change: ELL Fails Slow Change: ELL Works Namrata Vaswani, Univ. of Maryland, College Park

  39. ROC: Tracking Error(TE) • ELL: Detection delay = 7 for slow change , Detection delay = 60 for drastic • TE: Detection delay = 29 for slow change, Detection delay = 4 for drastic Slow Change: TE Fails Drastic Change: TE Works Namrata Vaswani, Univ. of Maryland, College Park

  40. Slow Change: Works! Drastic Change: Works! ROC: Combined ELL-TE • Plots for observation noise variance = 81 (maximum) • Detection Delay < 8 achieved for all rates of change Namrata Vaswani, Univ. of Maryland, College Park

  41. Human Action Tracking Cyan: Observed Green: Ground Truth Red: SSA Blue: NSSA Namrata Vaswani, Univ. of Maryland, College Park

  42. NSSA Tracks and Detects Abnormality Tracking Error ELL Red: SSA, Blue: NSSA Namrata Vaswani, Univ. of Maryland, College Park Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA Green: Observed, Magenta: SSA, Blue: NSSA

  43. Contributions • Slow and drastic change detection in general HMMs using particle filters. We have shown • Asymptotic stability / stability of errors in ELL approximation • Complementary behavior of ELL & OL for slow & drastic changes • Upper bound on ELL error is an increasing function of “rate” of change, with all increasing derivatives • Stochastic state space models (HMMs) for simultaneously moving and deforming shapes. • Stationary, non-stationary & p.w. stationary cases • Group activity & human actions modeling, detecting abnormality • NSSA for tracking slow abnormality, ELL for detecting it • PSSA + ELL : Apply to activity segmentation Namrata Vaswani, Univ. of Maryland, College Park

  44. Other Contributions • A linear subspace algorithm for pattern classification motivated by PCA • Approximates the optimal Bayes classifier for Gaussian pdfs with unequal covariance matrices. • Useful for “apples from oranges” type problems. • Derived tight upper bound on its classification error probability • Compared performance with Subspace LDA both analytically & experimentally • Applied to object recognition, face recognition under large pose variation, action retrieval. • Fast algorithms for infra-red image compression Namrata Vaswani, Univ. of Maryland, College Park

  45. Ongoing and Future Work • Change Detection • Bound on Errors is increasing fn. of rate of change: Implications • CUSUM algorithm, applications to other problems • Non-Stationary & Piecewise Stationary Shape Activities • Application to sequences of different kinds of actions • PSSA + ELL for activity segmentation • Time varying number of Landmarks? • What is “best” strategy to get a fixed no. ‘k’ of landmarks? • Can we deal with changing dimension of shape space? • Sequence of Activities, Multiple Simultaneous Activities • Multi-Sensor Fusion, 3D Shape, General shape spaces Namrata Vaswani, Univ. of Maryland, College Park

More Related