My PhD Project

1. MyPhD Project Quantifying Visual Cues of Psychological States in Interview Situations (QVCPS) 18/05/2005 Enrica Dente Email: enrica.dente@imperial.ac.uk Website: http://www.enricadente.com/imperial/

2. Contents • Problem • Hypothesis • Project Objectives • Project Plan • Achievements to Date • Conclusion Enrica Dente, QVCPS Project

3. Problem • Most lie detection approaches can only detect “psychological states”, not “deception” (i.e. are less than 80% accuracy) [Vrij, 2004] • Manual observation of human behavior subjective and limited [Burgoon, 2005] • Polygraph, 100 years old lie detector problematic but still in use • Increased interest in alternative approaches: • Automated Face Analysis (i.e. changes in facial expressions) • Thermal imaging (i.e. changes in blood flow around eyes) • FMRI (i.e. BOLD) • Brain fingerprinting (i.e. brain waves) • Eye tracking (i.e. irregular eye movements in front of images ) • Intrusiveness yet to be addressed. Enrica Dente, QVCPS Project

4. Hypothesis • Non skilled liars more likely to experience cognitive load and controlled behavior[Vrij, 2004] • Decrease of hand/finger movements when subjects lie yet to be proved (or disproved) • Quantification of observations required • Focus on tracking of hand/finger movements in interview situations • Real life cases can make a difference in results but problematic [Vrij 2005] • Verbal behavior required to add context to the tracking Enrica Dente, QVCPS Project

5. Project Objectives • To automate the tracking of finger/hand movements in interview situations • Tocompute the frequency of hand and finger movements normalised to the baseline of each individual • To build a biomechanical model to aid hand gesture tracking • To provide a simple means of accessing and referring to the verbal behaviour, when required • Later, to compare the movements of the interviewee with the movements of the interviewer. Enrica Dente, QVCPS Project

6. Project Plan • Year 1: • Skin Colour Modelling • Hand Tracking and Finger Position • Coding behaviour markup • Background Modelling • Graphical User Interface • Experimental Protocol • Year 2 and Year 3: • Kinematic models of upper body, limb and finger motion • Introduction of verbal behavior to add context to tracking • Train system to quantify different types of hand and finger movements • Writing up. Enrica Dente, QVCPS Project

7. Achievements to Date: 1) Segmentation (a) Skin Colour Modelling • Color Predicate[Kieldsen, 1996] • Bayesian Posterior Map and Parzen colour space probability density estimates more accurate. (b) Hand Tracking and Finger Position • Connected Component and central moments • Centroid correspondence • Hands separation based on area and distance heuristics • Complex wavelet decomposition to detect changes in finger position. • Next Step: background modelling using parametric methods to address robustness. Figure 1 -Hand Tracking Figure 2 -Finger Orientations Enrica Dente, QVCPS Project

8. (a) Skin Colour Modelling • Joint Conditional Density function in hue and sat for hand and non-hands (i.e. Parzen density estimation) where H and S represent hue and sat, Hn and Sn represent the observed values amongst the NH hands training set pixels, KH is a normalised constant and σ0 species the width of the kernel used in smoothing • Bayesian Posterior Map to identify most probable locations of skin in each frame where we assume that p(H,S | Hands) is approximated by the left hand side of Equation (1) and p(H,S | NonHands) is approximated by the left hand side of Equation (2). (1) (2) Enrica Dente, QVCPS Project

9. Figure 3 - Likelihood function for hand regions in a), for non-hand regions in b) (a) Skin Colour Modelling (cont.) • Parzen colour space probability density estimates for hand and for non-hand regions • There is overlap between the two classes Enrica Dente, QVCPS Project

10. (b) Hand Tracking and Finger Position HAND TRACKING: • Centroid correspondence of each object across frames from binary image created in (a) • Connected component analysis to label blobs. • Switch from a one-hand to two-hand state by using distance and area heuristics. FINGER POSITION: • Complex wavelet decomposition • Orientation of local image structure relatively invariant to local phase • Filter bank of complex wavelets, each tuned to one of four directions • We compare output of four complex masks to make estimate of the presence of a line or an edge in each direction • Vector indicating the direction of local image structure estimated by a weighted vector summation operation • Multirate scheme (i.e. series of different scales). Enrica Dente, QVCPS Project

11. Finger Position Estimate • K/2 of the real and imaginary impulse responses of wavelets at scale 1 for K = 8 • Orientation field O(l) (m, n) is defined by: Figure 4 -WaveletKernels Enrica Dente, QVCPS Project

12. Figure 5 (a) Hands together (b) Hands moving Finger Position Estimate (.cont) • Histogram of orientations for each hand in each frame • Weighting of the product of the orientation • Field magnitude and posterior skin tone map to appropriate histogram bin • Orientation density functions can be used to detect finger movement! Enrica Dente, QVCPS Project

13. Achievements to Date:2)XML Markup for Behavior Coding Check Verbal behavior: Check Non Verbal behavior: Enrica Dente, QVCPS Project

14. Achievements to Date:3) Graphical User Interface (GUI) Enrica Dente, QVCPS Project

15. Achievements to Date:4) Experimental Protocol • Stakes low in current lie detection experiments: • Interviewees are asked to lie on simulated actions • Interviewees less motivated to lie than in real-life situations, performance less effective • Vrij’s real-life police interview limited to visual observation of one liar • Problem: questions based on prior knowledge of what the interviewees will lie on and when • Our experimental protocol aims to raise the stakes of deception detection in interview situations: • Solution: design experiments where interviewees are not told in advance what they have to lie on. Enrica Dente, QVCPS Project

16. Literature Review1) Skin Colour Modelling • Non Parametric Methods • Create an histogram or lookup table (LUT) common and simple • However, storage required, bins may miss pixel values, noise removal and occlusion still an issue • Bayesian posterior map computed from “hands” and non “hands” histograms more promising than Colour Predicate • Parametric Methods • Model skin colour as components of mixtures of Gaussians • Provide robustness, no need for storage space and ability to interpolate or generalise the training data • However, they require accurate initialization and assume number of components to be known in advance Enrica Dente, QVCPS Project

17. Literature Review1) Skin Colour Modelling • Parametric Methods (cont.) • Foreground and background modelled by a joint probability density function • Model parameters (i.e. means and covariance) for the Gaussians estimated from the training data using maximum likelihood or Bayesian inference • Updated online based on colour, on position and/or on motion information • Skin probability computed from Gaussian Probability Density Functions (PDF's) • However, in interview situations update cannot be based on assumption that foreground is moving. Enrica Dente, QVCPS Project

18. Literature Review2) Kinematic 3-D models • Need accurate tracking of hand/finger gestures • Choice of degrees of freedom crucial • Choice of type of model (i.e. stick figure or statistical) • Need markers for validation • Need simple and consistent movements • Does deception occur fast or slow in interviews? • How can we track several movements at the same time? Number of cameras? • How visible can hand's edges be in interviews? Camera resolution? Enrica Dente, QVCPS Project

19. Conclusion • Problem and Hypothesis: • Most lie detection approaches detect “psychological states”, not “deception” • Non skilled liars more likely to experience cognitive load and controlled behavior • Quantification of observations required • Achievements to Date: • Hand Tracking using Bayesian Posterior Map • Finger Position using Complex Wavelets • XML markup for correlating visual and verbal behavior • Experimental Protocol • Next Step: • Background modelling to address robustness. Enrica Dente, QVCPS Project

20. Questions? Enrica Dente, QVCPS Project

21. References • [Vrij, 2004] Vrij Aldert, Psychology, Crime and Law, Challenging Interviewees During Interviews, The Potential Effects on Lie Detection, 2004 • [Vrij2, 2004] Vrij Aldert, Why Professionals fail to catch liars and how they can improve, Legal and Criminological Psychology, 9, 159-181, 2004 • [Vrij, 2002] Aldert Vrj, Detecting Lies and Deceit : The Psychology of Lying and the Implications for Professional Practice, John Wiley & Sons, Inc • [Watson, 1081] Watson, K. W. Oral and written linguistic indices of deception during employment interviews. (Doctoral dissertation, Louisiana State University, 1981). Dissertations Abstracts International, 42, 06A 2367. • [Parker et al., 2000] Parker, A. & Brown, J., Detection of deception: Statement Validity Analysis as a means of determining truthfulness or falsity of rape allegations. Legal and Criminological Psychology, 5, 237-259, 2000 Enrica Dente, QVCPS Project

22. References (cont.) • [Bradley, 1993] Bradley MT, Cullen MC, Polygraph lie detection on real events in a laboratory setting, 76(3 Pt 1):1051-8, Percept Mot Skills. June 1993 • [Etcoff, 2000] Nancy L. Etcoff*, Paul Ekman, John J. Magee, Mark G. Frank, NATURE , VOL 405, 11 MAY 2000, Lie detection and language comprehension, 2000 Macmillan Magazines Ltd, Available Online: http://www.nature.com/cgi-taf/DynaPage.taf?file=/nature/journal/v405/n6783/full/405139a0_fs.html&content_filetype=pdf • [Kieldsen, 1996] R. Kjeldsen and J. Kender. Finding skin in color images. In Second International Conference on Automatic Face and Gesture Recognition, 1996. • [Burgoon, 2005] J. Burgoon et al. An approach for intent identi- • cation by building on deception detection. In Proc. of the 38th Annual Hawaii Int. Conf. on Detection of Deception: Collaboration Systems and Technology, 2005. • [Bharath, 2003] A. Bharath and J. Ng. A Steerable Complex Wavelet Construction and Its Application to Image Denoising. IEEE Transactions on Image Processing. June 2003 Enrica Dente, QVCPS Project