ENVIRONMENTAL AWARENESS

1. ENVIRONMENTAL AWARENESS Mahaneeya Raman Mahaneeya Raman - Extended Class Project

2. Automated Activity and Object Detection from Soldier-worn sensors • To monitor and assist soldiers • Monitor environment and soldier entity • Data acquisition through: • Omni-vision Camera • Body sensors • GPS/GIS Mahaneeya Raman - Extended Class Project

3. Agenda • Brief overview of Omni-vision Camera System • Brief overview of body sensors • Overview of GPS/GIS • Image Stitching –procedure and problems encountered • BodyMedia SenseWear sensor system Mahaneeya Raman - Extended Class Project

4. Omni-vision Camera System • Wide angle lenses (f= 1.7 mm ~ 110 degrees) • Higher image resolution than mirror cameras and cheaper ones. • Cameras daisy chained. • Just 1 Fire wire cable (signal +power) Mahaneeya Raman - Extended Class Project

5. Body sensors – to predict physical activity and emotion • Physiological data related to human activity and emotion • Physical activity – accelerometer signals, temperature, heat flux • Human emotion – Skin conductance (Galvanic Skin Response), heart rate Mahaneeya Raman - Extended Class Project

6. GPS/GIS • Map data to physical coordinates • Simplify the classification problem – index into previously collected databases • For visualization purposes – need Geographic Information System (GIS) data • Setback – extensive GIS data is not free!! Mahaneeya Raman - Extended Class Project

7. Image Stitching – steps involved • Camera Calibration • Correction for radial and tangential distortion • Projection onto cylindrical coordinates • Image stitching • Take-away from this work: • Problems faced - Parallax error • Suggestion on how to solve Mahaneeya Raman - Extended Class Project

8. Camera Calibration • To correct radial distortion and tangential distortion in images • Radial – Straight lines in real world appear curved in the image plane • Tangential – Image not located on a strict plane surface Mahaneeya Raman - Extended Class Project

9. Undistorting image and projection onto cylindrical coordinates • Polynomial distortion model • For aligning the images • To fit together as a panorama • Conversion from Cartesian to Cylindrical coordinates: r2 = x2 + y2 tan θ = y / x Mahaneeya Raman - Extended Class Project

10. Images Distorted Image Corrected image and projected onto cylindrical coordinates Mahaneeya Raman - Extended Class Project

11. Image Stitching • Load the two undistorted images • Select points using cpselect function in Matlab. Frame1_points and Frame2_points saved. • Determine transformation T, between the 2 images, x’ = Tx • Map one image onto the other based on the transformation. Mahaneeya Raman - Extended Class Project

12. Image Stitching output Final output Mahaneeya Raman - Extended Class Project

13. Image stitching of raw images – Parallax error Mahaneeya Raman - Extended Class Project

14. Understanding Parallax error • Distance related measurement error • High for images that are close to the camera lens • Occurs because Same points are located at different distances from two camera lenses, in both images • Distances between 2 given points is not the same in both images • Solution : • non-linear mapping has to be done • Ex. Radial Basis Function method / full planar perspective models Mahaneeya Raman - Extended Class Project

15. BodyMedia® SenseWear PRO2 Armband • Body sensor worn on upper right arm • Timestamp, Memory, Battery indicators • Connects to PC through USB cable • Collects through 6 continuous streams of data channels • Stores 30 channels of data Mahaneeya Raman - Extended Class Project

16. BodyMedia® SenseWear PRO2 Armband • 2 axis accelerometer • Heat flux sensor • Galvanic Skin Response sensor • Skin Temperature sensor • Near-Body Temperature Sensor Mahaneeya Raman - Extended Class Project

17. Sensor signal graph for typing-writing Skin Temperature Longitudinal Accelerometer Heat Flux Transverse Accelerometer Galvanic Skin Response Mahaneeya Raman - Extended Class Project

18. Applications of SenseWear Mahaneeya Raman - Extended Class Project

19. Data Collection • Pattern of data collected: • Typing – writing • Walking – simply sitting • Playing ‘quake’ – watching a comedy clip • Sample rate - 4 samples/second • Video surveillance using fire-i cameras while data is being collected • Timestamp between activities Mahaneeya Raman - Extended Class Project

20. Recorded video using 2 fire-i cameras Mahaneeya Raman - Extended Class Project

21. Sensor signal graph for typing-writing, walking - sitting, playing ‘quake’-relaxing Longitudinal Accelerometer Transverse Accelerometer Heat Flux Mahaneeya Raman - Extended Class Project

22. Data Analysis • Generate signal graph and excel sheets using bodymedia’s innerview software • Notice Accelerometer signals change considerably • Use a classifier algorithm • 50% data – training • 50% data – testing Mahaneeya Raman - Extended Class Project

23. KNN and LDA classifier – considered Algorithms for comparison • KNN – K Nearest Neighbor Algorithm • If x – to be classified, and (x1, y1), . . . , (xk, yk) are x’s k nearest neighbors, and d(x, xi) = distance between x and xi, x is classified into the nearest neighbor cluster. • LDA • Method to find linear discriminant boundaries between K classes • Define K linear discriminant functions for K classes • Classify x to the class with the largest value for its discriminant function Mahaneeya Raman - Extended Class Project

24. Conclusion • Useful GIS data can be integrated with GPS data for effective localization and environment analysis • Parallax error – can be solved by applying a non-linear transformation like RBF or full planar perspective models • Pending - Compare classifier algorithms to classify activity, which can be extended to predict human emotion as well Mahaneeya Raman - Extended Class Project

25. References [1] CAMEO: The Camera Assisted Meeting Event Observer – Paul E. Rybski, Fernando de la Torre, Raju Patil, Carlos Vallespi, Manuela Veloso, Brett Browning. [2] Image Warping - Mikkel B. Stegmann , Informatics and Mathematical Modelling, Technical University of Denmark. [3] Creating Full View Panoramic Image Mosaics and Environment Maps - Richard Szeliski and Heung-Yeung Shum, Microsoft Research. [4] 16 papers on bodymedia applications - http://www.bodymedia.com/research/whitepapers.jsp Mahaneeya Raman - Extended Class Project

26. Thank you Mahaneeya Raman - Extended Class Project

27. Matlab toolbox for calibration • Intrinsic calibration – focal length for each image axis, an image center, 3 terms of radial distortion, and 2 terms of tangential distortion. • Checker-board method – straight lines with easily localizable end points and interior points can be found in several orientations throughout the image plane. Mahaneeya Raman - Extended Class Project

28. Matlab toolbox for calibration • Images of checkerboard at different inclinations (horizontal, vertical, diagonal) • Provide size of the square • Extract grid corners of all images, one by one. • Provide size of window of squares chosen • Corner extraction - verification Mahaneeya Raman - Extended Class Project

29. Distortion Model – “Plumb Bob” model • P – point in space coordinate vector, XXc= [Xc;Yc;Zc] in camera ref. frame • Project P on the image plane according to intrinsic parameters (fc, cc, alpha_c, kc) • xn – normalized (pinhole) image projection, xn =[ Xc/ Zc;Yc/ Zc] = [x ; y] • Let r2 = x2 + y2 Mahaneeya Raman - Extended Class Project

30. Distortion Model – “Plumb Bob” model • After lens distortion, the new normalized point coordinate, xd = [xd(1) ; xd(2)] xd = (1+ kc(1)r2 + kc(2)r4 + kc(5)r6 )xn + dx • Where, dx – tangential distortion vector dx = [2kc(3)xy+ kc(4)(r2 + 2x2) ; kc(3)(r2 + 2y2)+ 2kc(4)xy ] Mahaneeya Raman - Extended Class Project

31. Distortion Model • The final pixel coordinates, x_pixel = [xp;yp] • Xp = fc(1)(Xd (1) + alpha_c*Xd (2)) +cc(1) • Yp = fc(2) Xd (2) +cc(2) • Therefore, [xp;yp;1] = KK [xd(1);xd(2);1] where, KK = [fc(1) alpha_c*fc(1) cc(1) ; 0 fc(1) cc(1) ; 0 0 1 ] Mahaneeya Raman - Extended Class Project

32. Image Stitching • Load 2 images, distortion corrected and projected onto cylindrical coordinates. • Select points using cpselect function in Matlab. Input_points and base_points saved. • Determine transformation T, between the 2 images, x’ = Tx x s cosα s sinα tx x y =-s sinα s cosα ty y 1 0 0 1 1 Mahaneeya Raman - Extended Class Project

33. Image Stitching - continued • By rearranging the equation so the warping parameters is the vector t in, x’ = Zt x’ x y 1 0 0s cosα y’ =y -x 0 1 0s sinα 1 0 0 0 0 1 tx ty 1 Mahaneeya Raman - Extended Class Project

34. Radial Basis Function method • The input layer is the set of source nodes (sensory units). • The second layer is a hidden layer of high dimension. • The output layer gives the response of the network to the activation patterns applied to the input layer. • The transformation from the input space to the hidden-unit space is nonlinear. • On the other hand, the transformation from the hidden space to the output space is linear. Mahaneeya Raman - Extended Class Project

35. Benefits of SenseWear • Detect energy expenditure during certain non-ambulatory activities • Detect increased effort and energy expenditure associated with load carrying. • Measure heat produced by the body as a result of basic metabolism and, as well as, all forms of physical activity. • Small, unobtrusive, and comfortable to wear. • It is not invasive and does not alter normal patterns of motion or activity Mahaneeya Raman - Extended Class Project