Prepared by: Zeke Susman Dept. of Electrical and Computer Engineering Utah State University

1. ECE5320 MechatronicsAssignment#01: Literature Survey on Sensors and Actuators Topic: Optical Flow Sensor Prepared by: Zeke Susman Dept. of Electrical and Computer Engineering Utah State University E: zeke.susman@gmail.com; F: (435)797-3054 (ECE Dept.) 3/11/2010

2. Outline • Reference list • To probe further • Major applications • Basic working principle illustrated • Methods and Implementations Compared • Major specifications • Limitations • And many more relevant issues in applications (such as, how to choose, cost information, where to buy etc.) ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

3. Reference list • http://en.wikipedia.org/wiki/Optical_flow • John L. Barron, David J. Fleet, and Steven Beauchemin (1994). "Performance of optical flow techniques". International Journal of Computer Vision (Springer). http://www.cs.toronto.edu/~fleet/research/Papers/ijcv-94.pdf. • http://www.pages.drexel.edu/~kws23/tutorials/opticFlow/opticFlow.html • Selim Temizer, Optical Flow Based Robot Navigation, http://people.csail.mit.edu/lpk/mars/temizer_2001/Optical_Flow/ • Jeongho Shin, et All, Optical flow-based real-time object tracking using non-prior training active feature model, Real-Time Imaging, Volume 11, Issue 3, Special Issue on Video Object Processing, June 2005, Pages 204-218, ISSN 1077-2014, DOI: 10.1016/j.rti.2005.03.006. (http://www.sciencedirect.com/science/article/B6WPR-4GBD6Y4-1/2/76e94d843fe22a8c7f7db5c8534ac500) ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

4. To explore further (survival pointers of web references etc) • Algorithms • Anandan P. (1989) A computational framework and an algorithm for the measurement of visual motion. Int. J. Comp. Vision 2, pp 283-310 • Fleet D.J. and Jepson A.D. (1990) Computation of component image velocity from local phase information. Int. J. Comp. Vision 5, pp. 77-104 • Horn B.K.P. and Schunk B.G (1981) Determining optical flow. AI 17, pp. 185-204 • Lucas, B. and Kanade, T. (1981) An iterative image registration technique with an application to stereo vision. Proc. DARPA IU Workshop, pp. 121-130 • Singh A. (1992) Optic Flow Computation: A Unified Perspective. IEEE Computer Society Press • [Beyeler et al., 2006] Beyeler, A., Mattiussi, C., Zufferey, J.-C., and Floreano, D. (2006). Vision-based altitude and pitch estimation for ultra-light indoor microflyers. Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, pages 2836–2841. http://www.ab-ware.com/papers/icra06_beyeler_final.pdf • Hardware Implementations • F.Aub´epart and N.FranceschiniBioinspired optic flow sensors based on FPGA: Application to Micro-Air-Vehicles Microprocessors and Microsystems, 31:408–419,2007 • P.C. Arribas and F.M.H. Maci´a. FPGA implementation of Santos-Victor optical flow algorithm for real-time image processing: an useful attempt. Proceedings of SPIE, 5117:23–32, 2003. • Garratt M.A. et all, FPGA implementation of an Optic Flow Sensor using Image Interpolation Algorithm, http://www.araa.asn.au/acra/acra2009/papers/pap116s1.pdf ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

5. Major applications Optical Flow is an idea taken straight from nature to gain information about the environment with a single solid state sensor. Some areas of application are: • Robot Navigation • Object tracking • Image Segmentation • Altitude estimation • Motion Estimation • Visual Odometery • Air Flow measurement The following slides will high light some of the more common applications. ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

6. Robot Navigation • The green arrows indicate a velocity field or the relative motion from one frame to the next. • The wall exhibits a consistent velocity field, except in the doorway. • The lack of velocity in the door way indicates a passable area. http://people.csail.mit.edu/lpk/mars/temizer_2001/Optical_Flow/Images/flow1-2.gif ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

7. Object Tracking • The optical flow of the image allows the subject to be tracked from frame to frame. • This is also an example of object segmentation by clustering the points in the image with the same velocity the velocity field of the subject stands out from the non moving back ground. Jeongho Shin, et All see references ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

8. Altitude Estimation • If the speed of the sensor is known or constant, the rate at which the objects on the ground of the image is moving can be used to determine the altitude. • High altitude flight the ground moves slowly • Helicopter or ultra light the ground moves fast • Ground vehicle the surface of the street zooms h Vsurface ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

9. Air Flow measurement • A laser is projected over a distance. • The light is diffracted as it passes through particles in turbulent motion. • The diffraction of light creates a visible pattern similar to the motion of hot air above a road or in a desert mirage. • The OF sensor uses the velocity fields of the diffraction and can make estimates of how the particles and thus the air is moving. Smoke stack Light Src OF Sensor ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

10. Basic Principle • The basic principle behind Optical Flow is the relative change in bright patches between two images provide an indicator of motion. t0 t1 ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

11. Real Example • Velocity Fields indicate: • The camera is moving • ground is moving back wards • Vehicle in the picture is moving slightly faster than and in the same direction as the camera. • Notice the velocity fields indicating the motion of the light spots in the image: • Headlights • Sun reflection on roof and contours of body • Patches of light colored sand in the back ground http://robots.stanford.edu/cs223b05/notes/CS%20223-B%20T1%20stavens_opencv_optical_flow.pdf ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

12. Methods and Implementations • Intensity-based differential • Lucas and Kanade • Horn and Schunck • Frequency-based filtering • Energy-based (Heeger) • Phase-based (Fleet and Jepson) • Correlation-based • Singh • Anandan’s technique ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

13. General Algorithm All the methods share the same three general steps: • Pre-filtering or smoothing of the images • Generation of basic measurable information • Flow field generation through integration The next several slides discuss the pros and cons of the most common implementations of the previously listed methods. ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

14. Frequency-based Fleet and Jepson (Pros) (Cons) Sensitive to temporal aliasing Lack of single measure of confidence Processing intensive Requires large sequences of images 10+ Very slow • Phase based • Most accurate • Works well with dense structures with homogeneous inputs ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

15. Intensity-based Differential Lucas and Kanade (Pros) Cons Sensitive to temporal aliasing Lack of single measure of confidence Processing intensive Requires large sequences of images 10+ • Gradient based • Works well with dense homogeneous inputs • Sub-pixel optical flow measurements are achievable. • Linear noise sensitivity • Fast output ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

16. Correlation-based Singh (Pros) (Cons) Poor sub-pixel displacement estimation Suffers greatly from the aperture problem Does not handle scenes with significant dilation Lack of reliable confidence measures • Handles pure translation and High velocities reasonably well. • Able to produce reasonable estimates with short sequences (2-3 frames) ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

17. Summary of Methods • In summary the Fleet and Jepson (FJ) and the Lucas and Kanade (LK) are the most accurate and popular algorithms for implementing Optical Flow. The final choice depends on the application. ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

18. Major Specifications • When deciding on a specific method of Optical Flow to use in your project the things to decide on are: • Accuracy Requirements • Measure of confidence • Type of environment (cluttered, multiple moving objects, dilation of the view) • Processing power and speed • Hardware vs Software based implementation ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

19. Major Limitations • Aperture problem • Changing light condition • Extreme frame rates • Low texture views • Occlusion • Multiple sources of motion • Spatial or Temporal aliasing ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

20. Aperture Problem • Which velocity vector produced the change from A to B? Left Down ? A B Diagonal t0 t1 ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

21. Lighting Conditions • As the area of focus moves across a shaded area this can result in non constant brightness which confuses the search for bright areas from image to image. • A moving shadow can also act like a moving object across a static back ground. ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

22. Aliasing • Aliasing is a result of under sampling the analog signal, video in most cases. • The results of sampling show up not only in the time frequency domain but in the spatial domain as well. • Appropriate filtering is needed by most OFS methods. ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators

23. Where to get Because the field of robotics is relatively young and the high variability of the applications most users of Optical Flow choose to implement their own sensors using open source and commercially available software. There are a lot of papers describing OF implementations in FPGAs but the commercial market is small. There is only one commercial hardware implementation which is to be released later this year (2010). • RoboRealm – software • Cost: $89.00 • http://www.roborealm.com/index.php • OpenCV – software • Cost: free • http://opencv.willowgarage.com/wiki/ • Centeye - hardware • http://www.centeye.com/index.html • Cost ?? – (available some time in 2010) ECE5320 Mechatronics. Assignment#1 Survey on sensors and actuators