Overview of Our Sensors For Robotics

1. Overview of Our Sensors For Robotics

2. Machine vision • Computer vision • To recover useful information about a scene from its 2-D projections. • To take images as inputs and produce other types of outputs (object shape, object contour, etc.) • Geometry + Measurement + Interpretation • To create a model of the real world from images.

3. Topics • Computer vision system • Image enhancement • Image analysis • Pattern Classification

4. Related fields • Image processing • Transformation of images into other images • Image compression, image enhancement • Useful in early stages of a machine vision system • Computer graphics • Pattern recognition • Artificial intelligence • Psychophysics

5. Vision system hardware

6. Image Processing System

8. Image Representation

9. Image • Image : a two-dimensional array of pixels • The indices [i, j] of pixels : integer values that specify the rows and columns in pixel values

11. Sampling, pixeling and quantization • Sampling • The real image is sampled at a finite number of points. • Sampling rate : image resolution • how many pixels the digital image will have • e.g.) 640 x 480, 320 x 240, etc. • Pixel • Each image sample • At the sample point, an integer value of the image intensity

12. Quantization • Each sample is represented with the finite word size of the computer. • How many intensity levels can be used to represent the intensity value at each sample point. • e.g.) 28 = 256, 25 = 32, etc.

13. Color models • Color models for images, • RGB, CMY • Color models for video, • YIQ, YUV (YCbCr) • Relationship between color models :

15. 6.7. Digital Cameras

16. Digital Cameras • Technology • CCD (charge coupled devices) • CMOS (complementary metal oxide semiconductor) • Resolution • 60x80 black/white up to • several Mega-Pixels in 32bit color However:Embedded system has to have computing power to deal with this large amount of data!

17. Vision (camera + framegrabber)

18. Digital Cameras • Performance of embedded system: 10% - 50% of standard PC

19. Interfacing Digital Cameras to CPU • Interfacing to CPU: • Completely depends on sensor chip specs • Many sensors provide several different interfacing protocols • versatile in hardware design • software gets very complicated • Typically: 8 bit parallel (or 4, 16, serial) • Numerous control signals required

20. Interfacing Digital Cameras to CPU • Digital camera sensors are very complex units. • In many respects they are themselves similar to an embedded controller chip. • Some sensors buffer camera data and allow slow reading via handshake(ideal for slow microprocessors) • Most sensors send full imageas a streamafter start signal • (CPU must be fast enough to read or use hardware buffer or DMA) • We will not go into further details in this course. However, we consider camera access routines

21. Simplified diagram of camera to CPU interface

22. Problem with Digital Cameras • Problem • Every pixel from the camera causes an interrupt • Interrupt service routines take long, since they need to store register contents on the stack • Everything is slowed down • Solution • Use RAM buffer for image and read full image with single interrupt

23. Idea • Use FIFO as image data buffer • FIFO is similar to dual-ported RAM, it is required since there is no synchronization between camera and CPU • When FIFO is half full, interrupt is generated • Interrupt service routine then reads FIFO until empty • (Assume delay is small enough to avoid FIFO overrun)

26. Bayer Pattern

27. De-Mosaic

29. Conversion in Digital Cameras • Bayer Pattern • Output format of most digital cameras • Note: 2x2 pattern is not spatially located in a single point! • Can be simply converted to RGB (drop one green byte) 160x120 Bayer → 80x60 RGB • Can be better converted using “demosaicing” technique 160x120 Bayer → 160x120 RGB

31. CMUCAM2+ CAMERA www.seattlerobotics.com • The camera can trackuser defined color blobs at up to 50 fps (frames per second) • Track motion using frame differencing at 26 fps • Find the centroid of any tracking data • Gather mean color and variance data • Gather a 28 bin histogram of each color channel • Manipulate horizontal pixel differenced images • Arbitrary image windowing • Adjust the camera’s image properties This camera can do a lot of processing

32. This camera can do a lot of processing • Dump a raw image • Up to 160 X 255 resolution • Support multiple baud rates • Control 5 servos outputs • Slave parallel image processing mode off of single camera bus • Automatically use servos to dotwo axis color tracking • B/W analog video output (Pal or NTSC) • Flexible output packet customization • Multiple pass image processing on a buffered image

34. Vision Guided Robotics and Applications in Industry and Medicine

35. Contents • Robotics in General • Industrial Robotics • Medical Robotics • What can Computer Vision do for Robotics? • Vision Sensors • Issues / Problems • Visual Servoing • Application Examples • Summary

36. Industrial Robot vs Human • Human advantages: • Intelligence • Flexibility • Adaptability • Skill • Can Learn • Can Estimate • Robot Advantages: • Strength • Accuracy • Speed • Does not tire • Does repetitive tasks • Can Measure Robot needs vision

37. Industrial Robot • Requirements: • Accuracy • Tool Quality • Robustness • Strength • Speed • Price Production Cost • Maintenance Production Quality

38. Medical (Surgical) Robot • Requirements • Safety • Accuracy • Reliability • Tool Quality • Price • Maintenance • Man-Machine Interface

39. What can Computer Vision do for (industrial and medical) Robotics? • Accurate Robot-Object Positioning • Keeping Relative Position under Movement • Visualization / Teaching / Telerobotics • Performing measurements • Object Recognition • Registration Visual Servoing

40. Vision Sensors • Single Perspective Camera • Multiple Perspective Cameras (e.g. Stereo Camera Pair) • Laser Scanner • Omnidirectional Camera • Structured Light Sensor

41. Vision Sensors • Single Perspective Camera Single projection

42. Vision Sensors • Multiple Perspective Cameras (e.g. Stereo Camera Pair)

43. Vision Sensors • Multiple Perspective Cameras (e.g. Stereo Camera Pair)

44. Vision Sensors • Laser Scanner

45. Vision Sensors • Laser Scanner

46. Vision Sensors • Omnidirectional Camera

47. Vision Sensors • Omnidirectional Camera

48. Vision Sensors • Structured Light Sensor Figures from PRIP, TU Vienna

49. Issues/Problems of Vision Guided Robotics • Measurement Frequency • Measurement Uncertainty • Occlusion, Camera Positioning • Sensor dimensions

50. Visual Servoing • Vision System operates in a closed control loop. • Better Accuracy than „Look and Move“ systems Figures from S.Hutchinson: A Tutorial on Visual Servo Control