Curb Detector

1. Curb Detector

2. Automated Image Analysis for Robust Detection of Curbs Project Leader Name & Functional Area Wende Zhang (GM R&D / ECS Lab) David Wettergreen (CMU) Timing Date______________ Initial May, 2014 Midterm iMay, 2015 Final May, 2016 Detect and classify features using learning-based method Resources 2013 2014 2015 Total Material Cost[US$] 100k 100k 100k Total Headcount (GM) 0.1 0.1 0.1 Total Headcount (CMU) 1 1 1 • Description • Curbs are important cues on identifying the boundary of a roadway. Drivers understand an appropriate parking spot as defined by the curbs when reverse or parallel parking. Detecting curbs and providing information to assist drivers is an important task for active safety. Curb location is also crucial to autonomous parking systems. • Visual indications of curbs are widely various in the appearance. For example, under perspective imaging, projection of 3-dimensional curbs into 2-dimensional image plane distorts most of the curbs’ geometry properties, such as its angle, distance, and ratio of angles. Also, all curbs might be seen different because of age, wear, damage and lighting. Methods of detecting, localizing, and classifying curbs must address this diversity. This is to say, there is not a fixed template or set of templates that could be applied to reliably detect curbs through images. • Nevertheless visual appearance is how human drivers successfully detect curbs. Although physical structure can be sensed with some ranging sensors it not distinctive (two offset planes) or diagnostic of the roadway edge. Therefore we choose to pursue visual appearance. • This new project will develop an automated curb detection through : • Choosing appropriate features, learning those features to detect, and classifying the detected curbs • Utilizing the calibrated camera to fuse the 3D geometry information • One year development plan: detect, localize, and classify curbs using in-vehicle vision sensor with backward looking view with wide field of view • Motivation/Benefits • Identify the boundary of a road way in urban driving • Understand an appropriate parking spot as defined by the curbs when reverse or parallel parking • Deliverable / Technology Insertion into GM (What, When, Where) • Problem Definition: Survey of curbs • Data collection: Database of definite curb images and diverse curb images • Application: Detect curb features in perspective imagery • Experimental validation and performance analysis • Annual report GM Confidential

3. Use Case Slides

4. Assumptions • Color monocular camera • Known camera motion • Known intrinsic parameters • Maximum speed dependent upon frame rate

5. Use Cases • Parking lots • Backward parking • Parallel parking • Driveways • Roadways • Single lane • Multi-lane

6. GM Parking lots • Scenario : Curbs exist behind of a vehicle; rear-view camera with wide field of view • Success : Detect and localize curbs on images;(Optional) estimate the distance from a vehicle to curbs CU R B CU R B

7. GM Parking lots • Scenario : Parking curbs exist behind of a vehicle; rear-view camera with wide field of view • Success : Detect and localize parking curbs on images;(Optional) estimate the distance from a vehicle to parking curbs CU R B CU R B

8. GM Driveways • Scenario : Curbs exist at the side of the entrance of driveway; front-view camera with wide field of view • Success : Detect and localize curbs on images and indicates driveways as traversable path Driveways CURB CURB

9. GM GM Roadways • Scenario : Curbs exist at the side of the road; wide field of view camera • Success : Detect and localize curbs on images and indicates curbs as the non-traversable path and the boundary of road CURB CURB

10. Flow Chart Detection : Localize relevant curbs in each image Edge Segmentation Texture : Localize the detected curbs in remained images Tracking

11. Edge Detection Distorted Undistorted Edge Bird’s-eye view Edge

12. Segmentation

13. Texture Classification

14. Development Plan • Develop and test simple features • Train classifiers to detect and localize curbs • Evaluate classifier performance • Add complex features • Test quantify detection and localization performance • Train color classifiers to interpret appropriate parking spots • Motion Stereo to exploit 3D geometry

15. Scheme Extract Features Classification Tracking Edge Detection • Horizontal • Long features • Thin features • Color • Texture • Curvature • Appearance based tracking • Filters • Edges • Intensitydifferences • Gradients • Geometricconsideration

16. Data collection • Using 180 degree field of view camera • Install underneath the side mirror, tilt 45 degree down to the ground Sample images

17. Camera Calibration • Wider field of view, more distortion • Camera calibration is necessary in order to find geometry constrains (e.g., edges…) • Using OCamCalib (Omnidirectional Camera Calibration Toolbox) to calibrate camera Sample undistorted images

18. Shape Information • Edge detection • HOG feature

19. Edge Detection Extract Dominant Edges Input Image at t Undistorted Image Edge Detection Sequential RANSAC

20. Edge Detection Extract Dominant Edges Input Image at t Undistorted Image Edge Detection Sequential RANSAC

21. Edge Detection Extract Dominant Edges Input Image at t Undistorted Image Edge Detection Sequential RANSAC Two or Three parallel lines with small offsets are important cue for curbs

22. HOG feature

23. HOG feature

24. HOG feature Input image HOG map Score map Output image

25. Prerequisite • The maximum distance of ‘Curb Detection’ from a vehicle should be defined. • Given extrinsic parameters and the maximum distance, the followings can be estimated. • Different size of HOG model • Region of interest short medium long

26. GM GM Geometry calculation Local Area 2.7-3.6 m (9-12 feet) Maximum detect distance CURB

27. Examples Cadillac SRX http://www.cadillac.com/srx-luxury-crossover/features-specs/dimensions.html Cadillac CTS http://www.cadillac.com/cts-sport-sedan/features-specs/dimensions.html

28. GM Geometry calculation Maximum detect distance = 2.1 meter CURB

29. Image Sample with distance measure 1m 1m 1m • The center of the camera is 1.05m from the ground. • The angle of the camera is 45 degree down from the horizontal. • ROI will be reduced. (Red transparent rectangle) • ROI will be changed based on the extrinsic parameter.

30. Lane Markings • Since lane markings have strong edges, we need to eliminate outputs from lane markings. • Parts of images which contain lane markings can be removed by detecting white blobs.

31. Result Video

32. Performance Measure • Choose 300 testing images • Positive samples: images which contains full length of curbs • Negative samples: images without curbs • We consider curbs are detected when the horizontal length of the detected curbs are bigger than half of the horizontal length of image. • Since the size of image is 480 by 720, we consider curbs are detected and the sum of the length of the detected curbs are bigger than 360.

33. Performance Measure length of detected curb > 0.5 total length of image

34. Future Works • Features of curb detection • Redundant information through multiple images • Include tracking system to recover false negatives • Continuity • Develop likelihood function to recover false negatives and remove false positives • Height • Front-view camera • Mount 180 degree field of view camera on the front bumper

35. Front-view Camera Configuration