Monocular Vision Sensor for Curb Detection

3. Objectives Develop reliable methods of detecting, localizing, and classifying featuresassociated with curbs using in-vehicle, low-cost, monocular vision sensor Localize curbs within a range of 5 meters with 99% accuracy

4. Approaches for Curb Detection Using Mono Camera Images • Appearance-based image analysis (~ Nov. 2013) • Extract features • Evaluate performance • Geometry-based image analysis (~ May. 2014) • Structure-from-motion to estimate camera motion • Multi-resolution plane sweeping algorithm to create 3-D point cloud • Plane fitting to detect curb • Combine appearance and geometric analysis (This Review)

5. Approaches for Curb Detection Using Mono Camera Images • Appearance-based image analysis (~ Nov. 2013) • Extract features • Evaluate performance • Geometry-based image analysis (~ May. 2014) • Structure-from-motion to estimate camera motion • Multi-resolution plane sweeping algorithm to create 3-D point cloud • Plane fitting to detect curb • Combine appearance and geometric analysis (This Review)

6. Appearance-based image analysis

7. Edge Detection

8. Detect Curb Using HOG* Feature Curb model HOG image Input image * Histogram of Oriented Gradients

9. Approaches for Curb Detection Using Mono Camera Images • Appearance-based image analysis (~ Nov. 2013) • Extract features • Evaluate performance • Geometry-based image analysis (~ May. 2014) • Structure-from-motion to estimate camera motion • Multi-resolution plane sweeping algorithm to create 3-D point cloud • Plane fitting to detect curb • Combine appearance and geometric analysis (This Review)

10. Geometry-based image analysis Input image Depth image 3-D point cloud Ground plane estimation

11. Plane Fitting

12. Approaches for Curb Detection Using Mono Camera Images • Appearance-based image analysis (~ Nov. 2013) • Extract features • Evaluate performance • Geometry-based image analysis (~ May. 2014) • Structure-from-motion to estimate camera motion • Multi-resolution plane sweeping algorithm to create 3-D point cloud • Plane fitting to detect curb • Combine appearance and geometric analysis (This Review)

13. Schematic Overview Input at t Appearance at t Candidate regions Annotate curb region Input at t+1 Appearance at t+1 Geometry

14. Appearance • For each image, divide intom x n grids • m: image height /grid size (pixels) • n: image width / grid size (pixels) Image at t

15. Appearance • For each grid, classify among two classes (road, curb) • uniform Local Binary Pattern (LBP) Image at t

16. Local Binary Pattern threshold Binary: 00011110 Decimal: 30 255 0 1 2 3

17. Appearance Output at t Output at t+1 Intersect • Once all the grids of two images are classified, get the intersection of them

18. Geometry • Green lines shows the vectors from the interesting points of image at time t (blue dots) to those of image at time t+1 (red dots) • Calculate the 3-D points using camera matrix

19. Appearance + Geometry • For each grid, • Fit the best plane using 3-D points • Compute the normal vector • Determine the normal vector is a road surface or a curb surface

20. Appearance + Geometry

21. Extend Curb Region • If the appearances are similar, extend the curb region • Calculate the distance of LBPs using chi-square

22. Extend Curb Region

23. Track Curb Region Input at t+2 Input at t+3 Input at t+4 • For the next frames, tracking the appearance of the curbs • When tracking, keep checking the geometry constraint to remove the false positives if exist

24. Curved curb case Combine Analyses Extend Curb Region

25. Curb Detection using Production Camera Image size : 480 by 640 FOV: 180 degree

26. Test Curb Detection Image Size: 640 x 480 (pixels) ROI: 640 x 160 (pixels) Size of grid: 20 x 20 (pixels) Number of grids: 32 x 8 Output of the appearance-based curb detection

27. Test Curb Detection Remove outliers based on cluster size Find edges using Canny operator inside candidate region

28. Test Curb Detection - Fit polynomials to each segments, and check lines for similar curvatures (blue), and remove high curvatures (red) Annotate curb region on the original input image

30. Thank you Questions ?