OpenCV

1. Università del Salento Facoltà di Ingegneria Image Processing (Elaborazione delle Immagini) A.A. 2012/2013 OpenCV PART II – Twocase studies Dario Cazzato, INO – CNR dario.cazzato@ino.it

2. Introduction • Thislesson introduce the useof the OpenCVlibrary on realcases. • Twostudycases: • Stereo CorrespondenceProblem; • Segmentationof Video Sequences.

3. WhatWe’ve seen • WhyOpenCV? • Free, open source, for real-time application, cross-platform, constantlyupdated, strong partners and research. • C/C++: • “cv::Mat vs CvMat”. • Basiccomponents: • Matrixes, vectors, rectangles, sizes, images, datatype… • Some example.

4. Stereo CorrespondanceProblem

5. Whatis Stereo Vision? • Men: • Binocular vision. • Averagedistancebetweeneyes: 6cm. • An object/pointseenwitheyesisviewedasone, altough in the retina Wehavetwoimages. • The combined image is more than the sum of its parts. It’snottrivial!

6. Whatis Stereo Vision? • Otherconfigurations: • Animals: • Predator: binocularsight. • Prey: lateraleyes (toenlarge the fieldofsight). • Intersectinglineofsight (typical in stereo vision!). • Let’ssee some key conceptfrom a practicalpointofview, butall the problemisabsolutelylarger! • Youwillsee more detailaboutepipolargeometry and matrixescomputation at lesson.

7. Pinhole camera model • The simplestmodelof the camera. • Only a single rayentersfromanyparticularpoint, the pinhole aperture. • Thispointisprojectedonto the imageplane. • The focallenghtis the distancefrom the pinhole aperture to the imageplane.

8. Pinhole camera model • A realpointQisprojectedonto the imageplaneby the raypassingthrough the center ofprojection. • Thisintersectiongives q. Calibration Matrix (3X4) From “LearningOpenCV”, G.Bradski, A.Kaehler, O’Reilly.

9. Pinhole camera model • Homogeneous coordinate system. • IfyouhaveNdimension, use N+1 coordinates. Homography PerspectiveGeometry

10. Whatis Stereo Vision? 1 Camera isnotenough!

11. Whatis Stereo Vision? Withtwo (or more) cameraswe can computedepthbytriangulation, ifwe are abletofindhomologouspoints in the twoimages. EpipolarGeometry

12. Whatis Stereo Vision? • Foursteps: • Undistortion; • Rectification; • DisparityMap; • Triangulation.

13. Whatis Stereo Vision? • Undistortion: removaloftangential and radiallensdistortion. Thisproblemconcerns the single camera! Distortionvector (1X5)

14. Whatis Stereo Vision? • Undistortion: removaloftangential and radiallensdistortion.

15. Whatis Stereo Vision? • Rectify: output row-alignedimages (coplanar, with the same y-coordinate). Withrectifiedimages, we can searchfor a point in oneimage in the sameline (y-coordinate) of the secondone!

16. Whatis Stereo Vision? • Ofcourse, a stereo calibrationisneeded (extrinsic and intrinsicparameters): • Intrinsic: focallenght, distortion. • Extrinsic: MatrixesR, Tthataligns the twocameras (Essential Matrix E, youwillsee more at lesson). • We can divide the procedure in : • Stereo Calibration: computationof the geometric relations between the twocameras in space; • Stereo Rectification: “correction” ofindividualimagesasmadewithrow-alignedimageplane and parallelopticalaxes.

17. Whatis Stereo Vision? Look at the exampleofOpenCV, and the source stereo_calib.cpp. Rectificationturns the cameras in standard form!

18. Whatis Stereo Vision? Example1

19. Whatis Stereo Vision? From “LearningOpenCV”, G.Bradski, A.Kaehler, O’Reilly.

20. Whatis Stereo Vision? • Disparitymap: difference in x-coordinatesof the samepointviewed in the 2cameras. A mapiscreatedcomputing the disparityforall the points. It’sencodedas a grayscaleimage, wherefarerpoint are darker.

21. Whatis Stereo Vision? • Triangulation: difference in x-coordinatesof the samepointviewed in the 2cameras. Idea: (d:T = f:Z)

22. Stereo CorrespondenceProblem • Howtofindhomologouspoints? • Correlation-based - checking if one location in one image looks/seems like another in another image; • Feature-based - finding features in the image and seeing if the layout of a subset of features is similar in the two images.

23. So, why so challenging? • Occlusions

24. So, why so challenging? • Photometric transformations

25. So, why so challenging? • Uniformregions

26. So, why so challenging? • Noise • Specular surfaces • Perspectiveviews • Repetition • “No resultspossible” detection: • Sometimewe just wouldliketosay: “No correspondentpoint in the otherimageforthispoint” • …

27. Stereo CorrespondenceProblem LocalAlgorithm Winner take allstrategy

28. WTA Strategy • Sum ofAbsoluteDifference: • Sum ofSquaredDifferences:

29. WTA Strategy • Zero-meanNormalized Cross Correlation: • Not just the window, but a fast normalization; • ZNCC hasrange [-1,1]; • Wecompute the ZNCC foreach pixel center; • We take the maxvalue.

30. ZNCC

31. ZNCC • Whatwe can do toenhounce the model? • Ratiobetween first and secondmax; • Idea behind: if the maximum and a localmaximumhavesimilarvalue, the probabilityoferrorincrease, and wecouldrejectthesevalues (putting a treshold). • Spread; • Idea behind: a flat area meansrepetitivetexture. Just discardmaximum in flatpeaks. • Multiple windows; • Kernel shapebased on segmentation. • Computationtimeincreases!

32. ZNCC –Ourmodel • Twoenhouncements: • Check the epipolarline±size: • We can deal withnoise in the epipolargeometry; • For a fast computation, keepsizesmall! (1,2,3). • Inverse function: • We take the maximum, and wemake ZNCC againstartingfrom the right image; • If the newwinnerisn’t the startingpoint (or is more than a treshold far, anerroroccurred, so discard the point). Demo 1

33. Segmentationof Video Sequences

34. Background subtraction • Oneof the Video SequencesSegmentationalgorithms. • Goodwithfixed camera and static background. • High level goal: • People detection. • Surveillance: • Reactive; • Proactive.

35. From the Web

36. CodebookModel • BS: subtract the current frame from the background model. • Twophases: • Background training; • Foreground detection. • Improvementsto the base version.

37. Segmentation - background model • A codebookisbuiltforevery pixel; • A codebookiscomposedbycodewords, boxes; thatgrowto cover the common valuesseenover the time; • Samplesofeach pixel are clustered in set ofcodewords; • Incoming pixel: • Ithas a brightness in the brightnessrange AND Color Distortionlessthan a treshold = BACKGROUND; • Othervise FOREGROUND.

38. Segmentation–codebookmodel

39. Segmentation - mnrl MNRL (Maximum Negative RunLenght): letustomake the background learningwithobjectsmovement. Itrefinescodebookseparatingcodebooksthat can haveforegroundfrom the real background. MNRL = 50%.

40. Segmentation–detectingforeground The foregroundissimplydetectedcomputing the distanceof the sample from the nearest cluster mean.

41. Segmentation–codebookalgorithm

42. Segmentation–codebookimprovements Leftobject in the scene:

43. Segmentation–codebookimprovements Holesproblem:

44. Segmentation–codebookimprovements • LayeringModeling/Detection - 3classesofcodebook and 3parametersthatlettoswitch in the categories: • Permanent; • Non-permanent; • Training. • AdaptativeCodebookUpdating: • Retrainingisnot the solution!! • Global status updating at each frame; • Periodicalcleaniningof the oldcodebook.

45. Filtering the binary frame Medianfilter:

46. Filtering the binary frame Medianfilter:

47. Filtering the binary frame MorphologicalOperators Opening and Closing:

48. Filtering the binary frame Opening and Closing:

49. Filtering the binary frame

50. Filtering the binary frame