Finding Glass

1. Kenton McHenry Jean Ponce David Forsyth Finding Glass

2. Background • Layer Seperation (Szleski, Avidan, and Aniandan, CVPR'00), (Levin, Zomet, and Weiss, CVPR'04) • 3D Structure (Hata, Saitoh, Kumamura and Kaida, ICPR'96) (Ben-Ezra and Nayar, ICCV'03) (Miyazaki, Kagesawa and Ikeuchi, ICCV'03) (Murase, ICCV'90) • Recognition (Osadchy, Jacobs, and Ramamoorthi, ICCV'03) • Segmentation (Singh and Huang, CVPR'03)

3. (Adelson and Anandan, AAAI'90) • 0 < a≤ 1 • e ≥ 0 I = aIB+ e

4. Classifying Junctions Non-Reversing: transparency, ambiguous depth ordering Single-Reversing: transparency Double-Reversing: no transparency

5. (Singh and Huang, CVPR'03)

6. (Singh and Huang, CVPR'03)

7. Our Goal

8. The Background • The appearance of a glass object changes with the background (i.e. the scene w/o any transparent objects) • We have seen how knowledge of the background can be extremeley useful in reconstructing transparent surfaces • Ideal situation: know the background, use background subtraction

9. Glass Objects and their Edges Why? • Highlights • Mirrors • Hysteresis

10. Adelson et al Revisited • Though they focus on junctions they are classifying edges • The proposed rules are binary cues between a transparent object and its background

11. Proposed Method • Break edges into small segments and classify them based on the information from the two sides • Properties of glass: transparency, refraction and reflection

12. Cues • Transparency • Color Similarity • Overlay Consistency • Refraction • Texture Distortion • Blurring • Reflection • Highlights

13. Color Similarity • (HSV) Hue • (HSV) Saturation

14. Overlay Consistency

15. Texture Distortion • Filer Bank: 2 scales, 6 orientations (0,p)

16. Blurring • DCT • Shift in mean in frequency space

17. Highlights • Highlights on smooth shiny surfaces tend to have a profile with a sharp spike (Healey and Binford, '87), (Nayar, Ikeuchi and Kanade, '91)

18. Highlights • Iteratively fit a line to perimeter (starting from threshold of 1.0) • Plot line fit errors

19. Highlights

20. Single Classifier • 5 cues provide 6 values • SVM with Gaussian kernel • Must be conservative with false positives • Classifier can achieve high accuracy on training data • Move hyperplane until true positives < 30%

21. Multiple Classifiers • If we were to consider the 6 values as logical propositions we could write: glass ⇐ similar_color ∧ high_alpha ∧ (low_emmission ∨ highlight ∨ smoother ∨ distortion)

22. Multiple Classifiers • We can re-write the previous statement as four different statements of three propositions: glass ⇐ similar_color ∧ high_alpha ∧ low_emmission glass ⇐ similar_color ∧ high_alpha ∧ highlight glass ⇐ similar_color ∧ high_alpha ∧ smoother glass ⇐ similar_color ∧ high_alpha ∧ distortion

23. Multiple Classifiers • Each proposition is a seperatley trained classifier of lower dimension • Combining the sub-classifiers • Logical OR • Weighted Sum • Exponential Model

24. Global Integration • Due to conservativeley built classifiers we will have few positives • Hysteresis: connect positves along a common edge • Snakes (Kass, Witkin, Terzopoulos, '87)

25. Experiments • Training Set: 15 images, 6 with glass objects in front of various backgrounds, 9 with no glass objects • 333 positive examples • 4581 negative examples • Test Set: 50 images, 35 with glass objects, 15 with no glass objects at all

26. Experiments Precision 68.76% 56.04% 58.78% 56.04% 73.70% Single SVM Multiple SVM's + OR Multiple SVM's + Weighted Sum Multiple SVM's + Exponential Model Multiple SVM's + Weighted Sum (sampled)

27. Results

28. Results

29. Results

30. Results

31. Classifying Regions as Glass • We need not restrict ourselves to regions around edges • Given two regions we ask the question “is one region a glass covered version of the other?”

32. Over Segmentation • We want regions of similar material (Felzenszwalb and Huttenlocher, '04) • Can adjust size of super-pixels (degree of over-segmentation) with smaller k values • Use color, texture, and edgels to set weights

33. Discrepency • We use our previous classifier as a measure of how much two regions don't belong two the same material (i.e. glass and not glass) • Use distance from seperating hyperplane (Platt, '00) • Large values: far on the postive glass side • Small values (negative): far on the not glass side • Reasonable if data takes a normal distribution • Drop blur cue since DCT can't be done on non-rectangular regions.

34. Ambiguities • Discrepency is high for a material and a glass covered version of that material, but also for two completley different materials • Above example has two possible segmentations

35. Affinity Aij = 1 – aij / p

36. Affinity • Because of refraction most straight background edges that pass through the glass will appear broken • Edges from glass contour ussually the longest smoothest edges in the area

37. Affinity

38. Certainty of Discrepency/Affinity • High discrepency: likely different materials • Low discrepency: cannot ascertain whether one regions is glass and the other is background • High affinity: likely same material • Low affinity: not very informative, edge path may just have been broken

39. Objective Function • We wish to maximize our measures • First term: maximize discrepency between glass and other stuff • Second term: maximize affinity in the glass • Third term: minimize affinities between glass and other • Combinatorial problem!

40. Relaxed Objective Function • Relax region constraints • Treat pixels as a sampling of an underlying continuous function

41. Geodesic Active Contours

42. Curve Evolution

43. Experiments Precision 68.76% 56.04% 58.78% 56.04% 73.70% 77.03% Single SVM Multiple SVM's + OR Multiple SVM's + Weighted Sum Multiple SVM's + Exponential Model Multiple SVM's + Weighted Sum (sampled) Proposed Method

44. Results

45. Results

46. Results

47. Results

48. Results