Fast, Multiscale Image Segmentation: From Pixels to Semantics

1. Fast, Multiscale Image Segmentation: From Pixels to Semantics Ronen Basri The Weizmann Institute of Science Joint work with Achi Brandt, Meirav Galun, Eitan Sharon

4. Camouflage

5. Camouflage Malik et al.’s “Normalized cuts”

11. Our Results

12. Segmentation by Weighted Aggregation A multiscale algorithm: • Optimizes a global measure • Returns a full hierarchy of segments • Linear complexity • Combines multiscale measurements: • Texture • Boundary integrity

13. The Pixel Graph Couplings (weights) reflect intensity similarity Low contrast – strong coupling High contrast – weak coupling

14. Normalized-cut Measure Minimize:

15. Saliency Measure Minimize:

16. Multiscale Computation of Ncuts • Our objective is to rapidly find the segments (0-1 partitions) that optimize E. • For single-node cuts we simply evaluate E. • For multiple-node cuts we perform “soft contraction” using coarsening procedures from algebraic multigrid solvers of PDEs.

17. Coarsening the Graph • Suppose we can define a sparse mapping such that for all minimal states

18. Coarse Energy • Then • PTWP, PTLP define a new (smaller) graph

19. Recursive Coarsening

20. Recursive Coarsening Representative subset

21. , sparse interpolation matrix Recursive Coarsening For a salient segment :

22. Weighted Aggregation aggregate aggregate

23. Hierarchical Graph • Pyramid of graphs • Soft relations between levels • Segments emerge as salient nodes at some level of the pyramid

24. Importance of Soft Relations

25. Physical Motivation • Our algorithm is motivated by algebraic multigrid solutions to heat or electric networks • u- temperature/potential • a(x, y) – conductivity • At steady state largest temperature differences are along the cuts • AMG coarsening is independent of f

26. Determine the Boundaries 0 0 1,0,0,…,0 1 P

27. Hierarchyin SWA

28. Texture Examples

29. Filters (From Malik and Perona) Center- surround Oriented filters

30. A Chicken and Egg Problem Problem: Coarse measurements mix neighboring statistics Solution: Support of measurements is determined as the segmentation process proceeds Hey, I was here first

31. Texture Aggregation • Aggregates assumed to capture texture elements • Compare neighboring aggregates according to the following statistics: • Multiscale brightness measures • Multiscale shape measures • Filter responses • Use statistics to modify couplings

32. Recursive Computation of Measures • Given some measure of aggregates at a certain level (e.g., orientation) • At every coarser level we take a weighted sum of this measure from previous level • The result can be used to compute the average, variance or histogram of the measure • Complexity is linear

33. Use Averages to Modify the Graph

34. Adaptive vs. Rigid Measurements Original Averaging Geometric Our algorithm - SWA

35. Adaptive vs. Rigid Measurements Original Interpolation Geometric Our algorithm - SWA

36. Adaptive vs. Rigid Measurements

37. Adaptive vs. Rigid Measurements

38. Adaptive vs. Rigid Measurements

39. Adaptive vs. Rigid Measurements

40. Adaptive vs. Rigid Measurements

41. Texture Aggregation Fine (homogeneous) Coarse (heterogeneous)

42. Multiscale Variance Vector

43. Multiscale Variance Vector

44. Variance: Avoid Mixing aggregation Sliding window

45. Leopard

46. More Leopards…

47. And More…

48. Birds

49. More Animals

50. Boat