Segmentation of Building Facades using Procedural Shape Prior

1. Segmentation of Building Facades using Procedural Shape Prior Olivier Teboul, Loïc Simon, PanagiotisKoutsourakisand Nikos Paragios

2. Problem Rectified Facade image

3. Problem Rectified Facade image

4. Problem Rectified Facade image Segmentation into K classes

5. Problem • Rectified Facade image • Segmentation into K classes • windows

6. Problem • Rectified Facade image • Segmentation into K classes • windows • walls

7. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies

8. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors

9. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs

10. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs • sky

11. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs • sky • shops

12. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs • sky • shops • Enforce architectural constraints

13. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs • sky • shops • Enforce architectural constraints • alignment

14. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs • sky • shops • Enforce architectural constraints • alignment • consistent topology

15. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs • sky • shops • Enforce architectural constraints • alignment • consistent topology • Allow flexibility among façade layouts

16. Problem • Rectified Facade image • Segmentation into K classes • windows • walls • balconies • doors • roofs • sky • shops • Enforce architectural constraints • alignment • consistent topology • Allow flexibility among façade layouts • # floors • # windows • geometry

17. Applications • Image-based Modeling • Texture Databases

18. State of the Art • Grammar-free methods Mean Shift, Level Set, MRF-based methods Do not guarantee architectural consistency ! • Grammar-based methods • Image Driven : • Müller et al. Siggraph 07 • Koutsourakis et al. ICCV 09 • Grammar Driven : • Ripperda et al. DAGM 06 So far, lack of tying shape grammar strength with strong image support

19. Contributions • Shape Grammar driven approach • Learning the visual appearance of semantics • Energy minimization framework combining supervised learning with procedural shape prior • Database of Parisian facades http://www.mas.ecp.fr/vision/Personnel/teboul/data.html

20. Split Grammars • Shape grammar [Stiny 72] • Dictionnary of shapes • Replacement rules • Split rules only, along a direction [Wonka & al. Siggraph 03] • The semantics of the rule LHS, and (RHSi) are part of the dictionary D. • The geometry of the rule is LHS RHS1 RHS2 RHS3 … w1 w2 w3

21. Shape Grammar For Façade Modeling • Start from an axiom (Image) • Sequentially apply replacement rules • The derivation tree keeps track of the building structure

22. Shape Grammar For Façade Modeling • Start from an axiom (Image) • Sequentially apply replacement rules • The Derivation tree keeps track of the building structure

23. Shape Grammar For Façade Modeling • Start from an axiom (Image) • Sequentially apply replacement rules • The Derivation tree keeps track of the building structure

24. Shape Grammar For Façade Modeling • Start from an axiom (Image) • Sequentially apply replacement rules • The Derivation tree keeps track of the building structure

25. Shape Grammar For Façade Modeling • Start from an axiom (Image) • Sequentially apply replacement rules • The Derivation tree keeps track of the building structure

26. Constraining the grammar : factorization • If rules are applied independently, the derivation may lead to inconsistent buildings, in terms of alignment and topology across floors • Idea : apply the same rules on the same semantics

27. Advantages of Factorization • Produces only realistic buildings • Reduces the dimension of the space of shapes Factorization is a natural way to fight the curse of dimensionality • Allows a uniform representation of facade segmentations (independently from the layout topology). A segmentation is described by a fixed sequence of rules:  π = (r1, r2, …, rM)

28. Learning the vocabulary • Randomized Forest classifiers [Lepetit & Fua PAMI 06] • Code available at: www.mas.ecp.fr/vision/Personnel/teboul/source_code.html data annotations

29. Learning the vocabulary • The feature vectors are patches around the pixels data annotations

30. Learning the vocabulary Histogram

31. Learning the vocabulary Tests Histogram

32. Learning the vocabulary

33. Learning the vocabulary

34. Learning the vocabulary

35. Learning the vocabulary

36. Learning the vocabulary

37. Segmentation from Classification • Learning with 20 annotated images • Test on 10 other 10 annotated images input classification Ground truth Window probability Wall probability (red= 0 blue = 1)

38. Segmentation from Classification • Learning with 20 annotated images • Test on 10 other 10 annotated images

39. Segmentation energy

40. Segmentation energy • Single Pixel x

41. Segmentation energy • Single Pixel x

42. Segmentation energy • Single Pixel x • Single Region R R

43. Segmentation energy • Single Pixel x • Single Region R R

44. Segmentation energy • Single Pixel x • Single Region R R

45. Segmentation energy • Single Pixel x • Single Region R • Segmentation π

46. Segmentation energy • Single Pixel x • Single Region R • Segmentation π

47. Segmentation energy The energy ties the versatile grammar (π), with a strong (but noisy) image support from supervised learning.

48. Optimization • Start from an initial seed π0 = (r10, r20, …, rM0) π0

49. Optimization • Start from an initial seed π0 = (r10, r20, …, rM0) • Perform perturbations of the rulesπi = (r1i, r2i, …, rMi) π0

50. Optimization • Start from an initial seed π0 = (r10, r20, …, rM0) • Perform perturbations of the rulesπi = (r1i, r2i, …, rMi) π0