Photo-Quality Enhancement based on Visual Aesthetics

1. Photo-Quality Enhancement based on Visual Aesthetics S. Bhattacharya*, R. Sukthankar**, M.Shah* *University of Central Florida, **Intel labs & CMU

2. Motivation

3. Outline • Related work • Quality enhancement framework • Visual aesthetics • Aesthetic appeal assessment • Enhancement through recomposition • Experimental results • Conclusions • Future directions

4. Related Work • Low-level (dehazing etc.) K. He, J.Sun X. Tang, “Single Image Haze Removal using Dark Channel Prior”, CVPR 09 • Domain-specific (face beautification etc.) T.Leyvand,et al. , “Data-Driven Enhancement of Facial Attractiveness”, ACM SIGGRAPH 08

5. Overview Assessment Engine Aesthetic Features Appeal Prediction Image Semantics, Aesthetic Features Aesthetic Model Input Image Enhanced Image Recomposition Enhancement Engine

6. Visual Aesthetics: Rule of Thirds Motivated by Renaissance Paintings… http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htm Rule of thirds: Subject of interest is aligned to one of the stress points Professional photographs also abide this: http://howtophotography.org/wp-content/uploads/2010/06/rule-of-thirds-photo2.jpg

7. Visual Aesthetics: Golden Ratio ~1.618k Sky Sky Land Sea k http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htm Divine proportion: Horizon divides sky and sea/land according to golden ratio. An example professional photographic composition: http://www.dptips-central.com/rules-of-composition.html

8. Modeling Aesthetics: Dataset Single subject Compositions (384) Landscapes/Seascapes (248) http://www.flickr.com

9. Modeling Aesthetics: User study … Rank Assignment between 1-5 1 2 15 14 Single subjects Landscapes/Seascapes Ground Truth Appeal Factors http://www.flickr.com

10. Modeling Aesthetics: User study 1.76 4.23 Best rated images Poorly rated images

11. Modeling Aesthetics: User study Poor compositions Good Compositions User Agreements Appeal Factor Intervals

12. Modeling Aesthetics: Features (a) Relative Foreground Location (Rule of Thirds) Visual Attention Center Stress Point

13. Modeling Aesthetics: Features (b) Visual weight deviation from Golden Ratio (Divine Proportion) Yk Yg

14. Experiments (Assessment) • Learn Support Vector Regression models • Prediction accuracy: • Single subject compositions ~ 87% • Landscapes/Seascapes ~ 91% Smooth mapping between Appeal factor and Aesthetic Features • Relative Foreground Location • Visual Weight Deviation

15. Spatial Recomposition

16. Why Cropping does not work? Optimal Crop

17. Recomposition: Algorithm I Input Image Optimal Object Placement Spatial Recomposition In-painting Semantic Segmentation Single Subject? Labeled Elements

18. Semantic Segmentation Input Image Segmented Foreground Sky Geometric Context Classifier* Post Processing Horizon Support *D. Hoiem, A.A. Efros, and M. Hebert, "Geometric Context from a Single Image", ICCV 2005

19. Optimal Object Placement Find x that Maximizes Appeal Support Neighborhood Labeled Image s.t. neighbors stay “like neighbors” + Intensity Term Gradient Term

20. Optimization (Example) X PAF = 3.22 Original Image Semantic constraint prevents this Optimal Solution PAF = 3.31 PAF = 4.53 PAF = 3.68

21. Perspective Scaling Scaled Foreground Visual Attention Center Scaling Factor Optimal location Vanishing Point

22. Inpainting Foreground Hole Inpaint Hole • Yunjun Zhang. Jiangjian Xiao. Mubarak Shah, “Region Completion in a Single Image”, EUROGRAPHICS 04

23. Recomposition: Algorithm 2 Input Image Semantic Segmentation Land/Sea scape? Optimally Crop/Expand Labeled Elements Visual Weight Balancing

24. Balancing Visual Weights • Ratio of Current extents Yk+h Yk • h= vertical extent of the balanced image Yg Yg • Solve for h (sign of h determines crop/expansion)

25. Experimental Results • Single Subject Composition • Horse is moved to a more visually pleasing location • Scaled appropriately • Appeal increases by 64% • Before Recomposition • After Recomposition

26. Results PAF = 4.29 PAF = 2.45 • Before • After PAF = 3.98 PAF = 4.46

27. Results PAF = 4.02 PAF = 4.34 • Before • After PAF = 3.13 PAF = 4.19

28. Results PAF = 3.77 PAF = 4.25 • Before • After PAF = 3.92 PAF = 4.11

29. Results PAF = 2.71 PAF = 3.26 • Before • After PAF = 4.68 PAF = 4.06

30. Results • Optimally cropped support region to increase weights for sky • Appeal factor increased by 51% • Visual weight balancing • Before Recomposition • After Recomposition

31. Balancing Visual weights PAF = 4.02 PAF = 3.83 • Before • After PAF = 4.38 PAF = 3.92

32. Balancing Visual weights PAF = 4.71 PAF = 4.02 • Before • After PAF = 4.49 PAF = 4.17

33. Not Perfect • Algorithm says nice, humans: otherwise Fa = 2.41 (Ground Truth) PAF = 2.34 Fa = 2.54 (Ground Truth) PAF = 3.63 • Before • After

34. Summary: Optimal Placement • Increased # of Highly rated Images • Decreased # of Poorly rated Images • Before • After

35. Summary: Visual Weights • Increased # of Highly rated Images • Decreased # of Poorly rated Images • Before • After

36. Conclusion • Intelligent photo recomposition • Can also be used for aesthetic filtering • Easy to use practical tool

37. Future Work • Synthesizing ideal image from many photos of the same scene • Recompositionfor videos

38. Questions?