Composition-Guided Image Acquisition

1. Composition-Guided Image Acquisition Serene Banerjee Ph.D. Defense, April 28th, 2004 http://www.ece.utexas.edu/~serene Committee Members: Prof. Ross Baldick Prof. Alan C. Bovik Prof. Brian L. Evans (Advisor) Prof. Wilson S. Geisler Prof. Joydeep Ghosh Prof. Robert W. Heath, Jr. Computer Engineering Curriculum Track Dept. of Electrical and Computer Engineering The University of Texas at Austin

2. “One day Alice came to a fork in the road and saw a Cheshire cat in a tree. ‘Which Road do I take?’ she asked. ‘Where do you want to go?’ was his response. ‘I don’t know,’ Alice answered. ‘Then,’ said the cat, ‘it doesn’t matter.” Lewis Carroll Alice in Wonderland Composition-Guided Image Acquisition

3. Outline • Introduction • Motivation • Overview of contributions • Summary of previous research for main subject detection • Contributions • Online main subject detection • Aesthetic enhancements, given main subject • Blur background objects merging with main subject • Conclusions Composition-Guided Image Acquisition

4. No foreground / background distinction Too muchbackground Main subjectcropped Motivation • Problem: Amateur photographers take unappealing pictures (e.g. personal and business use) • Help users take better pictures with digital cameras Composition-Guided Image Acquisition

5. Avoid Merger Amateur Shot Professional Rule-of-thirds Blur background Amateur Shot Professional Professional Amateur Placement Enhance Picture Appeal • Improving photograph appeal [Savakis, Etz & Loui; 2000] • Photographic composition • Objective measures • People/expression • Examples of photographic composition rules Composition-Guided Image Acquisition

6. Enhance Acquired Picture Appeal • Goal: Provide well-composed alternative pictures during image acquisition in digital still cameras • Solution: Framework for in-camera automation of photographic composition rules • Acquire picture user intended to take • Locate main subject by combining optical and digital image processing on a supplementary picture • Apply composition rules to user-intended picture • Place main subject according to rule-of-thirds • Blur entire background given main subject location • Blur background objects that merge with main subject • User takes intended picture and framework also returns three alternative pictures Composition-Guided Image Acquisition

7. Offline Main Subject Detection • Neural network based training [Luo, Etz, Singhal & Gray; 2000-2001] • Cluster multi-level wavelet coefficients [Wang et al.; 1999-2001] • Iterative classification from variance maps [Won, Pyan & Gray; 2002] Composition-Guided Image Acquisition

8. Automating Composition Rules • In-camera online framework • Provide alternatives to user during image acquisition • One-pass low-complexity algorithms [Banerjee & Evans; 2003-04] • Independent of scene content and setting • Amenable to fixed-point implementation • Match processing on digital still cameras Original color image Detect main subject Supplementary picture Generated picture with rule-of-thirds Rule-of-thirds Background blur Generated picture with blur Mitigate merger Generated picture without mergers Composition-Guided Image Acquisition

9. Digital Still Cameras • Converts optical image to electric signal • Software control • Shutter aperture and speed • Focus • Zoom • White balance • Additional hardware could control • Camera angle • Aspect ratio: landscape or portrait Composition-Guided Image Acquisition

10. Outline • Introduction • Contributions • Online main subject detection • In-camera segmentation of the main subject • Low-complexity one-pass algorithm • Amenable to implementation in digital still cameras • Aesthetic enhancement, given main subject • Mitigation of mergers with background objects • Conclusions Composition-Guided Image Acquisition

11. Contribution #1 Online Main Subject Detection • Auto-focus main subject • Take supplementary picture • Open shutter aperture (takes 1s) to blur objects not in focus • In-focus edges stronger than out-of-focus edges • Process supplementary picture to find main subject mask • Enhance in-focus edges • Detect strong edges • Close boundary Scene Auto-focus filter Open shutter for blur Supplementary picture Compute intensity 3x3 Highpass filter Detect sharper edges Close boundary Binary main subject mask Composition-Guided Image Acquisition

12. Contribution #1 Main Subject Detection: Formulation • Supplementary picture has intensity function, I • IHand ILare highpass and lowpass versions • For background image, contribution from ILis greater • Goal: Identify pixels contributing high frequencies • I is modeled as mixture of IHand IL • Highpass filtering of I enhances main subject edges where k 1 Composition-Guided Image Acquisition

13. Contribution #1 Step 1: Enhance In-focus Edges • Subtract smoothed image from sharpened one • Strong edges in main subject, weak edges in background Highboost image + Σ - Edge-enhanced image with stronger main subject edges Supplementary image Lowpass image Composition-Guided Image Acquisition

14. Contribution #1 Step 2: Detect Strong Edges • Canny edge detector detects strong edges [Canny; 1986] • Selects weak edges only if they are connected to strong edges • Laplacian of Gaussian detector [Burt & Adelson; 1983] • Selects edges based on zero crossings of second derivative • Either detects weak and strong edges or eliminates weak edges from main subject (depends on threshold) Laplacian of Gaussian Canny edge detector Composition-Guided Image Acquisition

15. Contribution #1 Step 3: Generate Mask • Goal: Generate closed contour from strong edges • Gradient vector flow [Xu, Yezzi & Prince; 2001] • Balances forces • Internal: spline characteristics • External: normal of gradient of detected strong edges • Outer boundary of detected sharp edges is initial contour • Change shape of initial contour, depending on gradient • Approximate lower complexity method • Select leftmost & rightmost “ON” pixel and make row pixels in between them “ON” • Can detect convex regions but fails at concavities Composition-Guided Image Acquisition

16. Contribution #1 Main Subject Detection Results Supplementary image Step 2: Strong edge detection Step 3a: Gradient of strong edges Step 1: Edge map Step 3b: Gradient vector flow field Step 3c: Initial contour Step 3d: Contour after 5 iterations (not mandatory) Main subject mask Composition-Guided Image Acquisition

17. Contribution #1 Implementation Complexity • Per-pixel complexity for algorithm[Banerjee & Evans; 2003-04] • Multi-level wavelet based[Wang, Lee, Gray, Wiederhold; 1999-2001] • Variance of multi-level wavelet coefficients: ~2X increase • k-means clustering: 2(image size)(no. of iterations)X increase • Iterative classification from variance maps[Won et al.; 2002] • Iterative maximum a posteriori segmentation: ~3X increase • Watershed refinement: 6 passes per pixel Composition-Guided Image Acquisition

18. Wavelet-based Variance maps [Wang et al.; 1999-2001] [Won, Pyan & Gray; 2002] Contribution #1 Comparison With Previous Methods Proposed algorithm Original image [Banerjee & Evans; 2003-4] Composition-Guided Image Acquisition

19. Contribution #1 Limitations • Frequency-based features not applicable if • Main subject does not have enough high frequencies • Background not blurry enough • Could incorporate region-based features Example of an image where the proposed algorithm fails to detect the main subject, the flower Composition-Guided Image Acquisition

20. Outline • Introduction • Contributions • Main subject detection • Aesthetic enhancement, given main subject • Reposition main subject to follow rule-of-thirds • Simulate background blur for motion or clarity • Mitigation of mergers with background objects • Conclusions Composition-Guided Image Acquisition

21. Contribution #2 Rule-of-Thirds • Better interaction of main subject with image background • Center of mass of main subject at 1/3 or 2/3 picture width (or height) from the left (or top) edge Main subject in center of picture Main subject follows rule-of-thirds Outdoor setting; the flower is main subject Composition-Guided Image Acquisition

22. Contribution #2 Rule-of-Thirds Algorithm • Compute center-of-mass of main subject • 2 multiply-accumulates, 1 memory read per pixel • 1 division per image • Locate closest one-third corner • 8 compares per image (4 comparisons of (x,y) points) • Shift picture so center-of-mass falls at desired corner • Mirror undefined boundary pixels • Best case: no change to image • Worst case: 1/3 rows/columns need to be shifted • Average (main subject in middle): shift 1/6 rows/columns • 0 to 2 memory accesses per pixel Composition-Guided Image Acquisition

23. Contribution #2 Ideal Background Blur Example Background blur emphasizes main subject, the shell, and aids in constrained image communication Indoor setting; no humans in picture Composition-Guided Image Acquisition

24. Contribution #2 Simulated Background Blur • Possible camera blurs • Background blur: shutter aperture • Linear blur: subject or camera motion • Radial blur: camera rotation • Zoom: change in zoom • Digital alternatives • Original image masked with detected main subject mask • Region of interest filtering performed on non-masked pixels • Complexity: 9 multiply-accumulates and 4 memory accesses per pixel for convolution with symmetric 3x3 filter Composition-Guided Image Acquisition

25. Contribution #2 Results (1) Supplementary image with main subject(s) in focus Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur Outdoor setting; human main subject Composition-Guided Image Acquisition

26. Contribution #2 Results (2) Supplementary image with main subject(s) in focus Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur Outdoor setting; human main subject Composition-Guided Image Acquisition

27. Contribution #2 Results (3) Supplementary image with main subject(s) in focus Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur Indoor setting; no human subjects Composition-Guided Image Acquisition

28. Outline • Introduction • Contributions • Main subject detection • Aesthetic enhancement, given main subject • Mitigation of mergers with background objects • Framework for background analysis and merger detection • Low-complexity one-pass algorithm for merger mitigation • Conclusions Composition-Guided Image Acquisition

29. Contribution #3 Ideal Merger Mitigation Example Unwanted mergers avoided Background bar merges with gymnast’s hand Composition-Guided Image Acquisition

30. Contribution #3 Mitigation of Mergers: Overview • Goal: Identify background objects merging with main subject • In-focus background object • Connected to main subject mask • Large area relative to image size • Merger detection • Color segmentation based on hue • Identify distracting background object based on distance to main subject and frequency content • Blur merging background objects to induce a sense of distance Merging background objects: trees and bush over right shoulder Composition-Guided Image Acquisition

31. Contribution #3 Segmentation of Background Objects • Hues above histogram average are dominant hues • Background is a mixture of dominant hues • Thresholds: average of two consecutive dominant hues Histogram of background hues and identified objects Background hues Thresholds = {87, 151} Composition-Guided Image Acquisition

32. Contribution #3 Merger Object Detection • Define Frequency Inverse Distance Measure for each disjoint background object Oi • Decreases with nearest distance (di) from main subject • Increases with high spatial frequency coefficients (ωiH) • Merged object: Object with highest transform value Composition-Guided Image Acquisition

33. Contribution #3 Measure Selection • Linear, division, and exponential forms to combine • High frequencies computed with residual in Gaussian pyramid decomposition • Euclidean distance measured from main subject mask Composition-Guided Image Acquisition

34. Contribution #3 Merger Mitigation Results Background tree and bush merging with main subject Blurred tree and bush appear to be farther away High frequency and inv. distance values for background Composition-Guided Image Acquisition

35. Contribution #3 Per-pixel Implementation Complexity For comparison, JPEG compression takes ~60 operations/pixel Composition-Guided Image Acquisition

36. Auto-focus filter Open shutter for blur System Prototype Scene Original color image Color Gaussian pyramid Transform coefficients Inverse distance transform Supplementary image Compute intensity Grayscale image Grayscale image Background segmentation X 3x3 Highpass filter Detect sharper edges Close boundary Intensity Gaussian pyramid Measure how close rule-of-thirds followed Binary main subject mask Detect merging object Automate rule-of-thirds Simulate background blur Reconstruct color pyramid Generated picture with rule-of-thirds Generated picture with blur Merger mitigated picture Composition-Guided Image Acquisition

37. Conclusion • Contributions • Combined optical/digital image acquisition • Provide online feedback to amateur photographers • Low-complexity one-pass method for main subject detection • Rule-of-thirds: placement of the main subject on the canvas • Simulated background blur: motion and depth-of-field • Mitigation of mergers with background objects • Deliverables • Prototype development for digital still image acquisition • Copies of MATLAB code, slides, and papers, available at http://www.ece.utexas.edu/~bevans/projects/dsc/index.html Composition-Guided Image Acquisition

38. Future Work • Automate other photographic composition rules • Best zoom • Available frames, lines of interest, best angle, balanced picture • Extension for video acquisition • Frame-by-frame basis • Compressed domain • Digital image stabilization: Subject mask as feature • Potential research impact: Video cameras, Surveillance, Image/video retrieval, Constrained image/video communication, Main subject detection for specific applications Composition-Guided Image Acquisition