Unsupervised Automation of Photographic Composition Rules

1. Serene Banerjee and Brian L. Evans http://www.ece.utexas.edu/~bevans/projects/dsc/index.html Unsupervised Automation of Photographic Composition Rules Computer Engineering Area Dept. of Electrical and Computer Engineering The University of Texas at Austin

2. Main subjectcropped Too muchbackground Motivation • Problem: Amateur photographers often takelow-quality pictures with digital still cameras • Personal use • Professionals who need to document (e.g.. realtors and architects) • Goal: Automate photographic composition rules and find alternatives to the picture being acquired • Analyze scene, including detection of main subject • Develop algorithms to automate rules Automation of Composition Rules

3. Blur background for action pictures Following rule-of-thirds Solution • Solution #1: Automatically detect main subject • Independent of indoor/outdoor setting or scene • Low implementation complexity, fixed-point computation • Solution #2: Automate a few photograph composition rules • Rule of thirds for placing the main subject • Simulated background blur for motion pictures or depth-of-field Automation of Composition Rules

4. 1: Main subject 2: Lenses 3: CCD 4: Imaging device 5: Raw data Digital Still Cameras • Converts optical image to electric signal using charge coupled device (CCD) • Software control • Zoom • Focus, e.g. auto-focus filter • Shutter aperture and speed • White balance: Corrects color distortions • Settings that can be controlled (with added hardware) • Camera angle • Aspect ratio: Landscape or portrait mode • Produces JPEG compressed images Automation of Composition Rules

5. Main Subject Detection Methods • Two differently focused photographs [Aizawa, Kodama, Kubota; 1999-2002] • One has foreground in focus, and other has background in focus • Significant delay involved in changing the focus • Bayes nets based training [Luo, Etz, Singhal, Gray; 2000-2001] • Bayesian network trained on example set and tested later • Training time involved: suited for offline applications • Multi-level wavelet coefficients [Wang, Lee, Gray, Wiederhold; 1999-2001] • Expensive to compute and analyze wavelet coefficients • Iterative classification from variance maps [Won, Pyan, Gray; 2002] • Optimal solution from variance maps and refinement with watershed • Suitable for offline applications involving iterative passes over image Automation of Composition Rules

6. Proposed Main Subject Detection • User starts image acquisition • Focus main subject using auto-focus filter • Partially blur background and acquire resulting picture • Open shutter aperture (by lowering f-stop) which takes about 1 s • Foreground edges stronger than background edges • While acquiring user-intended picture, process blurred background picture to detect main subject • Generate edge map (subtract original image from sharpened image) • Apply edge detector (Canny edge detector performs well) • Close boundary (e.g. gradient vector flow or proposed approximation) Automation of Composition Rules

7. k + Generate Edge Map • Symmetric 3 x 3 sharpening filter • For integer a and b, coefficients are • Integer when dropping 1/(1 + a) term • Fractional when -1 – 2ab < 1and 1/(1 + a) is power-of-two • Generate edge map • Subtract original image from sharpened image • Main subject region now has sharper edges + fsmooth(x,y) - g(x,y) fsharp(x,y) Smoothing filter + f(x,y) + + Sharpening filter Model for an image sharpening filter Automation of Composition Rules

8. Boundary Closure • Gradient vector flow method [Xu, Yezzi, Prince; 1998-2001] • Compute gradient • Outer boundary of detected sharp edges is initial contour • Change shape of initial contour, depending on gradient • Shape converges in approximately 5 iterations • Disadvantage: computationally and memory intensive • Approximate lower complexity method • Select leftmost & rightmost ON pixel and make row between them ON • Can detect convex regions but fails at concavities Automation of Composition Rules

9. Automation of Rule-of-Thirds • Goal: Center of mass of the main subject at 1/3 or 2/3 of the picture width (or height) from the left (or top) edge • Solution: • For n-D, define function that attains minimum when center of mass placed as desired and increases otherwise • Shift picture so that minimum is attained • Implementation: • For 2-D, sum of Euclidean distance from the 4 points • Measure which of the 4 points is closest to the current position of the center of mass • Shift picture so that the rule-of-thirds is followed Automation of Composition Rules

10. Simulated Background Blurring • Goal: Filter the image background and add artistic effects keeping the main subject intact • Solution: • Original image masked with detected main subject mask • Region of interest filtering performed on masked image • Possible motion blurs • Linear blur: subject or camera motion • Radial blur: camera rotation • Zoom: change in zoom • Applications • Enhance sense of motion where the main subject is moving • Digitally decrease the depth-of-field of the photograph Automation of Composition Rules

11. Automate rule-of-thirds Filter to generate edge map Generated Picture with Rule-of-Thirds Measure how close rule-of-thirds followed Auto-focus filter Original Image Detect sharper edges Lower f-stop for blur Binary Main Subject Mask Simulate background blur Generated Picture with Blur Close boundary Proposed Module Automation of Composition Rules

12. Implementation Complexity • Number of computations and memory accesses per pixel • Main subject detection: convolution with symmetric 3x3 filter, edge detection, approximate boundary closure • Rule-of-thirds: center of mass (1 division, 4 compares) , shift pixels • Background blurring: convolution with symmetric 3x3 filter • Digital still cameras use ~160 digital signal processor instruction cycles per pixel Automation of Composition Rules

13. Results (1) Detected main subject mask Original image with main subject(s) in focus Detected strong edges with proposed algorithm Rule-of-Thirds: Main subject repositioned Simulated background blur Automation of Composition Rules

14. Results (2) Detected main subject mask Original image with main subject(s) in focus Detected strong edges with proposed algorithm Rule-of-Thirds: Main subject repositioned Simulated background blur Automation of Composition Rules

15. Results (3) Detected main subject mask Original image with main subject(s) in focus Detected strong edges with proposed algorithm Rule-of-Thirds: Main subject repositioned Simulated background blur Automation of Composition Rules

16. Conclusion • Developed automated low-complexity one-pass method for main subject detection in digital still cameras • Processes picture taken with blurred background • All calculations in fixed-point arithmetic • Automates selected photographic composition rules • Rule-of-thirds: Placement of the main subject on the canvas • Simulated background blur: motion and depth-of-field • Applications: digital still cameras, surveillance, constrained image compression, and transmission and display • Copies of MATLAB code, poster, and paper, available at http://www.ece.utexas.edu/~bevans/projects/dsc/index.html Automation of Composition Rules