Example-Based Composite Sketching of Human Portraits

1. Example-Based Composite Sketching of Human Portraits Hong Chen1,2, Ziqiang Liu1,2, Yingqing Xu1, Chuck Rose3, Heung-Yeung Shum1, David Salesin4,5 NPAR 2004 1 Microsoft Research , Asia 2 University of California, Los Angeles 3 Microsoft Corporation 4 University of Washington 5 Microsoft Research

2. Result

3. Outline • Introduction • Related work • System framework • Composing a face • Composing hair • Examples • Conclusions and future work

4. Introduction • An interactive system for generating human portrait sketches • Input • A human face image • Output • A sketch that exhibits the drawing style of a set of training examples provided by an artist • Style of Japanese cartooning

5. Introduction (Cont.) • Propose a composite sketching approach • Decompose the data into components that are structurally related to each other. • As the eyes or mouth • Propose a system which combines two separate but similar subsystems. • The face subsystem • The Hair subsystem

6. Introduction (Cont.) • After these have been independently processed, they are carefully recomposed to obtain the final result.

7. Related Work • NPR and digital arts • Used to depict facial images with an artistic style • Digital facial engraving [Ostromoukhov, SIGGRAPH 99] and Caricature generation [Brennan, 85] • Emulate traditional artist tools to assist users in drawing pictures with a certain style • Rare attempt to generate digital paintings by learning from artists

8. System Framework • Goal • Create a system that could leverage the artist’s skill with a high degree of automation. • A training set • The gamut of east Asian female faces • Divide the portrait system • The face subsystem • The hair subsystem

9. System Framework (Cont.) • The face subsystem • Segment into sub-problems for each of the natural facial features. • i.e. eyes, mouth • Global and Local models • The hair subsystem • Be handled carefully with a structural model and a detailed model. • These sub-problems are tackled independently

10. the color face image an input image the corresponding sketch matches the style of the training examples Composing a Face • Objective • Construct a model to take an input image I and generate a sketch I’ that matches the style of the training examples. • Training Set: • Model:

11. Composing a Face (Cont.) • Split into two layers • Global : • capture how the artist places each face element in the sketch image. • Local : • mimic how the artist draws each independent element locally.

12. Composing a Face (Cont.)

13. Drawing the Facial Component with the Local Model • A human face is decomposed semantically into 6 local components. • Left & right eyebrows, left & right eyes, a nose, and a mouth

14. Drawing the Facial Component with the Local Model (Cont.) • Extract the accurate shape and associated texture information using an Active Shape Model. • Determine to which prototype the component belongs • Build a different classifier for each type of component to cluster the input components into the appropriate prototype. • KNN interpolation

15. K Nearest Neighbor • The goal is to find a class label for the unknown example xu • - Euclidean distance • - K=5

16. Composing the Face Using the Global Model • The global model: • Use to arrange each face element on a canvas. • For drawing facial caricatures • Relationship of elements to others of their ownkind. • Relationship of elements to their surrounding and adjacent elements.

17. Composing the Face Using the Global Model(Cont.) • For the representation of Ig

18. Composing the Face Using the Global Model (Cont.) • By not tying these relations to fixed value • The model can adjust the size of the feature as the overall size of the head is changed. • Generate Ig • Use Active Shape Model • 87 control points • Determine the placement of the face elements on the cartoon canvas.

19. The Placement of the Face Elements on the Cartoon Canvas • Each element needs five parameters {(tx, ty), (sx, sy),θ}

20. Composing Hair • Hair cannot be handled in the same way as the face • Hair has many styles. • Is not structured in the same regular way that faces are • A single unit • Render using long stroke • There is no clear correspondence between regions of two different hairstyles.

21. Hair System Flow

22. Hair System Flow (Cont.) • Structural components • Coarsely segment the hair into fivesegments • Each indicates important global information about the hair. • The detail model • Add uniqueness and expression to a portrait.

23. Hair Composite Model • The global hair structure or impression is more important than the detail.

24. Extracting the Image Features for the Hair • Determine the image features of the hair • Match against the database • An estimated alpha mask • Hair strand orientation fields

25. Fitting Structural Components • For an input image, finding the best training data. • Classify input image into the correct style. • Find the best training example.

26. Fitting Structural Components(Cont.) • Deform hair components to a standard shape using a multi-level freeform deformation warping algorithm. • Hair orientation vector • G = [gx1,gy1, gx2,gy2 ,… , gxn,gyn] • Alpha value • α = [α1,α2 ,…,αn] • E(H1,H2) = ||G1-G2||+w||α1-α2||

27. Fitting Structural Components(Cont.) • Shape ~ a set of corresponding key points • S = [x1,y1,…,xm,ym] • Find the best matched training example • Minimize the distance combining the appearance and the shape distance

28. Fitting Detail Components • Different kinds of detail require slightly different approaches. • Boundary and bangs

29. Fitting Detail Components (Cont.) • Boundary details • The alpha value and orientations for these patterns are quite different.

30. Fitting Detail Components (Cont.) • Bang detail components • Used to detect the bang • Segment out the bang regions in the alpha mask • The orientation field can be inspected. • Determine the length of the bang line

31. Synthesizing the Hair Sketch • The stroke in the training samples are all divided into two classes. • Boundary strokes and streamline strokes • Link points • The points in the strokes crossing the boundary of a structural component • Face contour exaggeration • Adjust the each part of the inner hair boundary according to the corresponding face contour

32. Composing the Structural Components • Warp to the target coordinates. • Match link points those in the same class. • Adjust to the average position and link the corresponding strokes. • Remove unmatched streamline strokes.

33. Add the Detail Strokes • Detail components are connected to strokes generated by the component match. • Warp to the target coordinates from the global phase

34. Examples (b) Result of local model. (c) Result of local model plus global model. (d) Result without local model and global model.

35. Examples (Cont.) • Compare the effect of adding detail strokes. (b) Result of composing structural components. (c) Composing with detail components.

36. Examples (Cont.) • Combine the face and the hair • Neck, shoulder, and clothing are chosen from a set of templates supplied by the artist.

37. Examples (Cont.)

38. Conclusions • Adapting a global/local hybrid was an effective approach for generating face portrait sketches. • Application • Create a virtual person for cartoon style online games or chat environments • Create sketched portraits for places where a sketch is preferred over a photograph

39. Future Work • Limitations and future work • Add to the training set • Encompass faces of many racial backgrounds • Render male images • A third subsystem to handle • Aging, injury, spectacles, and jewelry • A face in profile