1 / 36

A Study on the 3D Facial Feature Extraction and Motion

A Study on the 3D Facial Feature Extraction and Motion Recovery using Multi-modal Information. Sang Hoon Kim, Hankyong National Univ., Korea, south wind1104@hanmail.net http://auto.hankyong.ac.kr/kimsh. Experience.

mindy
Download Presentation

A Study on the 3D Facial Feature Extraction and Motion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Study on the 3D Facial Feature Extraction and Motion Recovery using Multi-modal Information Sang Hoon Kim, Hankyong National Univ., Korea, south wind1104@hanmail.net http://auto.hankyong.ac.kr/kimsh

  2. Experience 1989-1994 Researcher, Institute of LG Semiconductor co.,Ltd 1994-1999 Research Assistant, Dept. of Electronics, Korea University 1999-2001 Senior Research Scientist, Imaging and Media Research Center, KIST(Korea Institute of Science and Technology) 1999- present Assistant professor in Dept. of Information and Control, Hankyong National University, Kyonggi-do, Korea Research Interests Face detection and tracking in real time Pose estimation Multi-modal fusion for object detection and recognition

  3. Problems are face detection • How to detect facial area exactly from complex background • in variable conditions. • How to define and extract facial features location(and how many • features are proper to do next step) • How to recover 3D facial features and global motion from • the 2D information. • How to combine the whole results to synchronize a synthetic • face model with a real face. global motion recovery and application Evaluation method is important !! to prove the accuracy of 3D recovered information

  4. Categorization of methods for face detection • Knowledge-based top-down methods • Bottom-up feature based methods (this work classified here from the survey) • Template matching • Appearance-based methods “ Detecting Faces in Images: A Survey ” IEEE Trans. on PAMI, vol.24, no.1, Jan 2002 Ming-Hsuan Yang, David J. Kriegman, Narendra Ahuja

  5. Transform 3D information into FAP of MPEG4 and synchronize a generic model with a real face facial features and motion extraction using paraperspective camera model Facial features extraction using moving color information Overall Blockdiagram

  6. Generalized Skin Color Distribution(GSCD) The skin color distribution in the normalized (r,g) domain can be approximated by the 2D Gaussian distribution GSCD in (r,g) domain

  7. GSCD color transform 1. Color Transform defined as Z(x,y) = GSCD ( r(x,y) , g(x,y) ) 2. Result intensity value of enhanced facial regions suggests the possibility of being facial color GSCD input color image Ic(x,y) gray level image Z(x,y)

  8. LoG and MPC Disparity Map Left camera Image grab Enhanced image Disparity map Right camera LoG MPC stereo matching Disparity D(x,y) is Vth : predefined threshold value

  9. MPC similarity measure disparity map results with Random-dot stereo image (a) left image (b) right image (c) SAD (d) NCC (e) MPC similarity measures matching ratio comparison when noises are increased ( 0% ---> 20% )

  10. MPC similarity measure

  11. Object 1 frequency Object 2 Background disparity threshold 2Disparity Histogram(DH) 1. Definition : occurrence frequency of each disparity value obtained from MPC disparity map 2. Meaning : the locations and number of objects

  12. Range Segmentation 1. The outline of the DH curve is smoothed with average filtering 2. Region having the smallest disparity value is defined as background (Although it has skin-colored components, they can be regarded as noises or useless data) 3. Regions having frequency values less than a specified threshold are assumed to have no object and merged to background region 4. Regions having continuous disparity values greater than a specified threshold are defined as objects with same label

  13. Moving Color Enhancement 1 This idea based on the fact that regions detected as faces only using skin color are unstable and have variable skin color space due to the lighting condition, so adding the motion information to the skin color enhanced region would be effective so as to increase the probability of faces So given an GSCD transformed image, region that hasa low gray value, which means low probability of face, should be detected as a face only when it haslarge motion, while region that has high gray level, which means high probability of face, can be detected as a face even when it has small motion.

  14. Moving Color Enhancement 2 The SWUPC(Sigmoid function Weighted Unmatched Pixel Count) operation emphasizes only the region with a motion in the skin-color enhanced region. where

  15. Sigmoid Function

  16. (e) (d) (a) (c) (b) • (a) present frame (b) previous frame • (c) UPC result (d) WUPC result (e) SWUPC result Moving color enhanced results

  17. Face Detection Results using multi-modal fusion

  18. Test image 1 Step 2 Step 3 Step 4 Step 5 Final detection

  19. Test image 2 Step 2 Step 3 Step 4 Step 5 Final detection

  20. Test image 3 Step 2 Step 3 Step 4 Step 5 Final detection

  21. Test image 4 Step 2 Step 3 Step 4 Step 5 Final detection

  22. Test image 5 Step 2 Step 3 Step 4 Step 5 Final detection

  23. Test image 6 face region detection result using range,skin color and motion information (a)input color image(t=1)(b)input color image(t=2) (c)MPC disparity map(d)range segmented image(g)skin color transformed image (h)swupc transform image (i)final face area

  24. FDP of MPEG4 snhc vs facial feature set for the work 23 74

  25. Present image GSCD Previous image GSCD BWCD Final eyes, eyebrows SWUPC Morphological Result Erosion Dilation Extracted facial features Automatic Facial Features Extraction - eyes, eyebrows

  26. Automatic Facial Features Extraction - eyes, eyebrows

  27. Hypothetical Image Plane Image Plane Focal length, l World Origin paraperspective camera model Paraperspective Camera model  why use? This model can describe the change of object’s depth information Refer to “ A Paraperspective factorization method for shape and motion recovery” Conrad J.Poelman and Takeo Kanade

  28. Basic matrix Final M,S recovered (1) A recovered (2) (3) (8) SVD- M,S separation (4) (7) (6) (5) Sequential SVD Process (Matrix time series)

  29. -100,0,-50 0,0,0 0,100,-50 0,-100,-50 100,0,-50 Pyramid type 3D test object

  30. Multi-view Synthetic Pyramid test images

  31. 20th frame with motion viewed from two cameras(left and right camera) 40th frame with motion viewed from two cameras(left and right camera) Multi-view Synthetic Pyramid test images

  32. Shape Recovery Results using synthetic 3D shape

  33. Motion Recovery Results using synthetic 3D shape

  34. Motion and Shape Recovery Results using synthetic face image synchronized with real face

  35. (front view) (left view) (right view) (upward face) (downward face) Real facial information --> Face Animation Parameter

  36. Conclusions 1. This technique works well when detecting faces using GSCD(Generalized Skin Color Distribution) combined with the Range Segmentation using MPC(Matching Pixel Count) DH(Disparity Histogram) and Object’s moving color information(SWUPC) 2. Automatic facial features extraction(95%) using moving color information(SWUPC, BWCD) and morphological process. 3. Motion and 3D facial features recovery(80%, 208 of 210 frames) using multi-view paraperspective model and factorization(SVD) method 4. The results can be used for various applications

More Related