1 / 22

A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras

A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras. Yuxiao Hu Hao Tang. Problem Statement. Collect multi-view face video with expressions; Potential researches Co-articulation of facial expression and lip movement:

connor-finch
Download Presentation

A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras Yuxiao Hu Hao Tang

  2. Problem Statement • Collect multi-view face video with expressions; • Potential researches • Co-articulation of facial expression and lip movement: • Non-frontal view audio/visual speech recognition-lip reading

  3. Relevant Works • Static 2D face databases: FERET, CMU PIE, ORL, Yale Database, UMIST, etc • Static 3D face databases: 3D-RMA, GavabDB, YorkDB, XM2VTS database, FRGC database,etc • 3D Dynamic face databases: • CMU FIA, no markers, no audio • Intel Research China Database, not synchronized

  4. Highlights TotalSolution: Both hardware and software MultiView+Synchronization+RealTime Flexibility: Flexibly extended from 2 cameras to 5 cameras; Supplementary Tools: camera calibration, color space conversion, 2D facial feature tracking 3D face shape recovery

  5. Physical Setup FoamHead

  6. System Diagram Camera Calibration Video Data Capture Color De-mosaicing Facial Feature Tracking 3D Shape Reconstruction

  7. Synchronization-Hardware Configuration DragonFly Camera

  8. Synchronization-Software Implementation … Buffers … N Buffer Overrun? Buffer Overrun? Re-sync Y Y Y Time Stamp Matched? N AVI Y Compression

  9. Offline Color De-mosaic Reconstructed RGB color image Raw Data: Color represented in Sparse (Stippled) Pattern Reconstructed RGB color video Raw Data

  10. Camera Calibration • Find the intrinsic and extrinsic parameters • Use Camera Calibration Toolbox for Matlab • Two-step procedure • Find projection matrix using Direct Linear Transformation • Use as initialization for nonlinear minimization of mean squared re-projection error

  11. Camera Calibration (cont’ed) Camera 1 Camera 2 Camera 1 Camera 2

  12. Camera Calibration (cont’ed)

  13. Camera Calibration (cont’ed) Camera 1

  14. Camera Calibration (cont’ed) Average re-projection error < 0.2 pixels (0.16279,0.13482) and (0.16439,0.12685) Camera 1 Camera 2

  15. Facial Marker Tracking • Simple but effective tracking algorithm

  16. Facial Marker Tracking (cont’ed) • Statistical marker collocation model

  17. Facial Marker Tracking (cont’ed)

  18. Facial Marker Tracking (cont’ed)

  19. 3D Reconstruction: Stereo Triangulation • A bit of theory

  20. 3D Reconstruction: Stereo Triangulation (cont’ed)

  21. Deliveries • The data acquisition system of camera array • Tools for Color De-mosaicing • The calibration data and tools • Some sample data result • 3D ground truth data and labeling tool • Technical Report

  22. Outline (4Ws+2Hs) • Why (do we do this?) • Who (has done the related work?) • What (we proposed to do?) • How (did we achieve our goal?) • Why (we need to do so?) • How (we evaluate our work?)

More Related