1 / 47

MURI review meeting 09/21/2004

Dynamic Scene Modeling. Christian Frueh Avideh Zakhor. Video and Image Processing Lab University of California, Berkeley. MURI review meeting 09/21/2004. Dynamic Scene Modeling. 4D Capture of a dynamic scene 3D geometry/depth + time Applications: Battlefield scenario

ednagreen
Download Presentation

MURI review meeting 09/21/2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Dynamic Scene Modeling Christian Frueh Avideh Zakhor Video and Image Processing Lab University of California, Berkeley MURI review meeting 09/21/2004

  2. Dynamic Scene Modeling • 4D Capture of a dynamic scene • 3D geometry/depth + time • Applications: • Battlefield scenario • Event analysis, modeling, and visualization • Action classification and recognition

  3. Battlefield Scenario

  4. Battlefield Scenario

  5. Objectives • Minimal interference with objects in the scene • Especially visible domain  humans • Capture 3D depth as well as intensity • Capture, model, and reconstruct a time-varying scene at video-rate • Off-the-shelf components • Low cost: e.g. camcorders, halogen lamp • Experiments: • Indoors • Offline processing

  6. Proposed Acquisition Setup IR camera VIS-light camera rotating mirror IR line laser vertical IR line projector

  7. Proposed Approach • Active system • Structured infrared light (IR) for depth estimation invisible to human eye • Project static pattern of vertical IR stripes • Sweep horizontal IR line vertically • Capture with camcorder + IR filter • Depth via triangulation • Synchronized video camera for texture acquisition • 3D arena equipped with stationary cameras/projectors

  8. Prototype System Reference object for H-line Digital camcorder with IR-filter Sync electronic VIS-light camera rotating mirror PC IR line laser Roast with vertical slices Halogen lamp with IR-filter

  9. Video camera Video sync generator H-laser, polygonal mirror Camcorder with IR filter Control PC Halogen lamp with IR filter Stripe pattern forV-Lines Prototype System

  10. Use multiple parallel lines Depth From Structured Light Principle: Triangulation Light plane object laser ray baseline camera obtain depth along 1 line camera baseline How can we get dense depth?

  11. Depth From Structured Light Problem: How to identify/distinguish individual lines?

  12. Track V-lines in frame Identify V-lines Via the Horizontal Line Sweep horizontal laser line across scene, e.g. with 1Hz Only one horizontal line easy to identify Rotating mirror t0 Depth along this line can be computed Depth at intersections of horizontal (H) and vertical (V) lines is known line laser • 2 points + vertical -> V-plane equation -> depth Intra-Frame Tracking Problem: Depth only along some V-lines

  13. Track V-lines across Frames • H-line sweeps across scene every V-line intersects with H-line in some frame • Track V-lines across frames • For each V-line, search for identified V-lines in previous/future frame around same location • Use V-line plane equation from previous / future frame • Inter-Frame Tracking 8 frames later

  14. Captured Video Streams IR video stream VIS video stream Frame rate: 30 Hz (NTSC) Frame rate: 10 Hz Synchronized with IR video stream

  15. Overview of Processing Steps IR video stream VIS video stream Foreground identification V-Line detection H-Line detection & Foreground identification Intra-Frame Tracking Depth Inter/Extra-polation Inter-Frame Tracking VIS Projection Dense Depth Frames

  16. Find H-line spot on reference object H-Line Detection (1) How to determine current H-light plane equation?

  17. Problem: Some wrinkles appear like H-lines H-Line Detection (2) Apply horizontal edge filter to IR-frame

  18. H-Line Detection (3) 372 371 H-line is at different location in every frame Wrinkles are roughly at the same location across 2 frames: limited motion Solution: H-feature is only a H-line, if location changes

  19. H-Line Detection (4) Before After Before

  20. H-Line Detection: Result

  21. V-Line Detection Start with infrared image

  22. V-Line Detection (2) Apply vertical edge filter

  23. V-Line Detection (3) Thin out vertical edges

  24. V-Line Detection (4) Track vertical edges

  25. Clip V-lines To “Active Area” Background differencing

  26. Clip V-lines To “Active Area” (2) Difference thresholding

  27. Clip V-lines To “Active Area” (3) Region defragmentation via segmentation & majority voting => IR-active regions

  28. Clip V-lines To “Active Area” (4) Clipping of V-lines to IR-active regions

  29. Clip V-lines To “Active Area”: Result V-lines

  30. Depth Estimation for V-lines • Search for intersection point with H-line • For every point on V-line, search for H-line point in proximity • Choose closest H-line point for light plane computation • Intra-frame tracking: • Track the V-line in the image and compute depth for each of its pixels

  31. Intra-Frame Tracking Depth from intersection with H-lines

  32. Inter-Frame Tracking t0 • Object moves forward  lines shift right • Object moves backwards  lines shift left • If V-line pattern on object shifts less than half the line spacing, V-lines can be tracked across frames moving object t1 t2 vertical laser plane camera • For each unidentified V-line, search within half the line spacing for a identified V-line in the previous or subsequent frame • If found, use light plane equation

  33. Inter-Frame Tracking: Forward Direction +Depth inferred from previous V-lines

  34. Inter-Frame Tracking: Fwd + Bckwd +Depth inferred from future V-lines

  35. Inter-Frame Tracking Inter-frame forward and backwards Intra-frame only Inter-frame forwards only

  36. Resulting Depth for V-lines

  37. Dense Depth From Sparse V-Lines • Depth lines sparse • No values between lines • Areas without depth information • Silhouette not accurate • Ideally: Depth value for every pixel in VIS image • Depth frame  VIS frame Project depth lines into visible image Accurate Silhouette from VIS image

  38. Projected V-Lines onto VIS Frames Use depth information to project V-lines into visible domain

  39. Fgnd/Bckgnd Separation in VIS-Frames Background subtraction followed by morphological operations/segmention

  40. Movies VIS-active areas Projected V-lines

  41. Dense Depth Interpolate/extrapolate to dense depth within marked foreground area

  42. Dense Depth Depth along V-lines Dense depth

  43. Results Depth video Visible video

  44. Overview of Processing Steps IR video stream VIS video stream Foreground identification V-Line detection H-Line detection & Foreground identification Intra-Frame Tracking Depth Inter/Extra-polation Inter-Frame Tracking VIS Projection Dense Depth Frames

  45. System Parameters and Trade-offs Camera: • Ideally: shutter time short to avoid motion blur • Limit: Sensitivity • Noise • Brightness, stripe contrast • Ideally: fast sweep, for small delay of V-Line identification • Limit: camera shutter time • Motion blurring  wide H-line H-line:

  46. System Parameters and Trade-offs V-lines: • Ideally: Many V-lines, for dense depth reconstruction • Limits: • (a) camera resolution  intra-frame tracking • (b) maximum object velocity inter-frame tracking • Ideally: Monochromatic IR-light, narrow bandwidth to reduce noise light • Limits: • cheap halogen lamp as light source • camera sensitivity

  47. Future Work • Extension to outdoors • Multiple capturing stations – scene from all sides • Potential interference of projected patterns • Extension to portable system • Improvements in processing • Consistency • Object constraints • Code optimization for speed-up • Rendering • Dynamic VRML model? • Custom renderer for interactive exploration?

More Related