1 / 26

Dezhen Song Department of Computer Science and Engineering Texas A&M University

CPSC 643. Aligning Windows of Live Video from an Imprecise Pan-Tilt-Zoom Robotic Camera into a Remote Panoramic Display. Dezhen Song Department of Computer Science and Engineering Texas A&M University. Supported in part by. Network PTZ Robotic Camera for Nature Observation.

nirav
Download Presentation

Dezhen Song Department of Computer Science and Engineering Texas A&M University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CPSC 643 Aligning Windows of Live Video from an Imprecise Pan-Tilt-Zoom Robotic Camera into a Remote Panoramic Display Dezhen Song Department of Computer Science and Engineering Texas A&M University Supported in part by

  2. Network PTZ Robotic Camera for Nature Observation • Panosonic HCM 280 • PTZ Robotic Camera: • 350° Pan, 120° Tilt, 42x Zoom • 200° per second servo speed • Network Video Camera: • Built-in streaming server • 640x480 pixels video • >30 frames per second • Low power consumption: <5 Watt • Affordable price: $ 1.2 K

  3. Panorama Tilt Pan Frame sequence Tilt Updated Part in Panorama Panorama Live frame sequence Time Real Time Panoramic Video

  4. Related Work • Multiple fixed cameras • [Swaminathan and Nayar 2000] • [Tan et al. 2004] • [Foote et al. 2000, 2001] • Single wide angle camera • [Baker and Nayar 1999] • [Nayar 1997] • [Xing and Turkowski 1997]

  5. Related Work: Image Alignment • Direct Method • Use pixel intensity value • Sensitive to luminance change • Need good guess for initial parameters input • Existing work • [Shum and Szeliski 1997] [Szeliski 1994, 1996] • [Coorg and Teller 2000] [Kang and Weiss 1997] • Frequency Domain Registration • Existing work • [Castro and Morandi 1987] [Reddy and Chatterji 1996]

  6. Related Work: Image Alignment • Feature-based Image Registration • Use feature points: Harris corner point, SIFT • Robust to luminance change • Faster than direct method • Existing work • [Torr and Zisserman 1997] [Brown and Lowe 2003] • [Zoghlami et al. 1997] [Hu et al. 2001] [Cho et al. 2003] • [Kanazawa and Kanatani 2002] [Zhang et al. 2002]

  7. Comparison: Panoramic Video

  8. Assumptions • Pan-tilt camera with a fixed base • Known intrinsic camera parameters • Calibrated camera before deployment • Inaccurate pan-tilt readings • May deteriorate over time • Standard video camera with HFOV ≤ 45o

  9. Review: Perspective Projection [Tsai86, 87] Intrinsic Parameters Extrinsic Parameters

  10. Problem Definition Re-projection: Project image B onto image A plane: Image alignment:

  11. Excessive Computation in Image Alignment Speed slow down caused by coupling re-projection and SSD: • Extensive float point computation • Coupled with Sum of Squared Difference (SSD) operation • A naive search takes O(km) re-projection operations • k: number of candidate pan/tilt pairs over feasible solution set. • m: number of overlapping pixels • Proposed solution: decouple re-projection and SSD • Spherical re-projection • Cell-based Alignment • Constant time alignment

  12. CY O v Image plane f t p CX CZ u Spherical Projection • Project image onto a spherical surface • Image =(p, t) on local spherical coordinate system { }

  13. 400 0.8 200 v 0 0.4 -200 t 0 -400 -0.4 -1000 -600 -200 200 u -0.8 -1.5 -1 -0.5 0 0.5 p Distortion under Re-projection Planar Spherical Two poses have 30o pan difference with the same 30o tilt value

  14. 0.8 0.4 t 0 -0.4 -0.8 -1.5 -1 -0.5 0 0.5 p Invariant under Spherical Re-projection

  15. Re-projection after Spherical Projection Define conversion between camera coordinate system and local spherical coordinate system Re-projection function between two local spherical coordinate system

  16. Lemma 1: If the spherical cell is small , define point and its corresponding point we have • Rc is a 2x2 rotation matrix • Cell distortion under re-projection is negligible.

  17. Introduce coefficient matrix H Radius f remains same Proof for Lemma 1:

  18. HFOV≤45o and VFOV ≤34o  0.956 ≤ cos(t) ≤ 1 Dropping cos(t) introduces ≤5% distortion for 20x20 cell Continue: Proof for Lemma 1 ∆f=0  substitute [m13, m23, m33]T

  19. Continue: Proof for Lemma 1

  20. Lemma 2: Rotation angle Θ of Rc can be approximated by where αis the dot product of Z axis of {CA} and {CB}

  21. Cj ε(Cj ) Algorithm

  22. Select kc cells from the overlapping region in O(1) • Sphere projection O(1) • Feature detection in the cell and searching regions O(1) • For each(δpB, δtB) O(1) • For each cell O(1) • Compute Cj • Compute , j=1, …, kc • Compute SSD between and • End For • Report sum of SSD across all cells • End For • Output solution with the minimum SSD Cell based Image Alignment

  23. Experiment and Results Speed test: • 881 milliseconds to align 21 320x240 images • 4 seconds for Autostitch program on same data set • Up to 25fps on a laptop PC

  24. Thank You! • Align video images from a camera that only differ in pan and tilt settings into a panorama at 25 frames per second. • Alignment is performed on a spherical surface to avoid excessive distortion caused by homographic transformation. • A constant time algorithm pre-rotates small pre-sampled squared patches on spherical surface for matching. • Experiments show that the alignment speed is 4.5x faster than the best method available. live video window

More Related