1 / 28

Single-view Video To Multi-view Video

Single-view Video To Multi-view Video. 台大資工系 黃 昱 翔 , 黃彥翔 指導教授 : 莊永裕老師. Goal. 把 2D 所拍攝的靜物 影片， 經由建構出虛擬 cameras 的方法 ，模擬人類雙眼 視覺的效果，再利用 雙 眼 視差， 經過處理，產生 出 新的 3D 的影片 。. Procedure. Camera Path Tracking & Feature Points Detecting Virtual Camera Path Generating Feature Points Projecting

carson
Download Presentation

Single-view Video To Multi-view Video

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Single-view Video To Multi-view Video 台大資工系 黃昱翔, 黃彥翔 指導教授: 莊永裕老師

  2. Goal • 把2D所拍攝的靜物影片，經由建構出虛擬cameras的方法，模擬人類雙眼視覺的效果，再利用雙眼視差，經過處理，產生出新的3D的影片。

  3. Procedure • Camera Path Tracking & Feature Points Detecting • Virtual Camera Path Generating • Feature Points Projecting • Image Warping(frame by frame) • Combine to a new 3D video

  4. Camera Path Tracking & Feature Points Detecting • Structure from Motion~ • To figure out the camera path 切成一張張tga圖片

  5. Camera Path Tracking & Feature Points Detecting • 經過parse這些output檔 可以得到camera位置以及其座標系、features的3D位置、哪幾個frame存在哪些feature、相機焦距等等資訊等等備用

  6. Virtual Camera Path Generating • 由取出的camera位置往其水平方向各延伸出一單位長的距離建造 virtual cameras

  7. Feature Points Projecting • 再以以下公式將該frame所有feature points 投影到兩個virtual cameras • # [ x' ] = [f/sx0 ppx] [ H0x H0y H0z ] [ 1 0 0 -Cx] [ X ] • # [ y' ] = [0f/syppy] [ V0x V0y V0z ] [ 0 1 0 -Cy] [ Y ] • # [ z' ] = [0 0 1 ] [ Ax Ay Az] [ 0 0 1 -Cz] [ Z ] • # [ 1 ] • 得到兩張新的feature points 圖 以及與原圖的對應關係

  8. Content-Preserving Warps for 3D Video Stabilization Hailin Jin AseemAgarwala Feng Liu Michael Gleicher

  9. Prerequisite • 用影片拍靜物。 • 用 voodoo 找出 camera 路徑和feature位置。 • 在camera 的左右邊各生出虛擬相機，當左右眼。 • 眼睛聚焦看 (stare) • 眼睛平行看 (parallel) • 把物體置中，紀錄 feature 點的位置。 • X Y XR YR XL YL • Feature在原本相機的座標： X , Y • Feature在右邊虛擬相機的座標： XR , YR • Feature在左邊虛擬相機的座標： XL , YL

  10. 238.697126.068155.048126.068322.346126.068 274.984117.66192.123117.66357.846117.66 324.19289.1555244.74889.1555403.63689.1555 440.07470.5983364.15970.5983515.9970.5983 317.98567.7392243.45267.7392392.51867.7392 319.242196.682241.479196.682397.004196.682 302.04359.8351229.32359.8351374.76259.8351 384.629115.924298.994115.924470.265115.924 329.48376.4908252.88176.4908406.08576.4908 148.94853.4643105.44453.4643192.45353.4643 315.921156.882235.337156.882396.506156.882 293.051108.342210.254108.342375.849108.342 161.91247.6016117.53947.6016206.28447.6016 405.702130.359317.013130.359494.392130.359

  11. Procedure • Pre-warping • Cut input image into square grids • Geometry relation between grid and feature point • Find homography • Content-Preserving Warp • Data term • Similar transform term (smooth term) • Optimization • Texture Mapping

  12. Cut input image into square grids 20 grids 640 pixels Grid左上角的座標 (x, y) 480 pixels 32 x 32 Grid 15 grids 共 15 x 20 = 300 個格子，共 (15+1) x (20+1)=336 個格子點。

  13. Geometry relation between grid and feature point • 紀錄feature點在哪個grid裡面。 • 紀錄feature點在該grid裡面的位置關係。 V2 V1 W[0] W[1] L12 P L14 V3 V4 W[3] W[2]

  14. Find Homography • An initial guess • Input: • 原本圖片的feature點座標 • 左眼/右眼的feature點座標 • Output: • A 3x3 Homography matrix

  15. Find Homography • 原本圖片的feature點，想要warp到左眼或右眼相機裡feature點的位置。 238.697126.068 274.984117.66 324.19289.1555 440.07470.5983 317.98567.7392 319.242196.682 302.04359.8351 384.629115.924 329.48376.4908 148.94853.4643 315.921156.882 293.051108.342 161.91247.6016 405.702130.359 155.048126.068 192.123117.66 244.74889.1555 364.15970.5983 243.45267.7392 241.479196.682 229.32359.8351 298.994115.924 252.88176.4908 105.44453.4643 235.337156.882 210.254108.342 117.53947.6016 317.013130.359 homography matrix 3x3 #feature x 2 #feature x 2

  16. Result of Pre-warping

  17. Procedure • Pre-warping • Cut input image into square grids • Geometry relation between grid and feature point • Find homography • Content-Preserving Warp • Data term • Similar transform term (smooth term) • Optimization • Texture Mapping

  18. Data Term • 主要限定格子點位置的term，定義格子點與feature之關係 • 所屬grid的四周頂點座標 (Vk) • 在grid裡面的位置關係 (wkT) • 新的feature在grid裡，位置(wk)應和原本差不多 Pk是左眼或右眼feature點的位置

  19. 填矩陣 (Initial Guess改過的) Feature 座標 20個格子 (21個格子點) 新的 格子點 座標 15個格子 (16個格子點) 2 x 21 x 16 = 672 2 x n A x b

  20. Similar Transform Term • Warp之後，保持格子點之間的相互位置關係不要變化太多而造成扭曲。 • 找出原本格子點V1, V2, V3的相互位置關係 • 新的格子點V1’, V2’, V3’希望也具有這樣的關係 V1 Ws = Salience of the grid = Var(intensity of pixels in the grid) + 0.5 V2 V3

  21. 找格子點相互位置關係 (找u,v) u V1 V2 V3 V2 V3 V3 V1 V2 V3 V2 u, v 即格子點之間的相互位置關係 v V2 V3

  22. 同理,

  23. 新的格子點也要符合u, v  * 每個點要把所有鄰居的constrain加起來，以下只包含一個constrain。  2 x 21 x 16 = 672 672(每個點輪流當V1) A x b

  24. Optimization • α:smoothnessweight • A linear system in Ax=b form • canbeeasily solvedbyusingSVD

  25. Optimization 2 x 21 x 16 = 672 2xn 672 x b A  解出的x矩陣就是新的格子點座標。

  26. Procedure • Pre-warping • Cut input image into square grids • Geometry relation between grid and feature point • Find homography • Content-Preserving Warp • Data term • Similar transform term (smooth term) • Optimization • Texture Mapping

  27. Texture Mapping • Input: • 新格子點座標(上一步得到的x矩陣) • 存成OpenGLtexture的原圖 • 一格一格貼

  28. Combine to a new 3D video • demo

More Related