Sp atiotemporal Re gularity F low ( SPREF )

1. Spatiotemporal Regularity Flow (SPREF) Mubarak Shah Computer Vision Lab School of Electrical Engineering & Computer Science University of Central Florida Orlando, FL 32765

2. What are good features? • Color Histograms • Eigen vectors • Wavelet Coefficients • Edges • Spatiotemporal Surfaces of edges • XY, XT, YT slices • Spatial/spatiotemporal tensors • SIFT • Optical Flow

3. SPREF • New Spatiotemporal feature for VACE • Generalization of Isophotes, Optical flow,… • Can be computed when gradient is zero • It analyzes whole region instead of a single pixel • Applications • Image and Video In-painting • Object removal • Video Compression • Tracking, Segmentation, …

4. Spatiotemporal Regularity • Definition:A spatiotemporal volume is regular along the directions, in which the pixels change the least. • SPatiotemporal REgularity Flow (SPREF) • 3D vector field ζ • models the directions of regularity • No motion (Spatial Regularity) • Depends on the regularity of a single frame • Presence of motion (Temporal Regularity) • Global motion • Single regularity model • Local motion • Multiple regularity models

5. Estimating SPREF • …gives the directions, along which the sum of the gradients is minimum: where F is the spatiotemporal volume, and H is a regularizing filter (Gaussian)

6. The energy function is modified according to the flow type: x-y Parallel: ζ(c1'[t], c2'[t],1) y-t Parallel: ζ(1,c2'[x], c3'[x]) x-t Parallel: ζ(c1'[y],1, c3'[y]) The SPREF Energy Functions

7. Solving for the SPREF • Approximate each flow component, cm'[p], with a 1D spline • Incorporates multiple frames in the solution. • Quadratic minimization of the energy functions • Solve for the spline parameters

8. Solving T-SPREF Equation

9. The original synthetic sequence (8 frames) x-y Parallelism: ζ(c1'[t], c2'[t],1) y-t Parallelism: ζ(1,c2'[x], c3'[x]) x-t Parallelism: ζ(c1'[y],1, c3'[y]) There are three types of planar parallelism constraints.

10. The SPREF Curves • … define the actual paths, along which the GOF is regular.

11. T-SPREF - An Overview • Demo

12. x-y Parallel SPREF

13. y x y t x y-t Parallel SPREF ζ(1,c2'[x], c3'[x])

14. y y x t x x-t Parallel SPREF ζ(c1'[y],1, c3'[y])

15. T-SPREF Results (Flower Sequence) Oblique View Top View Side View

16. T-SPREF Results (Alex Sequence) Oblique View Top View Side View x y y t x t t

17. The Affine SPREF (A-SPREF) • When the directions of regularity depend on multiple axes (zooming, rotation and etc.) • Precision of T-SPREF goes down • Translational flow model to Affine flow model • Affine (A-)SPREF Flow energy equation:

18. Comparison of T- and A- SPREF 1st row: A synthetic sequence from the Lena image. 2nd row: T-SPREF approximation to the underlying directions of regularity. 3rd row: A-SPREF approximation of the directions of regularity.

19. T-SPREF A-SPREF T-SPREF A-SPREF More examples

20. Comparison of T- and A- SPREF

21. Optical Flow Vs SPREF • SPREF carries similar but not necessarily the same information as the optical flow. • SPREF captures both the spatial and temporal regularity • Optical flow only cares about motion information in temporal direction. • When motion exists, the directions of xy parallel SPREF depend on direction of motion. • If the motion is globally translational, then xy-parallel SPREF converges to the optical flow.

22. Optical Flow Vs SPREF • Optical Flow is not well-defined where the spatiotemporal gradients are insignificant. • Spline-based formulation of SPREF minimizes over multiple frames. • The true optical flow usually lacks plane parallelism.

23. Optical Flow Vs SPREF

24. Applications of SPREF

25. Inpainting • Filling in the regions of missing data • Image Inpainting • Missing regions create spatial holes • Inpainting the missing region in the SPREF direction • Video Inpainting • Missing regions create spatiotemporal holes • Inpainting these holes require using the information from temporal neighbors.

26. Image Inpainting

27. Video Inpainting • Requires understanding the temporal behavior of the pixels. • The temporal behavior of the undamaged pixels gives clues about the behavior of the damaged pixels • Temporal behavior • Modeled explicitly by x-y Parallel SPREF

28. Video Inpainting • The algorithm (cont’d) • Estimate the x-y Parallel SPREF curves using the non-missing regions. • The pixels along the SPREF curves vary smoothly • Fit a spline to the non-missing pixels along each flow curve. • Interpolate the values of the missing pixels from the splines

29. Results • Big Bounce (Before)

30. Results • Big Bounce (Flow)

31. Results • Big Bounce (After)

32. Supervised Removal of Objects from Videos

33. Motivation • Object removal from videos • Preceding step to video inpainting • Manual selection of the object from each frame is required. • Time consuming • Use x-y Parallel SPREF to decrease the amount of manual work • Removal along the SPREF curves

34. Algorithm • Given a group of frames (GOF): • Compute the x-y Parallel SPREF, and the SPREF curves • Remove the object from the first and the last frames of the GOF • Remove the pixels along the curves, whose first and last pixels have been removed.

35. Results • Golden Eye (Final) • 86% reduction in manual work!

36. Video Compression Using SPREF

37. 3D Wavelet Decomposition • Problem • The spatiotemporalregularity of the GOF is not taken into account • Solution • Decompose the GOF along the SPREF directions • Entropy along these directions is lower: • Higher compression rate

38. SPREF-based Video Compression • Warping the wavelet basis along the flow curves • x-y Parallel : G(x,y,t) = (x+c1[t], y+c2[t], t) • y-t Parallel : G(x,y,t) = (x,y+c2[x],t+c3[x]) • x-t Parallel : G(x,y,t) = (x+c1[y], y, t+c3[t])

39. Choosing the correct SPREF type • The correct SPREF type is the one that minimizes the compression cost : Di + λRi • Di: Reconstruction error • λ: Lagrange multiplier • Ri: Bit cost of the bandelet and flow coefficients

40. Fi Segmentation for Optimal Compression • Find the segmentation of the GOF (F) into subGOFs (Fi), such that the total compression cost is minimized:

41. Oct-tree Segmentation • Recursively partition the GOF (F) into rectangular prisms (cuboids), Fi. • Compute the best flow and the compression cost for each cuboid. • Use split/merge algorithm to achieve the final segmentation. • Merge the child nodes if:

42. Compression results for frames 98-105 of the Alex sequence at 1000kbps

43. Compression results for frames 11-18 of the Akiyo sequence at 480kbps

44. Compression results for frames 99-106 of the Mobile sequence at 350kbps

45. Compression results for frames 26-33 of the Foreman sequence at 500kbps

46. Compression Results (b) (a) The bit-rate vs PSNR plots of (a) Alex, (b) Akiyo. Both SPREF-based compression and LIMAT framework are shown in the results. LIMAT framework, Secker and Taubman, IEEE TIP, 2004.

47. Compression Results (cond.) (b) (a) The bit-rate vs PSNR plots of (a) Foreman, (b) Mobile. Both SPREF-based compression and LIMAT framework are shown in the results.

48. Summary • SPREF • New Spatiotemporal Feature • Computes direction of regularity simultaneously in space & Time • Similar to optical flow, edge direction.. • SPREF is plane parallel (xy, xt, yt) • SPREF is computed using region/image information instead of a single pixel • SPREF is defined even when gradient is zero

49. Summary • Applications • Image and Video In-painting • Object Removal • Video Compression • Tracking • Segmentation

50. Orkun Alatas August 16th, 1977 - September 3rd, 2005