Algorithm For Image Flow Extraction- Using The Frequency Domain

1. Algorithm For Image Flow Extraction-Using The Frequency Domain

2. Image flow extraction what it is good for ? • early-warning systems• tracking & reconstructing• egomotion• visual segmentation• super resolution

3. Several methods:• differential based• region matching based• phase based • frequency based - spatiotemporal filtering

4. Motion in the frequency domain • the power spectrum of translating texture at velocity v = ( u,v) occupies a tilted plane in the frequency domain t = ux + vy• motion-sensitive gabor energy filters sample these planes efficiently Wt = UWx + VWy

5. 3D Gabor Energy Filters • a Gausssian multiplied by sine wave or cosine wave (BPF)• (t0 x0 y0) - the center frequency, maximum filter output• maximum filter output for translating texture with spatial frequency (x0 y0) and such velocity v = ( u,v ) that the temporal frequency is t = ux0 + vy0 = t0

6. • a family of filters : 12 filters with same bandwith, (x02 + y02)0.5= const, but different spatial and temporal frequencies• each velocity corresponds to different tilt of the plane thus to different distribution of filters outputs

7. gaussian pyramid Pattern flow HPF Image sequence gabor filtering The algorithm motion energy convolution measured energy theoretical energy pattern flow

8. Gaussian Pyramid •level L : sample density reduction by 2L & convolution with gaussian of size (2L+2 -3)x (2L+2 -3) • sample reduction- expands image spectrum • convolution- lower levels higher frequencies enhancement, higher levels lower frequencies enhancement • high level-high velocity, low level-low velocity (t = ux) 0-1.25 level 0 1.25-2.25 level 1 2.25-5 level 2

9. FFT of image FFT of pyramided image

10. HPF Filtering • image motion characterized by changes • changes-higher frequencies, enhance changes by HPF FFT

11. Gabor Filtering • filtering with 12 gabor sine phase filters & 12 gabor cosine phase filters

12. Motion Energy • the sum of the squared output of sine phase filter plus the squared output of sine phase filter• measure of Gabor energy that is invariant to the phase of the signal

13. Convolution with Gaussian • gabor filters are localized both in space-time and frequency domains, thus the motion energy is also localized• convolution with gaussian - enhancing center values (most reliable) decreasing far away from center values (least reliable)

14. Measured Energy • convolution with gaussian (25x25x7) divide the image into 25x25 sized segments, each of one can move in a different velocity • the center value in the convulved motion energy segment is the energy caused by a moving object in that segment

15. Theoretical Energy • Parseval’s theorem is used to derive equation that predicts the energy of gabor filter in response to moving texture , Ri(u,v)

16. Pattern Flow • for each moving object : 12 measured energies & 12 predicted energies• least squares estimate for u and v minimizes the difference between the predicted and measured energies • minimizing : Gauss-Newton method ...

17. Results • accurate for textures in motion containing spatiotemporal frequencies near the center frequencies

18. moving gaussians pattern flow

19. Car moving at velocity (2,-0.2) pixel per frame, the algorithm result (1.93,-0.16)

20. Future Improvements • adaptive center frequencies choosing • combining pyramid levels• substitute gaussian convolution with different method - e.g relaxation labeling• different numerical methods for minimization• DSP realization The End