Foreground Background detection from video

1. Foreground Background detection from video מאת:אבישג אנגרמן

2. The goal: dection of moving object

3. Why we need this? • video surveillance. • traffic monitoring. • Human detection. • video editing.

4. Fusing Complementary Operators to Enhance Foreground/Background Segmentation combine two probabilistic approaches: 1. Mixture of Gaussians Algorithm.2. Statistical Background Disturbance Technique

5. Mixture of Gaussians Algorithm • Pixel processes – • At any time, t, what is known about a particularpixel, {x0; y0}, is its history

6. Scatter plots of the red and green values ofa single pixel from the image over time.

8. The Algoritem • Model the values of a particular pixel as a mixture of Gaussians. • We determine which Gaussians may correspond to background colors-Based on the persistence and the variance of each of the Gaussians. • Pixel values that do not fit the background distributions are considered foreground until there is a Gaussian that includes them. • Update the Gaussians. • Pixel values that do not match one of the pixel's “background” Gaussians are grouped using connected components.

9. Model the values of a particular pixel as a mixture of Gaussians. • At time t we have k distributions of Gaussian for each pixel-determined by the available memory and computational power (Currently, 3-5 are used). • For each Gaussian we have: • -is an estimate of the weight of the ith Gaussian in the mixture at time t -(the portion of the data accounted for by this Gaussian) • . - is the mean value of the ith Gaussian in the mixture, at time t.

10. - covariance matrix of the ith Gaussian in the mixture at time t. This assumes that the red, green, and blue pixel valuesm are independent and have the same variances.

11. Gaussian probability density function. The probability of observing the current pixel value is:

12. Update the mixture model Stage 1 • Every new pixel value, Xt, is checked against the existing K Gaussian distributions until a match is found. • A match is defined as a pixel value within 2.5 standard deviations of a distribution.

13. Stage 2-No match • If none of the K distributions match the current pixel value, the least probable distribution is go out. • A new distribution with the current value as its mean value, an initially high variance, and low prior weight, is enter .

14. Stage 3 • The prior weights of the K distributions at time t are adjusted as follows: 1 for the model which matched and 0 for the remaining models. Thelearningrate

15. Stage 4 • The and parameters for unmatched distributions remain the same. • The parameters of the distribution which matches the new observation are updated as follows: The last mean The value of the new pixel

16. The last variance The distance of the new pixel from the updated mean.

17. Background Model Estimation • determine which of the Gaussians of the mixture are most likely produced by background processes. • We are interested in the Gaussian distributions which have the most supporting evidence = and the least variance. Why?

18. For “background” distributions when a static, persistent object is visible hige weight and relatively low variance. • New object occludes the background object creation of a distribution or the increase in the variance of an existing distribution the variance of the moving object is expected to remain larger than a background pixel until the moving object stops low weight and relatively hige variance.

19. Background Model Estimation • the Gaussians are ordered by the value of • Then, the first B distributions are chosen as the background model, where • T is a measure of the minimum portion of the data that should be accounted for by the background.

20. Statistical Background Disturbance Technique • The idea- separates the brightness from the chromaticity component.

21. Ei represents an expected color of a given ith pixel, in the reference or background image. • Ii represents the color value of the pixel in a current image. • The line OEi called expected chromaticity line.

22. -brightness distortion- obtained by minimizing • represents the pixel's strength of brightness with respect to the expected value. Less than 1 if it is darker 1 if the brightness of the pixel in the current image is the same as in the reference image greater than 1 if it brighter

23. CD-Color Distortion- The distance between the observed color and the expected chromaticity line.

24. Background Subtraction • Background modeling - constructs a reference image representing the background. • Threshold selection - determines appropriate threshold values used in the subtraction operation to obtain a desired detection rate. • pixel classication - classies the type of a given pixel, i.e., the pixel is the part of background (including ordinary background and shaded background), or it is a moving object.

25. Background Modeling • A reference background image computed over a number of static background frames. • Each pixel is modeled by a 4-tuple <Ei; si; ai; bi> the expected color value. Are the arithmetic means of the ith pixel's red, green, blue values computed over N background frames.

26. standard deviation of the ith pixel's red, green, blue values computed over N frame of the background frame. Cameras typically have different sensitivities to different colors, in order to make the balance weights on the three color bands we normalized the pixel color by its standard deviation.

27. Pixel Classication or Subtraction Operation • Original background (B): Brightness and chromaticity similar to those of the same pixel in the background image. • Shaded background (S): Similar chromaticity but lower brightness. • Highlighted background (H):Similar chromaticity but higher brightness. • Moving foreground object (F):Chromaticity different from the expected values in the background image.

28. the variation of the brightness distortion of ith pixel. the variation of the chromaticity distortion of the ith pixel

29. Pixel Classication • Different pixels yield different distributions of and CDi • In order to use a single threshold for all of the pixels, we need to rescale the and CDi

30. 0 H S B

31. If pixel from a moving object contains very low RGB values, This dark pixel will always be misclassied as a shadow. 0 H B S F

32. Automatic Threshold Selection • First, a histogram of the and are constructed. • The histograms are built during background learning period. • The total sample would be NXY.

33. The thresholds are now automatically selected according to the desired detection rate r. • A threshold for chromaticity distortion, , is the normalized chromaticity distortion value at the detection rate of r. • In brightness distortion is the value at that detection rate, and is the value at the (1-r)detection rate.

34. Clustering Detection Elimination

35. The problem- those pixels have very small variation in chromaticity distortion i.e. , very small bi. • Hence, limit the value of bi to default minimum bi. • an optimization process is performed. too big, and likely to exceed the threshold

36. The process: 1. assigning a default minimum bi value to all bi that are smaller than the default. 2.performing the detection on all frames. 3. compare the error rate of pixels that have bi bigger than the default value against the error rates of those pixels that have the default bi value. 4. A search is performed to find the default minimum bi that yields a balanced error rates.

39. Results

41. http://lecturer.it.kmitl.ac.th/thanarat/research/bgs.html

42. Demo

43. Speed-Up Technique Used in Our Implementation • Reduce number of operations at run-time:

44. Global S vs local si: • Screening Test: Parallel Processing: By dividing the images into segments and performing the operations independently on each segment.

45. Problems2 • New objects deposited into the scene and become part of the background scene-can be coped with by adaptively update the background model. • Highly specular surfaces (such as mirror, steel, or water surface) when the color of a point on such surfaces can change non-deterministically

46. Fusing Complementary Techniques Extending the Mixture of Gaussians to Remove Shadows and Highlights: Our first improvement is to extend the mixture of Gaussians approach to remove highlights and shadows. The Enhanced Foreground/Background SelectorThe extended mixture of Gaussians algorithm is joined with the SBD technique by ANDing the results of both operators.

47. Large size for the background distributions Dynamic Gaussian Background Distributions More background pixels but less accuracy of identifying the moving object pixels. Small background distribution sizes Maintain the accuracy of the extracted moving object but will not have the ability to contain all the background pixels

48. The algorithm first specifies two distribution sizes : small and large. • Used with the small distribution size. • The pixels identified as moving objects are tested using the statistical background disturbance technique. • If the algorithm doesn’t agrees ,the background distributions will be enlarged