Video Object Tracking and Replacement for Post TV Production

1. Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004

2. Outline • Brief project overview and targets achieved • New components added to the proposed system • Working principles of individual parts • Future work • Q & A

3. Overview • Post-TV productionprocesses and changes the content of a video clip • Difficult for computers to process video in a global view • Step-by-step programming enables the computer to “think” more thoroughly

4. Overview • We have introduced the following parts in the last semester: • Bitmap I/O • RGB <-> HSV converter • Edge detector • Edge equation finder • Translation detector • Texture mapper • New parts have been added in order to increase thefunctionality of the system

5. Additional Basic Components • The following parts have been added • Graphical User Interface (GUI) • Corner point finder • Video file reader and writer • Video player processor (frame-based) • Improved texture mapper • The following part has been modified • Bitmap I/O • Edge finder • Texture mapper • Removed Equation processor

6. Graphical User Interface • Although UI is not a main project part, a good GUI can significantly decrease the time needed for processing and program maintenance • Since C++ language is being used, the Microsoft Foundation Classes are adopted to develop the user interface • Provide most of the basic functions needed for information input and output

7. Graphical User Interface

8. Corner Point Finder • As we know that processing dots are much faster than processing lines, we may wish to try to find the corner points directly before finding the edge equations • Stick some brightly coloured labels at the corners of the rectangular surface • Can apply on cylindrical objects as well to indicated the curve control points • A simple K-Means method is used to group the points together.

9. Corner Point Finder • In order to know the orientation of the mapped texture, we need to know which is the “first” point • For simplicity we just make the first point larger so that it contains more corner points than the others First point

10. Corner Point Finder Group No. of Member • We repeat the process until all the corner points has been processed • Points which are too far away from the main groups are discarded • Find out the centroids of the groups and move them out to give an estimation of corner positions 1 1 1 2 1 5 2 1 2 5 3 1 3 5 4 1 4 5

11. Corner Point Finder

12. Video File I/O • As we are talking about video processing, it is inevitable to handle video reading and writing • DirectShow interfaces are being used in the project • Able to process any type of AVI video as long as the Window Media Player can play them

13. Video Player • We need to choose a video renderer that is suitable for our application • To reserve the function of real-time processing and video texture in the future, the Video Mixing Renderer 9 (VMR9) is used • Allows the rendering of more than one video or bitmap on the screen at the same time

14. Video Processor • Video are sequence of bitmap frames • We capture each video frame and process them in the same way as processing the bitmaps • After a video frame is being processed, we add them to an AVI file • Default video compressor is DivX Mpeg4 encoder (FourCC code “divx”)

15. Video Renderer Video System Structure

16. Texture Mapper • A graphics design technique used to wrap a surface of a 3-D object with a texture map • The 3-D object acquires a surface texture similar to the texture map. • Colors, brightnessvalues or altitudes

17. Texture Mapper Image coordinates Texture coordinates (r,c) (u,v) Mapping function

18. Scan-line conversion Scanline yk scanning order Scanline yk+1 • line by line • process each pixel on every line • last semester, we mapped cubes. How about this semester? for every line

19. Cylindrical mapping How to map this cylinder with different textures?

20. Cylindrical mapping • Mapping cylinders 3 parts: • Surface detection • Line scanning • Texture lookup + Orientation determination

21. Surface detection • Hough algorithm • to detect the two straight edges • four points are detected

22. Surface detection • Scanning from top to bottom • the scanning slope is orthogonal • yellow points are detected • by color transition

23. Surface detection • Model the curved edges by ellipses • x2 / a2 + y2 / b2= 1 • 2 ellipses are needed • rotational factor y x z

24. Scan-line mapping • Scan-line from left to right • Problem: How to determine the corresponding texture for each pixel, say p=(x,y) ? (x,y) y x z

25. Texture lookup • Rotate the cylinder to the upright position along z-axis • Translate the cylinder to (0,0) and determine the z coordinate p=(x’,y’,z’) • Rotate the along x-axis  upright from viewpoint (0,0) y h x z

26. Texture lookup • Detect the marker which indicates where the texture starts • Look up the texture coordinate (u,v) by (y’+h/2)/h = v tan-1(x’/z’) / 2 = u y x z (u,v)

27. Cylindrical mapping After mapping the top we have : MAPPING MAPPED CYLINDER TEXTURES

28. Spherical mapping How to map this sphere with different textures?

29. Spherical mapping • Mapping spheres Also 3 parts: • Surface detection • Line scanning • Texture lookup + Orientation determination

30. Surface detection • Scan from 4 directions and get the boundary points • Determine center

31. Line scanning • Scan from left to right. • For each pixel p=(x,y), look up the texture by transforming it to the upright position (0,0) y Upright x z

32. Texture lookup • Translate to origin • Look up z-coordinate • Rotate the ball along z-axis • Rotate the ball along x-axis  (x’,y’,z’) (5) (6) (0,0) y x z

33. Texture lookup • Lookup the texture by v = cos-1(y’/R) / u = (cos-1(x’/(R sin(v))) ) / (2)

34. Spherical mapping • After mapping: Mapped Sphere Texture

35. Shadow mapping • Mapping of surface brightness • Method: • Covert the image from RGB  HSV • Modify the V (intensity) value • 5 types of shadow mapping

36. Shadow mapping • Mapping with texture brightness  often unrealistic • Mapping with image brightness

37. Shadow mapping • Mapping with average brightness of the image  more realistic • Mapping with other shadow maps (e.g. wood)

38. Shadow mapping • Transparency mapping (background information has to be obtained)

39. Problem with sphere mapping The texture is “squeezed” at the pole

40. Solution Modify the texture to this: Mapped sphere becomes: No distortion at pole

41. Q & A