Generation of Virtual Image from Multiple View Point Image Database

1. ・ Generation of Virtual Image from Multiple View Point Image Database Haruki Kawanaka, Nobuaki Sado and Yuji Iwahori Nagoya Institute of Technology, Japan

2. Background • The soccer playing game has become popular in Japan since the World Cup Soccer of Japan and South Korea cosponsorship was held in 2002. • It is desired to see the game from various view points. • setting cameras at the reverse side of the goal • at the ceiling to view down. • many cameras are set at various locations • a camera has the function of pan−tilt−zoom ・ ・ But, these are not free viewpoints.

3. Previous method to generate a virtual image ・ ・ • large scale environment with many camera settings and installations • It takes much cost. • the application is restricted at only that stadium. • using a few cameras and a motion capture • at only the indoor space • the special wear and several markers ・ ・ ・ ・ ・ ・ It is difficult to use in actual games.

4. Present approach ・ ・ • The labels of the back number are generated as the virtual image. The pose of each player is not considered.

5. Study Purpose ・ ・ • generate a virtual image at another view point • from a real image • without the special environment

6. Proposed Method • The appropriate pose image of each player is determined from using multiple viewpoint image database of a player’s CG model. • Each pose image is synthesized at the position to the virtual scene. • The position of each player is assumed to be provided by the trajectory system.

7. Trajectory of two players Trajectory Recording System • We have developed Trajectory Recording System. ・ ・ ・ ・ ・

8. Flow of Proposed Method • Creation of database by CG model • Generation of virtual image for each player from image database • Recognize the pose of a player • Generate the corresponding virtual image from another view point • Synthesis of another view point image ・

9. Creation of database by CG model ～ Images for Database ～ • Image Database (Human model) is created by CG. • Various motions (run, walk, shot, pass, heading, trap etc) 280 poses • From 8 view direction (rotation with every 45°) • total 2240 images ・ ・ ・

10. Creation of database by CG model ～ Processing of Each Image～ CG Image of human model • To eliminate of many factors such as the condition of light source, skin color, hair and uniform… • It is necessary to save the data size and the search time of the image database. • Image database is created using the silhouette. • This depends on the difference of pose but does not depend on such factor of each player. Silhouette ・ ・

11. N N Creation of database by CG model ～ Processing of Each Image～ Normalize the image size 1-dimensional data • The image size for each pose is normalize. • The rectangle region which surrounds the silhouette of each pose, is extracted. • The extracted region just touches to the square with keeping its aspect ratio. • By the raster scan, one dimensional expansion of the normalized image is made. [00011000･･･] N×N ・

12. [00011000･･･] [01011010･･･] ・・・ [10001001･･･] Creation of database by CG model ～ Principal Component Analysis～ ・ Set of image data (all poses & all view) Image data If the sum of eigen values becomes over 90% effective ・ ・ Compress & Projection onto eigen space ・ Covariance matrix ・ eigen values, eigen vectors ・

13. Generation of virtual image for each player ～ Recognize the pose of a player(1)～ Detection of player N by eigen vector 1-D [0000111･･･] N Projection onto eigen space Normarize ・ ・ Each silhouette is normalized, changed to one dimensional vector and projected to a point in the eigen space. ・

14. Generation of virtual image for each player ～ Recognize the pose of a player(2)～ minimum distance A B e3 given image e2 e1 Result of search CG image Real image Eigen Space ・ ・ When Ais given, B is selected as the most similar sample. The pose of image A is recognized as B. ・

15. Generation of virtual image ～ Generate the corresponding virtual image from another view point～ Another view image is made from result of search according to view point & view direction of virtual image. ・ ＋ Coordinates are acquired from trajectory recording system (x, y) ・ Generated virtual scene ＋ ・ Virtual stadium created using the OpenGL ・

16. Experiments Actual original image Virtual image from the same view direction as original ・ ・ ・ The experiment of pose recognition ・ ・ ・

17. Experiments Virtual image from different view point Original image Player’s position is fixed. Viewpoint is moved. ・ Texture is used as soccer field. ・ It is also possible to generate an animation of movie by connecting each frame image sequentially.

18. Conclusion • A new approach to generate a virtual image from another view point is proposed. • Multi-image database to apply the eigen space method for the pose recognition. • This approach is simple but generates the reasonable virtual scene. ・ ・ ・

19. Future Works • It is difficult to discriminate the absolute pose of each player. • It is also difficult to treat the overlapped case in which two or more players cross. • Investigation of more effective matching approach is required to reduce the cost of time and memory. ・