Direct Mapping of X3D Scenes to MPEG-7 Descriptions

1. Direct Mapping of X3D Scenes to MPEG-7 Descriptions MarkosZampoglou, Patti Spala, KonstantinosKontakis, AthanasiosG. Malamos, J Andrew Ware

2. What is our motivation i-Promotion: we work on a platform for VR web advertising. The goal of this project is to join cloud stored X3D objects to more complicated scenes and to publish them in the way that we used to publish GIF Banners! The first questions we had to answer was how can we search and retrieve the resources according to technical as well as functional and contextual characteristics. The resources might be the 3D geometry as well as Routes, Textures (images or videos), audios and Sounds embedded on the 3D. What is missing is an efficient annotation of X3D in MPEG7. MPEG-7 defines several descriptors to facilitate the description of 3D entities involving geometries, textures, animations and content metadata. However, the majority of 3D descriptors are centered on vertices representation of 3D objects, lacking in the ability to provide optimized semantic descriptions based on XML encoded 3D objects. However the existence of the 3D descriptors helped us to extend them to fit in our case allowing us to still use the older and the newer descriptors at the same time. I-Promotion is a project granted by the European Union and the Hellenic General Secretary of Research and Technology under the ”COOPERATION 2009 /09SYN-72-956” Framework.

3. The i-Promotion platform HTTP HTTP WEB DigitalMedia Mobile WEB Mobile 3D banners Promotion Manager Client Devices Client Devices Web-sockets RTSP 3D worlds Authoring tool HTTP/SOAP I-promotion core HTTP Digital Media WEB/HTML5 WEB/HTML5 WEB WEB Mobile Mobile Digital TV Digital TV Device Interfaces Device Interfaces DeviceInterfaces Media framework CROSS 3D Hosts HTTP HTTP SOAP SOAP I Promotion ServiceProtocols (MPEG7, MPEG21, etc) CROSS3D Service Protocols Renderingengines Rendering engines Rendering engines Rendering engines 3D ModelRepositories 3D Model Repositories Cloud Virtual machines

4. The i-Promotion platform architecture • A cloud-based repository for 3D scenes • A remote rendering server • supplementing a potential local rendering approach • Service-oriented content distribution • HTML5-based clients • PCs, mobile devices, wall-to-wall projections… • Supplementary modules for content adaptation • Device Interfaces, Media Framework • Additional tools • 3D Worlds authoring tool, 3D banners promotion manager

5. The i-Promotion platform:Standards conformance • MPEG-7: Content description for 3D scenes • Content-based search • MPEG-21: Multimedia Item declaration & adaptation • Scene organization (thumbnails, descriptors, rights) • Content adaptation for extreme device flexibility • ???? : Scene structure composition • X3D: 3D scene format • Offers a choice between server-based rendering or local, HTML5 compatible rendering (via X3DOM)

6. MPEG-7 for 3D scenes • MPEG-7 – Part 3: Visual includes descriptors for 3D model shape. • Efforts towards full 3D scene description (shape, color, structure, light etc.) • 3DSEAM: X3D scene structure and object location (Bilasco et al. 2006) • Multimedia Interaction Model (Chmielewski2012) • MPEG-7 Shape and semantics for MPEG-4 3D scenes (Bonhomme et al. 2007) • Various RDF and OWL Semantic annotations

7. What MPEG 7 does support in 3D Geometry Descriptors MPEG-7 Part 3, Visual, defines two shape descriptors for 3D objects: the Shape3DSpectrum descriptor and the MultipleView Descriptor Container. Motion Descriptors MPEG-7 Part 3, Visual, defines three descriptors, The Camera Motion descriptor, The Motion Trajectory descriptor and the Motion Activity descriptor. Texture and Color Descriptors In order to describe textures, MPEG-7 Part 3, Visual, provides three texture descriptors: the TextureBrowsing Descriptor, the HomogeneousTexture Descriptor and EgdeHistogram Descriptor. The color description tools presented in, MPEG-7 Part 3, Visual, include four distinct color descriptors to annotate different color features, the DominantColor Descriptor, the ScalableColor Descriptor, and the ColorLayout and ColorStructure Descriptors. Meanwhile, two additional supporting descriptors are included, the ColorSpace and ColorQuantization Descriptors, which are used in conjunction with the DominantColor Descriptor, providing supplementary details to representative colors.

8. MPEG-7 Extension Descriptors

9. Automatic extraction of MPEG-7 descriptions • X3D and MPEG-7 are both XML-based standards • We use XSLT for the direct extraction of (extended-) MPEG-7 compliant descriptions from X3D scenes • Both scenes are kept: the original X3D scene is retained for rendering and display, while the new MPEG-7 description is used for search and retrieval

10. XSLT Mapping

11. Extensions in MPEG-7 Part 3 Visual

12. BoundingBox3D Elements

13. BoundingBox3D

14. BoundingBox3DType Fields

15. BoundingBox Mapping

16. Geometry3D Elements

17. Geometry3D

18. Geometry3DType Fields

19. Geometry3D Mapping

20. Metadata3DType Elements

21. Metadata

22. MetadataSet

23. Metadata3DType Fields

24. MetadataSet Mapping

25. MotionTrajectory Elements

26. MotionTrajectory

27. MotionTrajectory Fields

28. The MotionType attribute

29. MotionTrajectory Mapping

30. Interactivity3D Elements

31. Interactivity3D

32. Interactivity3D Fields

33. Interactivity3D

34. MetadataHeader Fields

35. MetadataHeaderMultimediaContent Descriptor Mapping

36. Viewpoint3D Elements

37. Viewpoint3D

38. Viewpoint3D fields

39. Viewpoint3D Mapping

40. Lighting3D Elements

41. Lighting3D

42. Lighting3D Fields

43. Lighting3D Mapping

44. Extensions in MPEG-7 Part 5 Multimedia Description Schemes

45. Profile3D & Script3D

46. Profile3D Elements

47. Profile3D Fields

48. Profile3D Version Mapping

49. Script3D Elements

50. Script3D Fields