Extensible 3D (X3D) Graphics Requirements for Video on the Web

1. Extensible 3D (X3D) Graphics Requirements for Video on the Web Don Brutzman W3C Workshop, Video on the Web 12 December 2007 Web3D Consortium www.web3D.org Naval Postgraduate School Monterey California USA

2. Topics • X3D Overview • Functional Requirements • Video in X3D Scenes • Converting 3D Models to Video • Recommendations

3. X3D Overview

4. www.web3D.org • The Web3D Consortium is a public-private partnership of industry, agencies, universities and individuals working on open standards for real-time 3D communication on the Web. • Web3D develops, implements, evaluates and writes the Extensible 3D (X3D) Graphics specification.

5. Today’s www.web3D.org page

6. Community rules • Thanks to steady innovation by Web3D members and users, new features continue to evolve and grow into X3D capabilities • Working groups identify requirements and write implementable specifications • Individuals can directly extend specification capabilities without formal “permission”

7. ISO • Formal annual review and ratification by the International Standards Organization (ISO) have made X3D an approved standard for real-world use, both on and off the Web • Experts from 12-15 nations review our specs • Immediate adoption by other governing bodies helps to increase deployment

8. W3C • Further collaboration by Web3D Consortium with the World Wide Web Consortium (W3C) has made X3D a "first-class citizen" on the Web, providing excellent (and growing) interoperability with other XML standards • Current joint work includes efforts on Efficient XML Interchange (EXI) for compression and processing improvements.

9. Intellectual Property Rights (IPR) • Web3D and W3C have similar policies • Any known patented technology must be declared by members prior to consideration by working groups • Any patented technology contributions must be licensed on a royalty-free (RF) basis for inclusion in an openly used Web standard http://www.web3d.org/membership • Caveat: any legal problem can be solved, but only in advance

10. Open Source • Open for any use, without license fees • Free = freedom to innovate • Not necessarily free cost (more like “free puppy”) • Common shared example implementation(s) • Not a reference implementation – the specification/standard hopefully provides that – but required nevertheless • Open source implementations are necessary to break logjams when company participants can’t resolve technical issues • Also provides self-sustaining business model for continued activity, improvement

11. IPR summary • Royalties incompatible with Web architecture • Open standards + open source are both needed • Open source by itself is necessary but insufficient • Proprietary implementations are great too • Standards organizations and IPR agreements provide a stable playing field for long term

12. X3D Profiles for Extensibility • Different levels of content complexity • Browsers can support increasing levels of capability • Authors can use the proper palette for intended delivery MovieTexture node is Immersive Profile

13. Example syntax • <Shape> • <Box/> • <Appearance> • <MovieTexture/> • <TextureTransform/> • </Appearance> • </Shape>

14. Family of X3D specifications • Abstract specification describes “how it works” • Equivalent encodings • XML .x3d, ClassicVRML .x3dv, Binary .x3db • Scene Access Interface (SAI) • Consistent programming in EcmaScript, Java

15. X3D Specifications honeycomb diagram

16. The key challenge is scalability • Because the only information systems capable of scalably growing to match global scope are the Internet and the World Wide Web, X3D deliberately follows the architectural principles of World Wide Web. • Architecture of the World Wide Web, Volume One http://www.w3.org/TR/webarch

17. Functional Requirements Video within X3D Scenes 3D Geometry from Video

18. Linkability • Linkability • Video content needs to be referencable by URI • X3D approach allows ordered list of equivalent URI addresses for reliable portability, access • “Bookmark” additions are desirable for video, similar to using Viewpoints as X3D bookmarks • Further annotation of location or time as part of URI is under consideration

19. Planet 9 Raygun demos raygun.planet9.com

20. Playability • Internal playability • Video is often used as a user-facing “billboard” within the X3D scene • Can be special display or match a particular view • Mixed rendering is challenging but feasible • Performance requirements are demanding • External playability • Anchor links or scripts may launch external video displays for multimedia experience

21. Interoperability • Small number of required formats: • PNG JPEG GIF images, WAV audio, MPEG-2 video • Motivated by performance and royalty-free (RF) • Alternate formats are allowed but might not be supported, url list can alleviate this problem • X3D browsers not (yet?) plugging in video codecs, but plugin interoperability continues to improve • Adoption of imagery, audio and video formats supported by W3C is shared goal for Web3D

22. Planet 9 RayGun scene 1 Webcam stream as MovieTexture Billboard

23. Animation • Need synchronization of video playback with 3D-centric event-based behavior model • Necessary for consistent playability • Connect simple X3D animation interpolators, sequencers • Unlocks both Ecmascript and Java, either inside X3D Script node or externally in HTML page • Future work assignment • investigate possible alignment of X3D event model with W3C’s Synchronized Multimedia Integration Language (SMIL)

24. Metadata overview • X3D included document metadata (similar to html meta tags) and typed Metadata nodes • Also Humanoid Animation (H-Anim) joint naming • Some metadata conventions are emerging • SAVAGE Modeling and Analysis Language (SMAL) • No metadata vocabularies yet formalized by Web3D • Won’t reinvent other unencumbered standards

25. Metadata requirements • Video metadata of interest • duration, loopability, whether file or streamed • time, geographic location for precise positioning in virtual environment • Camera direction (and perhaps settings) • Ability to add other metadata extensions • Metadata needs to be inspectable by browser • Use cases available for video embedded within, or external to, video content stream

26. Construction of 3D geometry • Current research demonstrating ability to automatically generate 3D models from carefully registered/recorded video streams • Variety of techniques are being developed • Applications include city modeling, tracking human pedestrians, classifying produced objects, querying associated databases, etc.

27. Planet 9 RayGun scene 2 Numerous efforts generate matching 3D geometry using video sources

28. Security • Scene authors often want to encrypt or digitally sign 3D content • Content is king, but creation is hard • W3C recommendations for XML Security appear appropriate: encryption, signature, PKI • Security needs to be compatible with digital rights management (DRM) scheme • Stream security needed, but not yet addressed

29. Digital rights management • X3D’s XML and Compressed Binary encodings allow use of W3C’s Security recommendations • XML Encryption • XML Digital Signature (for authentication) • Public key infrastructure • DRM appears to be feasible, but not used yet • More uses exist than Hollywood-commercial • See Sun’s DReaM project http://www.openmediacommons.org

30. Streamability • Numerous historic examples of streamed video in 3D environments • Streamed video in X3D browsers is feasible but infrequent • Likely due to performance and format permissions • Real Time Streaming Protocol (RTSP) might be mapped to distribute X3D event model among multiple players • Exemplar is Distributed Interactive Simulation (DIS) protocol • Clear shared standards strategy might motivate Web3D member support for necessary next-step efforts

31. Planet 9 RayGun scene 3 Browsers have ability to stream video within scenes

32. Synthesis • Merging recorded or live video with 3D geometry • Motion tracking for human avatars • View from a window • Likely future capability of broad interest

33. Transparency • Goal: superposition of selected video pixels of interest without box artifacts • Example: video talking head or human avatar without confounding background • Ability to encode some pixels as transparent • Red Green Blue Alpha (RGBA) pixel encoding is not practical for video streams • Designating certain color as transparent is feasible, e.g. “green screen”

34. Export • Recording of animated activity in immersive environment, as seen from one or more animated viewpoints • Practice becoming more commonplace, (e.g. video-game players save victorious sessions) • Different from playing back events within 3D • Hollywood cliché “What I really want is to direct” now becoming feasible • Similar requirements likely (e.g. video metadata)

35. Recommendations

36. Recommendations • X3D graphics requirements for Web video are technically and politically feasible • Interactive 3D performance is biggest challenge • Plethora of oddball non-RF codecs unsustainable • Royalty free model for video content thus essential • Consider X3D technical requirements for video when building the foundation for emerging multimedia Web

37. Contact • Don Brutzman • brutzman@nps.edu • http://web.nps.navy.mil/~brutzman • Code USW/Br, Naval Postgraduate School • Monterey California 93943-5000 USA • 1.831.656.2149 voice • 1.831.656.7599 fax