MPEG-4: Demos, BIFS, SNHC Visual

1. MPEG-4:Demos, BIFS, SNHC Visual Lihang Ying Department of Computing Science University of Alberta, Edmonton, Canada These slides are available online: www.cs.ualberta.ca/~lihang/Share/mpeg4

2. Outline • MPEG-4 Demos and Overview • Demos • Overview • BIFS-How to Organize MPEG-4 Contents • Examples Study • Synthetic and Natural Hybrid Coding(SNHC) – Visual Part • 2D Mesh Coding • 3D Mesh Coding

3. Demos • EnvivioTV: http://www.envivio.com/products/etv/content/technical.jsp

4. It’s a plug-in for realplayer, media player or quicktime

5. What can we learn from the demos? • MPEG-4 vs MPEG-1/2 • Not merely video and audio • Interactive • Object-based

6. MPEG-4 Overview • Why MPEG-4? • Interoperability: • Create once, use everywhere • Multi-network Delivery • Scalability • Different capacity of different devices

7. What’s MPEG-J? • API: • org.iso.mpeg.mpegj • org.iso.mpeg.mpegj.scene • org.iso.mpeg.mpegj.resource • org.iso.mpeg.mpegj.decoder • org.iso.mpeg.mpegj.net • Implement MPEG-4 Decoder conveniently with MPEG-J API • Create once, run on all kinds of devices and platforms

8. Not merely audio/video • Object-based • Interoperability • Interactive • Multi-network Delivery • Decoder: Using MPEG-J • Scalability: Different Capacity • Profile/Level

9. Outline • MPEG-4 Demos and Overview • Demos • Overview • BIFS-How to Organize MPEG-4 Contents • Examples Study • Synthetic and Natural Hybrid Coding(SNHC) – Visual Part • 2D Mesh Coding • 3D Mesh Coding

10. How to Organize Contents • Scene Description • Assemble objects into audiovisual scene • Scene description format—binary format for MPEG-4 scenes (BIFS) • Object Descriptor • Describe objects

11. initial object description BIFS update (replace scene) ES_Descriptor1 scene description ES_ID1 ES_Descriptor2 scene descriptor stream Video Source Audio Source ES_ID2 ES_IDii object descriptor update ES_IDi object descr. object descr. ES_Descr1 object descriptor stream ES_D1 ES_Desc2 ES_IDc ES_IDb ES_IDa visual stream (base layer) visual stream (e.g. temporal enhancement layer) audio stream

12. Scene Description - BIFS • Represented by XMT-A Format: • Similar to XML • Express bitstream syntax in document • Enable easy generation of bitstream parser • BIFS Examples: …

13. BIFS Example(1)–Trivial Scene(MPEG-2/DVD) • Scene Tree Layer2D Sound2D AudioSource Shape Bitmap Appearance MovieTexture

14. BIFS Example(1)–Trivial Scene(MPEG-2/DVD)

15. BIFS Example(2)–Movie with Subtitles

16. Event Response BIFS Example(3)–Buttons

17. Object Description • Syntactic Description Language (SDL) • Express bitstream syntax in document • Enable easy generation of bitstream parser • SDL Example:…

18. Syntactic Description Language • ObjectDescriptor class ObjectDescriptor extends ObjectDescriptorBase: bit(8) tag=ObjectDescrTag { bit(10) ObjectDescriptorID; bit(1) URL_Flag; const bit(5) reserved=0b1111.1; if (URL_Flag) { bit(8) URLlength; bit(8) URLstring(URLlength); } else { ES_Descriptor esDescr[1..255]; OCI_Descriptor ociDescr[0..255]; IPMP_DescritporPointer ipmpDescriPtr[0..255]; } ExtensionDescriptor extDescr[0..255]; }

19. Summary of Object Descriptor • ObjectDescriptor • ObjectDescriptorID • URL_Flag • ES_Descriptor // Elementary Streaming ES_ID, streamDependenceFlag, URL_Flag, OCRstreamFlag, streamPriority, DecoderConfigDescriptor, SLConfigDescriptor, IPI_DescrPointer, IP_IdentificationDataSet, IPMP_DescriptorPointer, LanguageDescriptor, QoS_Decriptor... • OCI_Descriptor // Object Content Information ContentClassificationDescriptor, KeywordDescriptor, RatingDecriptor, LanguageDescriptor, ShortTextualDescriptor, ExpandedTextualDescriptor, ContentCreatorNameDescriptor, ContentCreationDataDescriptor, OCICreatorNameDescriptor, OCICreationNameDescriptor, SmpteCameraPositionDescriptor, MediaTimeDescriptor, ... • IPMP_DescriptorPointer // Intellectual Property Management and Protection • Applications of OCI/IPMP–eDonkey’s problems

21. MPEG-4 Objects and Tools • Audio • Natural Audio • Synthetic and Natural Hybrid Coding(SNHC) • TTS/Structured Audio/… • Visual • Natural Video • Object-based/Scalability • SNHC • 2D/3D Mesh Object/Face and Body Animation • Image • Text …

22. Outline • MPEG-4 Demos and Overview • Demos • Overview • BIFS-How to Organize MPEG-4 Contents • Examples Study • Synthetic and Natural Hybrid Coding(SNHC) – Visual Part • 2D Mesh Coding • 3D Mesh Coding

23. [2D Mesh Coding] • Natural Video Coding • Block-based textual and motion coding • Shape information coding • 2D Mesh Coding • Designed for video manipulation • Natural images and video mapped on 2D meshes • Applications: Object tracking, Content-based video retrieval(e.g. motion-based queries), 2D animation, Augmented reality, …

24. Example • (a)original frame • (b)Mesh generated • (c)Text overlaid on video:Text moves along with the fish’s meshes

25. Architecture of 2D Mesh Coding

26. 2D Mesh Object • Support video coding by moving the vertices of the mesh • Topology of the mesh does not change in one session • Mesh Data includes: • Connectivity: how vertices are connected • Geometry: 2D coordinates of vertices • Motion: temporal difference of vertices’ positions

27. 2D Mesh Decoding Scheme

28. Coding of Mesh Motion • Mesh Traversal: • 1) Start from top-left, breadth-first • 2) Right(Next counterclockwise) • 3) Left • This order remain unchanged(intact) until next I-MOP is decoded • Mesh Motion Coding • Encoded based on previously encoded two neighboring vertices, e.g.

29. [3D Mesh Coding] • 2D Mesh Coding: • supports to map natural images and video mapped on 2D meshes • 3D Mesh Coding: • Represent and compress 3D objects onto which images and videos may be mapped • Compress static 3D models, not their animation

30. Functionalities • High compression • 2%-4% of VRML ASCII file • Incremental rendering • Building the model with part bitstream • Error resilience • Suffer less from network errors • Hierarchical buildup • Scalable bitstream with different resolutions, depending on viewing distance

31. Incremental Rendering

32. Data of 3D Mesh Object • Connectivity: • how vertices are connected • Geometry: • 3D coordinates of vertices • Photometry • Colors • Normals • Texture

33. Bitstream of 3D Mesh Coding • Connectivity Data • Vertex graph • Triangle tree • Triangle Data (Geometry&Photometry) • Contains: geometry coordinates, colors, normals, texture coordinates • Largest part of the bitstream

34. Bitstream of 3D Mesh Coding • Connectivity Data is packed separately and before the Triangle Data. • Benefits: • Incremental rendering: • Could decode Triangle Data incrementally since full Connectivity(topology) Data is already available • Shorten the latency • Error resilience: • Can form 3D structure even with some missing Triangle Data

35. Decoding Scheme of 3D Mesh

36. Vertex Graph

37. Triangle Tree

38. Coding of Geometry and Photometry Data • 1) Quantization • 2) Differential Coding • 3) Adaptive Arithmetic Entropy Coding • Code the differential values

39. 3D Mesh Coding Modes • Error-Resilience Mode • To minimize the impact of errors, divide into partition or packet • Render partitions independently • Progressive Transmission Mode • Scalable coding • Onebase layer • One or moreenhancement layers • Provide Forest Split operations • Contains face forest, triangle tree, triangle data

40. Forest Split Operation (a) Cut through the edges of vertex tree (b) Open the dotted line (c) Triangulate the opening to form a triangle tree (d) Refined mesh

41. References • Books: • Major Reference: Fernando Pereira,Touradj Ebrahimi,The MPEG-4 Book, Prenticle Hall PTR, 2002 • Natural Video Coding Technology: Joan L.Mitchell,etc. MPEG Video Compression Standard, Chapman&Hall, 1996 • MPEG Official Websites: • Overview: http://mpeg.telecomitalialab.com/standards.htm • Resources: http://www.m4if.org/resources.php • Demos: • http://www.envivio.com/products/etv/content/technical.jsp • http://www.ivast.com/aboutmpeg4/index.html • MPEG-4 Series Slides, Course Presentation of C640/2003 Winter, U. of Alberta: • http://www.cs.ualberta.ca/~anup/Courses/604/604_3D.htm

42. The End • Acknowledgements • Yongjie Liu • Michael Closson • Questions and Comments?