Analysis of compressed depth and image streaming on unreliable networks

1. Analysis of compressed depth and image streaming on unreliable networks Pietro Zanuttigh, Andrea Zanella, Guido M. Cortelazzo

2. The problem • Remote browsing of 3D scenes over wireless channels • 3D representations usually require huge amount of data • Real time browsing • Wireless links are typically unreliable • Robustness to packet loss issues

3. 3D Scenes 2 Basic type of data Geometry can be represented by a set of depth maps Texture Geometry

4. View warping • Depth information allows to associate each sample in each view to a point in the 3D space • If the camera positions are known it is possible to compute the position of the point in another view

5. Why using depth maps? • Both texture and geometry info are compressed by using JPEG2000 coding • Same coding/decoding scheme for both texture & geometry • Gain in simplicity • JPEG2000 is a wavelet coding scheme that yields multilayer representation • Gain in modularity • JPEG2000 is standardized • Gain in interoperability

6. Motivations • Today 3D system transmits info over TCP connections • TCP guarantees reliable transport at the cost of unpredictable time delay • Might impair navigation fluidity • UDP does not introduce extra delay but may experience packet losses • Might impair visual quality • A possible tradeoff might consist in using UDP together with protection schemes • Data packets have very different relevance: Unequal Error Protection (UEP) • A proper design of such schemes require good knowledge of the effect of packet losses on the reconstructed view…

7. Aim of this work • This work aims at shading light on these aspects, answering the following questions: • How performance degrades with increasing loss probability? • Which packets are more important? • Is it better to protect geometry or texture info?

8. Remote visualization of 3D scenes • Client-Server remote visualization system • Scene represented as a set of views and depth maps scalably compressed in JPEG2000 • Interactive browsing at client side • JPIP transmission protocol

9. Server Client Architecture of the system • The server holds the 3D scene description as a set of images and depth maps scalably compressed in JPEG2000 • It decides which elements of the compressed streams are the most suitable to be transmitted • The client renders the required view exploiting the data received from the server

10. Simulation scenario (1) • The server transmits 1 view (texture) and 1 depth map (geometry) • Depth information is used to warp the view to novel viewpoints • 2 Test models • ‘Goku’ (synthetic model) • ‘Soccer Player’ (reconstructed from real world)

11. Simulation Scenario (2) • Target scenario: wireless link, UDP protocol with no retransmissions • Lossy channel • Random packet loss (1%, 5% and 10%) • Loss of a consecutive packet batch • Comparison of the rendered views with and without packet loss

12. Original texture size ~ 40 KByte JPIP frame size: variable from ~0 byte to ~1 Kbyte Black area and highest resolution info transmitted in frames with very small size Lossless reconstruction example

13. Loss of texture information (1/2) • MSE due to the loss of texture packets vs packet loss rate & angle between available and required view (soccer player) • Distortion increases with the amount of lost packets (expected) • Distortion independent of the selected viewpoint

14. Loss of texture information (2/2) • Dependent on lost packet position • JPEG2000/JPIP transmit compressed data packets in order of relevance, losing earlier packets is worse • Unequal Error Protection could be exploited Plot shows MSE due to the loss of a batch of texture packets as a function of the position of the lost packets batch and of the angle between the available and required view

15. Loss of depth information (1/3) • Causes samples in the rendered views to be misplaced • Critical on edges • Big impact on MSE

16. Loss of depth information (2/3) Distortion increases with the amount of lost packets and depends on the position of the lost packet (as in the texture case)

17. Loss of depth information (3/3) MSE due to packet loss increases with the angle between available and required view (key difference with texture) 0 30

18. Depth and texture comparison Plot shows MSE due to the loss of depth and texture packets as a function of the amount of lost packets and of the angle between the available and required view • Depth information more important, but probably overestimated by MSE metric • Depth impact depend on the angle: • For small angles texture & depth errors have similar impact • For larger angles depth become much more important

19. Conclusions • Very different relevance of different packets (JPEG2000 transmits them in order of relevance) • Depth loss impact depends on the viewpoint, texture one does not

20. Final considerations (2)

21. Final considerations (3) • Adding redundancy might be detrimental in case packet losses are due to contention instead of wireless link errors • Cross Layer Optimization (CLO) techniques shall be used on the wireless link to shield end-to-end applications from wireless unreliability • Other transport protocols, such as Stream Control Transmission Protocol (SCTP), might be considered

22. Future work • Analysis of more complex simulation scenarios with multiple views and depth maps • Design of ad-hoc hybrid TCP-UDP (SCTP) protocols • UEP techniques for 3D models

23. We’d like to dedicate this work to Federico Maguolo He was supposed to join us on this project but his tragic death has prevent us from all the excellent ideas and contributions he would for sure have given to this work.