MobiUS: Enable Together-Viewing Video Experience across Two Mobile Devices

1. MobiUS: Enable Together-Viewing Video Experience across Two Mobile Devices Guobin Shen, Yanlin Li, Yongguang Zhang Microsoft Research Asia

2. Contents • Introduction • Collaborative Half-frame Decoding • System Architecture and Implementation • Experimental Results and Evaluation • Discussion and conclusion

3. Introduction Motivation: A new better-together mobile application paradigm when multiple mobile devices are placed together

4. A specific together-viewing video application A higher resolution video is played back across screens of two mobile devices placed side by side

5. Assumptions • One device has a higher resolution video whose size is about twice of its screen size while the other not • Two devices can communicate effectively and directly via high-speed local wireless networks • Two devices are homogeneous: same/similar software and hardware capabilities

6. Requirements • Real-time synchronous playback • At least 15 frames per second (fps) • Same frame rendered at two screens simultaneously • Energy efficiency • Work in resource-constrained environment • Limited processing power, memory, battery life … • Dynamic adaption • Expand the video on to two devices with another coming • Shrink it on to one screen with another leaving

7. Possible Solutions • Full-frame Decoding-based Approaches: • Thin client model • Thick client model • Half-frame Decoding-based Approaches: • Whole-bitstream transmission (WTHD) • Partial-bitstream transmission (PTHD)

8. Thin Client Model • Computation of Mb not utilized • Huge bandwidth demand • Unbalanced energy consumption • Short operating lifetime Decoded right half-frame Ma: Decode whole frame Mb Display left half-frame Display right half-frame

9. Thick Client Model • Computation power of both devices utilized • Less bandwidth requirement • Balanced energy consumption • Abuse more computation power than necessary Whole bitstream Ma Mb Decode whole frame Display the left half-frame Decode whole frame Display the right half-frame

10. Whole-bitstream transmission (WTHD) • Computation power of both devices utilized • Less bandwidth requirement • Balanced energy consumption • Abuse more bandwidth than necessary Whole bitstream Ma Mb Decode the left half-frame Display the left half-frame Decode the right half-frame Display the right half-frame

11. Partial-bitstream transmission (PTHD) • Computation power of both devices utilized • Less bandwidth requirement • Balanced energy consumption • Implementation complexity Right half-bitstream Ma Mb Decode the left half-frame Display the left half-frame Decode the right half-frame Display the right half-frame

12. Comparison Which method is the best?

13. However, there is no free lunch.

14. Background on Video Coding • Properties of video sequences: • Strong spatial correlation: each frame is an image • Strong temporal correlation: capturing instant of neighboring frames close to each other • Basic logic of video coding: Maximally strip off spatial and temporal correlations

15. Motion Compensated Prediction MCP creates recursive temporal frame dependency Challenges arise from motion, but is worsened by recursive temporal dependency Ma Mb Ma Mb Cross-boundary reference effect

16. How to perform efficient half-frame decoding? Cross-device collaboration (CDC) transmit the missing reference to each other

17. Fundamental Facts • Markovian effect of MCP a later frame only depends on a previous reference frame, no matter how the reference frame is obtained • Highly skewed MV distribution the motion vector is centered at the origin (0,0) more than 80% of motion vectors are smaller than 8

18. Push-based Cross-device Delivery Scheme

20. Can it be better in energy efficiency? Percentage of boundary blocks that require cross-device collaboration and their corresponding bandwidth requirement

21. Cumulative distribution functions of horizontal component of motion vectors for the whole frames and the boundary columns

22. The bandwidth requirement of the helping traffic is relatively high, reaching half of the bandwidth required for sending the half bitstream • To make best use of multiple radio interfaces, the streaming data should be low enough for the Bluetooth’s throughput to be capable of • More than 90% motion vectors are smaller than 16, the width of a macroblock

23. Guardband-based collaborative half-frame decoding scheme Each device decodes one more column of macroblocks

24. Is it a good idea?

26. How about larger extended half-frame?

27. Larger guardband is not so beneficial

28. Argument Shall we need CDC traffic for decoding the boundary blocks in the guardband? Yes, only if we need to decode the whole guardband correctly. However, we do not have to ensure the guardband to be correctly and completely decoded.

29. Different decoding schemes for guardband blocks

30. System Archtecture

31. automatically set up a network between two mobile devices

32. a simple radio signal strength based strategy Ensure a close proximity setting

33. Check capability of a newly added device and inform the content host about the arrival or departure of the other device

34. Application level synchronization strategy RTT-based synchronization procedure

35. RTT-based Synchronization Scheme Ma Estimate RTT Wait half RTT Display the next frame Display next frame Mb

36. Decoded frames Half-bitstreams for local device Half-bitstreams for the other device Hold and send/receive the cross-device collaboration data to the other device

37. Independent full-frame based fast DCT-domain down-scaling decoding module The guardband-based collaborative half-frame decoding module Parse the original bitsream into two half bitstreams and extract the motion vectors

38. Configuration of Two Devices

39. Experimental Results

40. Benchmark of Mobile Devices Mobile devices are cost-effectively designed, Just able to meet the real-time playback requirement for videos at the same resolution of the screen

41. Decoding Speed

42. Decoding Speed Both collaborative half-frame decoding schemes significantly improve the decoding speed. The guardband-based scheme is only slightly slower than the half-frame decoding case.

43. Synchronization

44. Synchronization Due to periodical synchronization

45. Synchronization Due to a large scene change with very high motion. It is tolerable because the human visual system is far less sensitive for such slight asynchronism, especially when the motion is large.

46. Energy Efficiency Collaborative half-frame decoding scheme leads to significant energy savings.

47. Discussions • Further optimization opportunities • Service provisioning • User study • Assumption on homogeneity

48. Further Optimization Opportunities • Computing saving the color space conversion consumes 30% of the overall time • Collaborative traffic reduction simple compression; error concealment technique • Energy consumption reduction save screen backlight energy consumption through the gamma adjustment make use of dynamic voltage scaling capability

49. Service Provisioning • New encoder profiles to generate completely self-constrained substreams each substream corresponds to half-frame • Efficient arbitrary resizing transcoding to generate video content with suitable resolution

50. User Study