Adaptive Content-Aware Scaling for Improved Video Streaming.

1. Adaptive Content-Aware Scaling for Improved Video Streaming. Avanish Tripathi Advisor: Mark Claypool Reader: Bob Kinicki

2. Outline • Introduction • Motivation • Related Work • Methodology • Experiments • Results • Conclusions and Future Work

3. Motivation • Internet disseminates enormous amounts of information • TCP is the de facto standard… but • TCP is not ideal for multimedia and… • 77% of all Web traffic is Multimedia, of this about 33% is streaming content. [Chandra, Ellis ’99]

4. Multimedia Flows… • …tend to use UDP with no congestion control • Other network protocols are being developed: • TFRC: smooth reduction in rates as against abrupt drops in TCP [Floyd et. al. ’00] • RAP: Architecture for delivery of layered encoded streams. [Rejaie et. al. ’99] • MPEG-TFRCP: Mapping MPEG to TFRC Protocol [Miyabayashi et. al. ’00] • Idea-rate based with smooth increase and decrease

5. Multimedia issues • Generally very high bandwidth requirements • Random packet drop by routers during congestion is detrimental to perceptual quality due to interdependencies between packets • Need application level solution…

6. …Media Scaling • Need media scaling: Application level data-rate reduction • Scaling types: • Temporal • Quality • Spatial • “Content of the stream should influence the choice of scaling mechanism” To the best of our knowledge this idea has not yet been employed

7. Related Work • Quality Scaling: Receiver-driven Layered Multicast [McCanne ’96] • Temporal Scaling: • Player for adaptive MPEG Streaming [Walpole et. al. ‘97] • Better Behaved Better Performing MM networking [Chung, Claypool ‘00] • Content based forwarding for differentiated networks: use priorities based on MPEG characteristics [Shin et. al. ’00] • Filtering System: used for media scaling of MPEG streams. [Yeadon ’96]

8. Outline • Introduction • Motivation • Related Work • Methodology • Experiments • Results • Conclusions and Future Work

9. Methodology: Content-Aware Scaling • Develop and verify motion measurement mechanism • Define temporal and quality scaling levels • Evaluate the potential impact of content-aware scaling • Build system to do content-aware scaling adaptively • Evaluate the practical impact of the full system

10. MPEG Overview • Three kinds of pictures • I- Intra encoding • P- Predictive encoding • B- Bi-directional predictive encoding • Subdivided into Macroblocks • Intra, predictive, interpolated macroblocks • Motion vectors are used for motion compensation

11. Motion Measurement • Higher percentage of interpolated macroblocks means low motion • Lower percentage of interpolated macroblocks means high motion • Conducted a pilot study to verify our hypothesis • Divide frame into 16 sub-blocks • Count the number of blocks that have motion • Correlate that with the percentage of Interpolated macroblocks.

12. Pilot Study Result: Motion Measurement

13. Motion Computation • Keep latency low so that the system is sufficiently reactive

14. Methodology: Content-Aware Scaling • Develop and verify motion measurement mechanism • Define temporal and quality scaling levels • Evaluate the potential impact of content-aware scaling • Build system to do content-aware scaling adaptively • Evaluate the practical impact of the full system

15. Filtering • We extend the system developed at Lancaster university • Frame dropping filter (Temporal Scaling) • Requantization filter (Quality Scaling)

16. User Study Details • 22 graduate and undergraduate students in the department • Platform: • 3 Pentium III machines with 128MB RAM running Linux • Clips were on local hard drives • Four ~10 second clips (2 high motion, 2 low motion) • Users rated the clips with numbers from 0 -100

17. User Study Details • Five versions of each clip: Perfect, Temporal Level 1, Temporal Level 2, Quality Level 1, Quality Level 2

18. Methodology: Content-Aware Scaling • Develop and verify motion measurement mechanism • Define temporal and quality scaling levels • Evaluate the potential impact of content-aware scaling • Build system to do content-aware scaling adaptively • Evaluate the practical impact of the full system

19. Results • Four men sitting at a bar • Low Motion ( 70 % interpolated macroblocks)

20. Results • A girl walks across a room while talking on the phone • Low Motion (57% interpolated Macroblocks)

21. Results • Rodeo scene where a man on horseback tries to rope a bull • High Motion (27% interpolated macroblocks)

22. Results • Car commerical • High Motion (20% interpolated macroblocks)

23. Methodology: Content-Aware Scaling • Develop and verify motion measurement mechanism • Define temporal and quality scaling levels • Evaluate the potential impact of content-aware scaling • Build system to do content-aware scaling adaptively • Evaluate the practical impact of the full system

24. Full System Architecture Quality Filter Internet High Motion Measurement MPEG Feedback Generator Server Client Input Low Temporal Filter

25. System Functionality • Server is capable of quantifying motion as the movie plays • The filtering system has five scale levels for finer granularity • The system is adaptive and scales movies in real-time depending on the loss pattern as received from the feedback module

26. User Study • Four clips (2 or more scene) ~30 seconds • Four versions of each • Perfect Quality • Temporal scaling • Quality scaling • Adaptive scaling • Bandwidth distribution functions: how often the rate changes • Every 3 seconds • Every 200ms • Fit the scale values(1 through 5) on a normal curve [Floyd ‘00]

27. Future Work • Accurately determine the threshold below which temporal scaling is unacceptable • More accurate bandwidth distribution function • Hybrid scaling methods (Quality + Temporal) • Audio Scaling

28. Conclusions • Application level solution to the problem of congestion due to unresponsive video streams • Developed a mechanism to quantify the amount of change in a video stream • Shown that content aware scaling can improve user perceived quality by as much as 50% • Developed a system to do adaptive content-aware scaling and are in the process of determining it impact on user perceived quality