Developing a Predictive Model for Internet Video Quality-of-Experience

1. Developing a Predictive Model for Internet Video Quality-of-Experience AthulaBalachandran, VyasSekar, AdityaAkella, SrinivasanSeshan, Ion Stoica, Hui Zhang

2. QoE$$$ CDN Better Quality Video $$$ Content Providers Users Higher Engagement The QoE model Diagram courtesy: Prof. Ramesh Sitaraman, IMC 2012

3. Why do we need a QoE model? Adapting video bitrates quicker Picking the best server Comparing CDNs

4. Traditional Video Quality Metrics User studies not representative of “in-the-wild” experience Subjective Scores (e.g., Mean Opinion Score) Does not capture new effects (e.g., buffering, switching bitrates) Objective Scores (e.g., Peak Signal to Noise Ratio)

5. Internet Video is a new ball game Engagement (e.g., fraction of video viewed) Subjective Scores (e.g., Mean Opinion Score) Quality metrics Objective Scores (e.g., Peak Signal to Noise Ratio)

6. Commonly used Quality Metrics Join Time Buffering ratio Rate of switching Rate of buffering Average Bitrate

7. Which metric should we use? Today: Qualitative Single-metric Engagement (e.g., fraction of video viewed) Subjective Scores (e.g., Mean Opinion Score) Quality metrics Buffering Ratio, Average bitrate? Objective Scores (e.g., Peak Signal to Noise Ratio)

8. Unified and Quantitative QoE Model Engagement (e.g., fraction of video viewed) Subjective Scores (e.g., Mean Opinion Score) Quality metrics Buffering Ratio, Average bitrate? ƒ (Buffering Ratio, Average bitrate,…) Objective Scores (e.g., Peak Signal to Noise Ratio)

9. Outline • What makes this hard? • Our approach • Conclusion

10. Complex Engagement-to-metric Relationships [Dobrian et al. Sigcomm 2011] Engagement Engagement Non-monotonic Average bitrate Engagement Quality Metric Threshold Ideal Scenario Rate of switching 10

11. Complex Metric Interdependencies Join Time Bitrate Join Time Rate of buffering Rate of switching Average bitrate Rate of buffering Buffering ratio Average bitrate

12. Confounding Factors Live Type of Video VOD Confounding Factors can affect: Engagement Live and Video on Demand (VOD) sessions have different viewing patterns. CDF ( % of users) Engagement

13. Confounding Factors Live Type of Video VOD CDF ( % of users) Confounding Factors can affect: Engagement Quality Metrics Live and Video on Demand (VOD) sessions had different join time distribution. Join Time

14. Confounding Factors Type of Video DSL/Cable Connectivity Engagement Wireless (3G/4G) • Confounding Factors can affect: • Engagement • Quality Metrics • Quality Metric  Engagement Users on wireless connectivity were more tolerant to rate of buffering. Rate of buffering

15. Confounding Factors Popularity Device Type of Video Location Connectivity Time of day Day of week Need systematic approach to identify and incorporate confounding factors

16. Summary of Challenges • Capture complex engagement-to-metric relationships and metric-to-metric dependencies. • Identify confounding factors • Incorporate confounding factors

17. Outline • What makes this hard? • Our approach • Conclusion

18. Challenge 1: Capture complex relationships

19. Cast as a Learning Problem Quality Metrics Engagement MACHINE LEARNING QoE Model Decision Trees performed the best. Accuracy of 40% for predicting within a 10% bucket.

20. Challenge 2: Identify the confounding factors

21. Test Potential Factors Confounding Factors Quality Metrics Engagement

22. Test Potential Factors Confounding Factors Test 1: Relative Information Gain Quality Metrics Engagement

23. Test Potential Factors Confounding Factors Test 1: Relative Information Gain Test 2: Decision Tree Structure Test 3: Tolerance Level Quality Metrics Engagement

24. Identifying Key Confounding Factors

25. Identifying Key Confounding Factors

26. Challenge 3: Incorporate the confounding factors

27. Refine the Model Adding as a feature Splitting the data Confounding Factors 3 e.g., VOD, TV Confounding Factors 1 e.g., Live, Mobile Confounding Factors 2 e.g., VOD, Mobile Confounding Factors Quality Metrics Quality Metrics Quality Metrics Quality Metrics Engagement Engmnt Engmnt Engmnt MACHINE LEARNING ML ML ML Model 2 Model 1 Model 3 QoE Model QoE Model

28. Comparing Candidate Solutions Final Model: Collection of decision trees Final Accuracy- 70% (c.f. 40%) for 10% buckets

29. Summary of Our Approach • Capture complex engagement-to-metric relationships and metric-to-metric dependencies  Use Machine Learning • Identify confounding factors Tests 3. Incorporate confounding factors Split

30. Evaluation: Benefit of the QoE Model Preliminary results show that using QoE model to select bitrate leads to 20% improvement in engagement

31. Conclusions • Internet Video needs a unified and quantitative QoE model • What makes this hard? • Complex relationships • Confounding factors (e.g., type of video, device) • Developing a model • ML + refinements => Collection of decision trees • Preliminary evaluation shows that using the QoE model can lead to 20% improvement in engagement • What’s missing? • Coverage over confounding factors • Evolution of the metric with time

32. EXTRA SLIDES

33. Choice of ML Algorithm Matters • Classify engagement into uniform classes • Accuracy = # of accurate predictions/ # of cases ML algorithm must be expressive enough to handle the complex relationships and interdependencies