1 / 34

A Quest for an Internet Video Quality-of-Experience Metric

A Quest for an Internet Video Quality-of-Experience Metric. Athula Balachandran , Vyas Sekar , Aditya Akella , Srinivasan Seshan , Ion Stoica , Hui Zhang. Internet Video is taking off. Improve Users’ Quality of Experience. Video Quality Metrics: The State of the Art. Subjective Scores

karlyn
Download Presentation

A Quest for an Internet Video Quality-of-Experience Metric

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Quest for an Internet Video Quality-of-Experience Metric AthulaBalachandran, VyasSekar, AdityaAkella, SrinivasanSeshan, Ion Stoica, Hui Zhang

  2. Internet Video is taking off Improve Users’ Quality of Experience

  3. Video Quality Metrics: The State of the Art Subjective Scores (e.g., Mean Opinion Score) Objective Score (e.g., Peak Signal to Noise Ratio)

  4. Problem 1: New Effects, New Metrics PLAYER STATES Joining Playing Buffering Playing EVENTS Buffer filled up Buffer filled up Buffer empty Switch bitrate

  5. Problem 1: New Effects, New Metrics PLAYER STATES Joining Playing Buffering Playing EVENTS Buffer filled up Buffer filled up Buffer empty Switch bitrate

  6. Problem 1: New Effects, New Metrics PLAYER STATES Joining Playing Buffering Playing EVENTS Buffer filled up Buffer filled up Buffer empty Switch bitrate

  7. Problem 1: New Effects, New Metrics PLAYER STATES Joining Playing Buffering Playing EVENTS Buffer filled up Buffer filled up Buffer empty Switch bitrate

  8. Problem 1: New Effects, New Metrics PLAYER STATES Joining Playing Buffering Playing EVENTS Buffer filled up Buffer filled up Buffer empty Switch bitrate

  9. Problem 1: New Effects, New Metrics PLAYER STATES Joining Playing Buffering Playing EVENTS Buffer filled up Buffer filled up Buffer empty Switch bitrate

  10. Problem 1: New Effects, New Metrics PLAYER STATES Joining Playing Buffering Playing EVENTS Buffer filled up Buffer filled up Buffer empty Switch bitrate Join Time Buffering Ratio Rate of buffering Rate of switching Average bitrate

  11. Problem 2: Opinion Scores  Engagement Opinion Scores - Not representative of “in the wild” experience - Combinatorial explosion of parameters Engagement as replacement for opinion score. (e.g., Play time, customer return rate)

  12. Internet Video QoE Subjective Scores MOS Objective Scores PSNR

  13. Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR

  14. Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR Join Time, Avg. bitrate, …?

  15. Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …)

  16. Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …)

  17. Outline • Need for a unified QoE • What makes this hard? • Our proposed approach

  18. Challenge: Complex Engagement-to-metric Relationships Engagement Quality Metric

  19. Challenge: Complex Engagement-to-metric Relationships [Dobrian et al. Sigcomm 2011] Engagement Engagement Non-monotonic Average bitrate Quality Metric

  20. Challenge: Complex Engagement-to-metric Relationships [Dobrian et al. Sigcomm 2011] Engagement Engagement Non-monotonic Average bitrate Engagement Quality Metric Threshold Rate of switching

  21. Challenge: Complex Metric Interdependencies Join Time Bitrate Rate of switching Rate of buffering Buffering Ratio

  22. Challenge: Complex Metric Interdependencies Join Time Bitrate Rate of switching Rate of buffering Buffering Ratio

  23. Challenge: Complex Metric Interdependencies Join Time Bitrate Rate of switching Rate of buffering Buffering Ratio

  24. Challenge: Complex Metric Interdependencies Join Time Avg. bitrate Rate of switching Rate of buffering Buffering Ratio

  25. Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies

  26. Casting as a Learning Problem Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies Engagement Quality Metrics MACHINE LEARNING QoE Model

  27. Impact of the ML algorithm • Classify engagement into uniform classes • Accuracy = # of accurate predictions/ # of cases ML algorithm must be expressive enough to handle the complex relationships and interdependencies

  28. Challenge: Confounding Factors Live and VOD sessions experience similar quality

  29. Challenge: Confounding Factors However, user viewing behavior is very different

  30. Challenge: Confounding Factors Devices Connectivity User Interest Need systematic approach to identify and handle confounding factors

  31. Domain-specific Refinement Engagement Quality Metrics MACHINE LEARNING QoE Model

  32. Domain-specific Refinement Engagement Confounding Factors Quality Metrics MACHINE LEARNING QoE Model

  33. Improved prediction accuracy Refined ML models can handle confounding factors

  34. Concluding Remarks • Internet Video needs unified quantitative QoE • What makes this hard? • Complex engagement-to-metric relationships • Complex metric-to-metric interdependencies • Confounding factors (e.g., genre, device) • Promising start • Machine learning + domain-specific refinements • Open Challenges • Coverage over confounding factors • System Design

More Related