Modeling and Evaluating Feedback-Based Error Control for Video Transfer

1. Modeling and Evaluating Feedback-Based Error Control for Video Transfer PhD Candidate: Yubing Wang - Computer Science, WPI, EMC Corp. Committee: Prof. Mark Claypool - Computer Science, WPI Prof. Robert Kinicki - Computer Science, WPI Prof. Dan Dougherty - Computer Science, WPI Prof. Ketan Mayer-Patel – Computer Science, UNC at Chapel Hill Ph.D. Dissertation Defense

2. Video Transfer 5 4 3 2 1 Too Late 5 3 1 5 4 3 2 1 Frame Loss Video Frames Server Error Propagation Internet Capacity Constraint Delay Constraint Client

3. Error Control 5 4 3 2 1 3 Retransmission Video Frames Server Change Coding Parameter Internet Local Concealment Client 3 NACK

4. Motivation • Frame loss degrades video quality • Feedback-based error control techniques use information from decoder to repair • Feedback indicates damage location. • Encoder and decoder cooperate in error control process. • Better than error control techniques where no interaction between encoder and decoder • Major techniques: RPS, Intra Update, Retransmission • Choice and Effectiveness depends on packet loss, RTT, video content and GOP size • No systematic exploration and comparison of impact of video and network conditions on the performance of feedback-based error control techniques

5. The Dissertation • Analyze video quality with feedback based error control • Develop analytical models to predict quality of videos streamed with RPS NACK, RPS ACK, Intra Update or Retransmission • Conduct systematic study of effects of reference distance on video quality • Validate analytical models through simulations • Analysis of loss rate, round-trip time, video content, Group Of Pictures (GOP) • Determine choice between RPS NACK, RPS ACK, Intra Update or Retransmission • Publications • “Impact of Reference Distance for Motion Compensation Prediction on Video Quality”, MMCN07 • “An Analytic Comparison of RPS Video Repair”, MMCN08 • “Modeling RPS and Evaluating Video Repair with VQM”, IEEE Transactions on Multimedia, 2009, (to appear)

6. Outline • Introduction • Background • RPS ACK • RPS NACK • Intra Update • Retransmission • Impact of Reference Distance on Video Quality • Analytical Models and Results • Model Validations • Conclusions

7. Reference Picture Selection (ACK) • The decoder acknowledges all correctly received frames • Only the acknowledged frames are used as a reference • Error propagation is avoided entirely • Distance from reference frame is reference distance • Reference distance increases with round-trip delay • Coding efficiency decreases as reference distance increases • Video quality degrades as coding efficiency decreases 6 7 1 3 4 5 2 ACK(1) ACK(2) ACK(3)

8. Reference Picture Selection (NACK) • The previous frame is used as a reference for encoding during the error-free transmission. • Reference distance is always 1 regardless of RTT • The decoder sends a NACK for the erroneous frame along with a reference frame number • Error propagation • Impact of loss increases with RTT 5 6 7 8 1 3 4 2 NACK(3)

9. Intra Update NACK(4) • Upon receiving a NACK from the decoder, encodes the current frame with intra mode • Frame is independently encoded without using any information from previous frames • Coding efficiency is reduced because of intra coding Intra-coded 1 3 4 5 6 7 8 9 2

10. Retransmission 3 NACK(3) Encoder • Retransmission of lost frames needs extra bandwidth • Packets arriving after their display times are not discarded but instead are used to reduce error propagation 5 6 7 8 9 1 3 4 2 Decoder 5 6 7 8 9 1 3 4 2

11. Outline • Introduction • Background • Impact of Reference Distance on Video Quality • Hypothesis • Methodology • Results and Analysis • Analytical Models and Results • Model Validations • Conclusions

12. Impact of Reference Distance on Video Quality • RPS selects one of several previous frames as a reference frame during compression • Distance from selected frame is reference distance • Higher reference distance, lower quality and vice versa • How reference distance affects video quality has not been quantified • A systematic study of the effects of reference distance on video quality • Data is needed for modeling RPS

13. Hypothesis • Low Motion: • The similarities among frames are high; • More macro-blocks are inter-coded; • High motion: • The similarities among frames are low; • More macro-blocks are intra-coded; • The y-intersect is determined by motion and scene complexity. • High-motion video sequences starts with low quality, degrade slower. • Low-motion video sequence starts with high quality, degrade faster.

14. Methodology • Select a set of non-compressed video clips with a variety of motion content. • All in YUV 4:2:2, CIF (352x288) • Each video sequence contains 300 video frames with a frame rate of 30 fps. • Change reference distances for each selected video sequence • Encode the video clips using H.264 • Measure video quality using • Peak-Signal-to-Noise-Ratio (PSNR) • Video Quality Metric (VQM) • Analyze the results.

15. PSNR vs. Reference Distance The relationship between PSNR and reference distance can be characterized using a logarithmic function:

16. VQM vs. Reference Distance The relationship between VQM and reference distance can be characterized using a linear function:

17. Outline • Introduction • Background • Impact of Ref. Distance on Video Quality • Analytical Models and Results • Assumptions • RPS ACK • RPS NACK • Intra Update • Retransmission • Result & Analysis • Model Validations • Conclusions

18. Assumptions 1 2 3 4 5 6 7 • Each GOB is independent from other GOBs in the same frame. • An independent video sub-sequence is referred to as a reference chain. • Each GOB is carried in a single network packet. • Reliable transmission of feedback messages are assumed. • Erroneously-decoded GOBs are repaired by local concealment. • Make no assumption on specific local concealment techniques. Assume independent packet loss with a random loss distribution. In this talk, GOB and Frame is exchangeable.

19. Model Parameters

20. Modeling of RPS ACK 1 3 4 5 2 ACK(1) ACK(2) The probability of decoding GOB (n) correctly using GOB (n-δ-i) as a reference: The probability of GOB (n) being successfully decoded is:

21. RPS ACK Modeling (cont.) The expected video quality for n-th GOB:

22. RPS NACK -- Model root C p 1- p GOB 1 [1] B p 1- p 1 3 4 5 2 GOB 2 (1) [1] p 1- p p 1- p A GOB 3 D [2] (1) (1) (2) NACK(1) NACK(2) p 1- p p 1- p 1- p p 1- p p GOB 4 (1) (2) (2) (3) [3] [1] (1) [1] • The probability of GOB (n) being successfully decoded: --- the probability of decoding GOB (n) correctly using GOB (n- δ -i) as a reference GOB Dependency Tree

23. Intra Update -- Model root p 1- p C GOB 1 B p 1- p F GOB 2 p 1- p p 1- p A E GOB 3 p 1- p p 1- p 1- p p 1- p p D GOB 4 • The probability of GOB (n) being successfully decoded: -- the probability of decoding GOB (n) correctly using Intra coding Intra-coded 1 3 4 5 2 NACK GOB Dependency Tree

24. Retransmission • Capacity constraint: • The n-th GOB in the reference chain being successfully decoded: • The expected video quality for GOB (n):

25. Outline • Introduction • Background • Impact of Ref. Distance on Video Quality • Analytical Models and Results • Assumptions • RPS ACK • RPS NACK • Intra Update • Retransmission • Result & Analysis • Model Validations • Conclusions

26. Analytic Experiments • Our analytical models consider a number of factors that may affect feedback-based repair performance: • Reference distance change • Loss probability • Round-trip time • Bitrate constraint • Video content • GOP Size • Select a set of video clips with a variety of motion content

27. Quality versus Round-Trip Time RPS ACK RPS NACK • Quality degrades with round-trip time increase • NACK resistant to degradation with round-trip time for low loss • ACK degrades uniformly with round-trip time

28. Quality versus Loss Rate RPS NACK RPS ACK • Quality degrades with loss rate increase • NACK degrades faster with high round trip times • ACK uniform degradation

29. RPS NACK vs. RPS ACK • Above trend line, ACK better. Below trend line, NACK better • Crossover points for low-motion are higher than for high-motion • Error propagation more harmful to quality than reference distance

30. Comparison • RPS NACK performs best in low loss • RPS ACK performs best in high loss • RPS ACK performs worst in low loss • Retransmission performs worst in high loss • Intra Update performs as well as RPS NACK as RTT increases RTT=80 ms RTT=240 ms

31. Outline • Introduction • Background • Impact of Ref. Distance on Video Quality • Analytical Models and Results • Model Validations • Methodology • Results • Conclusions

32. Validation -- Methodology 1(I) 2(P) 5(P) 6(P) 7(P) • Randomly drop controllable number of frames in input sequence based on given loss probability • Based on given round-trip time and randomly selected lost frames, regenerate video sequence • Encode video sequence generated in step 2 using H.264 • Measure average PSNR and VQM for encoded H.264 video sequence • Calculate average PSNR and VQM based upon video quality measured in step 4 RPS NACK, round-trip time = 2 frames, frame 3 is lost

33. Validation – RPS NACK • Error bar represents 95% confidence interval • As loss probability or round-trip time increases, the variance is increased • Simulation results are consistent with values predicted by analytical model for both PSNR and VQM

34. Outline • Introduction • Background • Impact of Ref. Distance on Video Quality • Analytical Models and Results • Model Validations • Conclusions

35. Major Contributions • Systematic study of effects of reference distance on video quality for a range of video coding conditions • Two utility functions that characterize impact of reference distance on video quality based upon study • Modeling prediction dependency among GOBs for RPS NACK and Intra Update using binary tree • Analytical models for feedback-based error control techniques including Full Retransmission, Partial Retransmission, RPS ACK, RPS NACK and Intra Update • Simulations that verify accuracy of our analytical models • Analytic experiments over a range of loss rates, round-trip times and video content using our models

36. Future Work • Explore and incorporate other existing video quality metrics or develop a new quality metric • Investigate how local concealment may affect the choice of feedback-based repair techniques • Investigate the impact of the extra bandwidth consumed by feedback messages on performance • Build a videoconference system that automatically adapts to the best repair techniques

37. Conclusions • Degree of video quality degradation is affected by video content • High-motion video sequences starts with lower quality, degrade slower. • Low-motion video sequences starts with higher quality, degrade more rapidly. • Mathematical Characterization of the relationship between video quality and reference distance: • PSNR: • VQM: • Analytical models reveal: • RPS NACK performs best in low loss • RPS ACK performs best in high loss, worst in low loss • RPS NACK outperforms RPS ACK over a wider range for low motion videos than for high motion videos • Retransmission performs worst in high loss • Intra Update performs as well as RPS NACK as RTT increases

38. Acknowledge • Prof. Claypool and Prof. Kinicki • Prof. Dougherty • Prof. Mayer-Patel from UNC at Chapel Hill • Faculty/Staff of Computer Science Dept., WPI • Huahui Wu, Mingze Li, Feng Li, and everyone from PEDS and CC groups • Attendees today • My Family