Quality of Multimedia Experience Past, Present and Future

1. Quality of Multimedia ExperiencePast, Present and Future Prof. Dr. Touradj Ebrahimi Touradj.Ebrahimi@epfl.ch

2. Today we will talk about… • What is “quality” of multimedia content? • Howis multimedia content “quality” measured today? • What are trends in assessment of “quality” in multimedia? • What are the challenges ahead?

3. Quality – a simple yet difficult concept • Like many human sensations quality is easy to understand but difficult to define • According to Wikipedia: • A quality (from Latin - qualitas) is an attribute or a property. • Some philosophers assert that a quality cannot be defined. • In contemporary philosophy, the idea of qualities and especially how to distinguish certain kinds of qualities from one another remains controversial.

4. A fundamental yet largely under-investigated concept • Aristotle classified every object of human apprehension into 10 Categories • Substance, Quantity, Quality, Relation, Place, Time, Position, State, Action, Affection Aristotle 384 BC – 322 BC

5. Some definitions according to dictionary • Definition 1 • General : Measure of excellence or state of being free from defects, deficiencies, and significant variations. • ISO 8402-1986 standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs"

6. Some definitions according to dictionary • Definition 2 • Manufacturing : Strict and consistent adherence to measurable and verifiable standards to achieve uniformity of output that satisfies specific customer or user requirements.

7. Some definitions according to dictionary • Definition 3 • Objective : Measurable and verifiable aspect of a thing or phenomenon, expressed in numbers or quantities, such as lightness or heaviness, thickness or thinness, softness or hardness.

8. Some definitions according to dictionary • Definition 4 • Subjective : Attribute, characteristic, or property of a thing or phenomenon that can be observed and interpreted, and may be approximated (quantified) but cannot be measured, such as beauty, feel, flavor, taste.

9. Definition according to ISO 9000 • ISO 9000: a family of standards for quality management systems • Quality of something can be determined by comparing a set of inherent characteristics with a set of requirements • High quality: if characteristics meet requirements • Low quality: if characteristics do not meet all requirements • Quality is a relative concept • Degree of quality

10. Quality – is in fact an elephant The blind men and the elephant: Poem by John Godfrey Saxe

11. Quality of Service in computer networks and communications • Quality of Service (QoS) refers to a collection of networking technologies and measurement tools that allow for the network to guarantee delivering predictable results

12. Network Quality Capacity Coverage Handoff Link Quality Bitrate Frame/Bit/Packet loss Delay User Quality Speech fidelity Audio fidelity Image fidelity Video fidelity Quality in QoS framework

13. Quality of Service in computer networks and communications • Quality of Service (QoS) • Resource reservation control mechanisms • Ability to provide different priority to different applications, users, or data flows • Guarantee a certain level of performance (quality) to a data flow • (Service) Provide-centric concept • Tightly related to the concept of Mean Opinion Score (MOS)

14. What is Mean Opinion Score (MOS)? • Widely used in many fields: • Politics/Elections • Marketing/Advertisement • Food industry • Multimedia • … • The likely level of satisfaction of a service or product as appreciated by an averageuser • Should be performed such that it generates reliable and reproducible results • Subjective evaluation methodology • More complex and difficult that it a priori seems

15. What is behind a MOS?

16. Subjective evaluation • A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices: • Test/lab environment • Test material • Test methodology • Analysis of the data

17. Test/lab environment • Type of Monitors/Speakers and other test equipments • Lighting /Acoustic conditions • Laboratory architecture, background, … • Viewing distance /Hearing position • …

18. Test material • Meaningful content for the envisaged scenario/application • Typical content • Worst case content • … p01 p06 p10 bike cafe woman

19. Test methodology • Subjects • Naïve or Expert? • Instructions • Which questions to ask subjects and how • Training • Presentation • Single or double stimulus • Sequential or simultaneous • Grading scale • Numerical • Categorical • …

20. ITU Recommendations for test methodologies • Test conditions and methodologies are specified in: • Recommendation ITU-R BT. 500-11 “Methodology for the subjective assessment of the quality of television pictures” (1974-2002). • Recommendation ITU-T P. 910 “Subjective video quality assessment methods for multimedia applications” (1999). • Recommendation ITU-R BT. 1788 “Methodology for the subjective assessment of video quality in multimedia applications” (2007). • Based on television scenario!

21. Test methodology (I) • Single Stimulus (SS) • Categorical numerical grading scale: • “Rate from 1 to 11” • Categorical adjectival grading scale: • Non-categorical adjectival or numerical grading scale 100 0 Excellent Bad

22. Test methodology (II) • Double Stimulus Impairment Scale (DSIS) • Categorical impairment grading scale:

23. Test methodology (III) • Double Stimulus Continuous Quality Scale (DSCQS) • Non-categorical adjectival or numerical grading scale: Sample 1 Sample 2 100 0 Excellent Bad 100 0 Excellent Bad Sample 2 Sample 1

24. Test methodology (IV) • Stimulus Comparison (SC) • Categorical adjectival comparison scale: • “same or different” • Non-categorical judgement: Much worse Much better

25. Test methodology (V) • Single Stimulus Continuous Quality Evaluation (SSCQE) (Very annoying) (Imperceptible)

26. Test methodology (VI) • Simultaneous Double Stimulus for Continuous Evaluation (SDSCE) (Much better) (Much worse) (Reference) (Test sequence)

27. Analysis of the data • Scores distributions across subjects is assumed to be close to normal distribution • Outlier detection and removal • Mean Opinion Scores (MOS) and 95% confidence intervals mij = score by subject i for test condition j. N = number of subjects after outliers removal. t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 degrees of freedom and a desired significance level α (α=0.05 in our case). σj= standard deviation of the scores distribution across subjects for test condition j.

28. What is behind a MOS?

29. Relationship between estimated mean values • Hypothesis test to find out whether the difference between two MOS values are statistically significant • Two-sided t-test: • t statistic: • Decision rule to reject H0:

30. MOS hypothesis test 6 5 4 3 2 1 0 JPEG 2000 4:2:0 JPEG 2000 4:2:0 JPEG 2000 4:4:4 JPEG 2000 4:4:4 JPEG JPEG JPEG XR MS JPEG XR MS 0.75 bpp 0.25 bpp 0.50 bpp JPEG XR PS JPEG XR PS 1.50 bpp 1.00 bpp 1.25 bpp Number of times H0 is rejected JPEG XR PS JPEG XR PS JPEG XR PS JPEG JPEG JPEG JPEG XR MS JPEG XR MS JPEG XR MS JPEG 2000 4:4:4 JPEG 2000 4:4:4 JPEG 2000 4:4:4 JPEG 2000 4:2:0 JPEG 2000 4:2:0 JPEG 2000 4:2:0

31. Objective quality metrics • Subjective tests are time consuming, expensive, and difficult to design • Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired • Full reference metrics • No reference metrics • Reduced reference metrics

32. FR, RR and NR scenarios • Full Referenceapproach: • ReducedReferenceapproach: • NoReferenceapproach: signal processing Input/Reference signal Output/Processed signal FR METRIC signal processing Input/Reference signal Output/Processed signal Features extraction RR METRIC signal processing Input/Reference signal Output/Processed signal NR METRIC

33. MOS predictors based on fidelity measures • Full Reference scenario • Metrics which look at the fidelity of the signal when compared to an explicit reference: processed signal = perfect quality reference signal + error signal

34. MOS predictors based on fidelity metrics • Examples • Mean Square Error (MSE) • Peak Signal to Noise Ratio (PSNR) • Maximum Pixel Deviation (Linf) • Weighted PSNR • Masked PSNR • Structural SIMilarity (SSIM) • Multiscale Structural Similarity (MSSIM) • Visual Information Fidelity (VIF) • etc…

35. Peak Signal to Noise Ratio • where: • M, N = image dimensions • Ima , Imb = pictures to compare • B= bit depth • Widely used because of its simplicity and ease in formalizing optimization problems! • For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation

36. PSNR for color images/video (I) • Several alternatives to compute PSNR for color images/video: WPSNR_PIX WPSNR_MSE WPSNR = w1PSNR1 + w2PSNR2 + w3PSNR3

37. PSNR for color images/video (II) • Which color space to use? • RGB • Y’CbCr • other? • Which weights to use? • w1=w2=w3=1/3 • w1=0.8, w2=w3=0.1 • other?

38. PSNR for color images/video (III) PSNR (dB) PSNR(dB) PSNR (dB) on B component: on G component: on R component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

39. PSNR for color images/video (IV) PSNR (dB) PSNR(dB) PSNR (dB) on Cr component: on Y’ component: on Cb component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

40. MOS predictors based on fidelity metrics • Human Visual System (HVS) based metrics • simulating properties of the early stages of the HVS • Examples: PSNR-HVS-M, etc… • simulating high level features of the HVS • Examples: Osberger’s metric, etc…  Better correlation with human perception.  High complexity. Visibility Map Computationalmodel of thevisual system

41. PSNR-HVS-M • where: • M, N = image dimensions • K = constant • = visible difference between DCT coefficients of the original • and distorted based on a contrast masking • Tc = matrix of correcting factors based on standard visually optimized • JPEG quantization tables • B= bit depth

42. MOS predictors based on fidelity measures • Metrics based on the hypothesis that the HVS is highly adapted for extraction of structural information from the content of a still image or video. • degradation of still images or video = perceived structural information variation • Structural Similarity by Wang et al.

43. Mean SSIM (MSSIM) • Luminance comparison function: (C1=constant) • Contrast comparison function: (C2=constant) • Structure comparison function: (C3=constant)

44. MSSIM vs PSNR PSNR = 24.9 dB for all the images Mean SSIM (MSSIM) MSSIM=0.9168 MSSIM=0.6949 MSSIM=0.7052

45. Specific distortion metrics • At times one is interested in specific types of distortions that occur in multimedia systems • Examples • Bluriness • Blockiness • Ringing/Mosquito noise • Jerkiness • etc…

46. Blur metric • A perceptual quality blur metric without a reference image. • Example: Gaussian blurred image JPEG2000 compressed image

47. NR blur metric

48. NR blur metric

49. Correlation subjective ratings / NR blur metrics 96% correlation 85% correlation

50. Multimedia communication – a definition • Multimedia is about sharing experience (real or imaginary) with others • In a way it all started with story telling and wall drawing around the fire in the caves of early men • Modern multimedia systems are evolved versions of the good old story telling and wall drawing, which hopefully offer increasingly richer experience • The degree of richness of the experience is measured by Quality of Experience (QoE)