1 / 21

Comparison of subjective test methodologies

Comparison of subjective test methodologies. P. Le Callet, R. Pépion. VQEG Berlin meeting June 2009. Context, methodologies and issues. Context:. HRCs (coder, processing, transmission …). Resolutions. ACR (5 , 11 categories …). Applications and services. Pair Comparison. SAMVIQ. DSCQS.

harriet-valdez
Download Presentation

Comparison of subjective test methodologies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Comparison of subjective test methodologies P. Le Callet, R. Pépion VQEG Berlin meeting June 2009

  2. Context, methodologies and issues Context: HRCs (coder, processing, transmission …) Resolutions ACR (5 , 11 categories …) Applications and services Pair Comparison SAMVIQ DSCQS The value (e.g. accuracy, stability) of protocols might depend on the context … and the targeted goals

  3. Outline • Study 1: ACR 5 vs SAMVIQ HD H264 • Study 2: Preference Tests vs SAMVIQ «processing» • Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing • Study 4: ACR5 for encoded + transmission error

  4. ... Good Study 1: ACR 5 vs SAMVIQ HD Motivations: HDTV high quality in a short range => quality measure should be precise and discriminative Absolute Category Rating (ACR) Subjective Assessment Methodology for Video Quality (SAMVIQ) - random order - only one viewing - category scale - no explicit reference - user-driven order - multiple viewing (natural?) - continuous scale - explicit reference

  5. Previous and new studies [Brotherton, 2006] correlation on CIF (352x288): CC(MOSACR, MOSSAMVIQ) = 0.94 New studies: • Resolutions: QVGA, VGA and HD 1080i50 (viewing distance according • to the resolution) • HRC: coding artefacts only (H264 AVC and SVC) visual field CC(MOSACR, MOSSAMVIQ) = RMSDiff= 6.73 QVGA 13° 0.969 0.942 9.31 VGA 19° 0.899 14.06 HDTV 33° ACR and seems to provide “equivalent” results up to a certain resolution

  6. Accuracy vs Number of observers confidence interval 24 number of observers « Suitable methodology in subjective video quality assessment: a resolution dependent paradigm » Stéphane Péchard, Romuald Pépion and Patrick Le CalletProceedings of the Third International Workshop on Image Media Quality and its Applications, IMQA2008, Chiba, Japan, September 2008

  7. Outline • Study 1: ACR 5 vs SAMVIQ HD H264 • Study 2: Preference Tests vs SAMVIQ «processing» • Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing • Study 4: ACR5 for encoded + transmission error

  8. Study 2: Preference Test vs SAMVIQ « processing » Motivations: HDTV pre post processing, comparison between format on a 1080p display = > No other impairments 1080p SRC Pre Processing (interleaced and down Scaling) Post Processing deinterleaced + up Scaling) 1080p PVS 1080i, 720p Pre Processing (deinterleaced + down Scaling) Post Processing (up Scaling) 720p

  9. Study 2: some results SAMVIQ Generally good agreement but …further analysis is required (Thurstone Mosteller, CI …) Preference Test 1080p SRC compared to other PVS 7 categories preference test

  10. Outline • Study 1: ACR 5 vs SAMVIQ HD H264 • Study 2: Preference Tests vs SAMVIQ «processing» • Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing • Study 4: ACR5 for encoded + transmission error

  11. Study 3: ACR5 vs ACR 11vs SAMVIQ encoded + processing Motivations: Comparison of 1080p50 with other HD and SD formats on a 1080p display => compression + processing Compression: H264 coder All formats (e.g. 1080p or i, 720p …) are coded at 3,6 and 9Mb/s and decoded before post processing. Processing: All formats are displayed in 1080p50 after decoding 1 deinterlacer : Smooth (VirtualDub/Avisynth), 2 Upscalers : Bilinear and Lanczos (VirtualDub/Avisynth).

  12. Study 3: PVS generation 29 HRC (8x3 HD +2x2 SD +1Ref) x 3 SRC = 87 PVS Deint 1080i50 Upscale 1 3Mb/s Upscale 2 720p50 Upscale 1 6Mb/s Deint 1280x 1080i50 Upscale 2 9Mb/s Not for SD Upscale 1 1280x 1080p50 Upscale 2 Upscale 1 Deint SD Upscale 2

  13. ACR5 vs ACR11: correlation correlation between ACR 5 and 11: 0.98

  14. Study 3: SAMVIQ vs ACR11, PVS generation 10 HRC (8HD +1SD +1Ref) x 2 SRC = 20 PVS Deint 1080i50 Upscale 1 Upscale 2 X 720p50 Upscale 1 Deint 1280x 1080i50 Upscale 2 Upscale 1 1280x 1080p50 Upscale 2 SD Deint Upscale 1

  15. Study 3: ACR11 vs SAMVIQ(on 20 PVS) • Good correlation between ACR and SAMVIQ (0.97) => may be questionnable for high quality score

  16. Study 3: score distribution ACR11 SAMVIQ ACR5

  17. Study 3: CI distribution

  18. Outline • Study 1: ACR 5 vs SAMVIQ HD H264 • Study 2: Preference Tests vs SAMVIQ «processing» • Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing • Study 4: ACR5 for encoded + transmission error

  19. Study 4: ACR5 encoded + transmission error • The goal : analyse the relation between the position of the transmission error and the MOS on SD sequences. • Each content is coded at 4 or 6Mb/s and some simulation of transmission errors are tested. • Advanced FEC and Error concealment technique (ROI based)

  20. Study 4: ACR5 encoded + transmission error 14 HRC (Trans- Mission Errors) 84 PVS X =

  21. Study 4: ACR5 encoded + transmission error Reminder: coding only (study 3)

More Related