1 / 24

Experiment on 3D mobile video by subjective tests for VQEG

Experiment on 3D mobile video by subjective tests for VQEG. Orange Labs Gilles TENIOU, FT/IMG/RD/TECH/OPERA/CVA Julien LIBOUBAN, FT/IMG/RD/TECH/OPERA/HEAT Jérôme BRIARD, FT/IMG/RD/TECH/OPERA/HEAT December, 26 th 2011. Content. part 1 Context of the study part 2 Tests description

zeke
Download Presentation

Experiment on 3D mobile video by subjective tests for VQEG

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Experiment on 3D mobile video by subjective testsfor VQEG Orange Labs Gilles TENIOU, FT/IMG/RD/TECH/OPERA/CVA Julien LIBOUBAN, FT/IMG/RD/TECH/OPERA/HEAT Jérôme BRIARD, FT/IMG/RD/TECH/OPERA/HEAT December, 26th 2011

  2. Content part 1 Context of the study part 2 Tests description part 3 Subjective tests results part 4 Conclusion Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  3. Part 1 Context of the study Orange Labs - Research & Development – High Speed Access Video profiles – 26th Jan. 10 interne Groupe France Télécom

  4. Context and objectives of the study • Context • Contribution in the 3GPP : Technical specification group services and system aspects in the release 11 (Mobile stereoscopic 3D video) • Objectives • Test video quality on 3D smartphone devices • Evaluate the SAMVIQ (ITU-R BT 1788) methodology on a mobile device • Find for 3D contents on mobile : • the best frame rate • the best video resolution Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  5. Part 2 Tests description Orange Labs - Research & Development – High Speed Access Video profiles – 26th Jan. 10 interne Groupe France Télécom

  6. Test description (1/6) – Video Sources • Video Sequences • Mix of sequences with different characteristics in terms of movement, textures, details, transitions… • 2 points evaluated : impact of the frame rate and the resolution on 3D contents. • Need of high resolution (full HD per view) and high frame rate 3D contents. Animated movie Football Interview Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  7. Test description (2/6) – Sources adaptation • Frame rate test preparation • Resolution fixed to the Smartphone's one : qHD (960x540) • Initial 1080i50 contents are transformed to 1080p50 using TDeint filter. • The frame rate is then divided by /2, /3, /4, /5 and /6 to obtain different frame rates without impacting smoothness and avoiding using frame interpolation. • The result frame rate are : • 25 fps • 16,6 fps • 12,5 fps • 10 fps • 8,33 fps Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  8. Test description (3/6) – Sources adaptation • Resolution test preparation • Frame rate fixed to 25 fps • Initial full HD per view content is transform to Side by Side using lanczos3 filter. • The resolution is then downscaled (lanczos3) to 5 different levels compatible with MPEG compression. • The result resolutions are : • WQVGA (320x176) • WHVGA (480x272) • WVGA (848x480) • qHD (960x540) • 720p (1280x720) Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  9. Test description (4/6) – Encoding • Since Smartphone's hardware doesn’t allow the playback of uncompressed content, encoding process is a mandatory step. • Encoder Solution • Software encoder : Ateme Kyrion File Encoder • Encoder common features: • Constant Bit Rate (CBR) • MP4 File format • Source encoding profile : • MPEG4 AVC/H.264 at 5Mb/s CBR • Depending on the resolution of the content, the Profile/Level change to Baseline@L3.0 to High@L4.0 Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  10. Test description (5/6) – Subjective test • Smartphone based tests • Test software adapted to Android Mobile • Test on a Sharp Aquos Phone 3D (SH80F) : • Android 2.3.4 • qHD (960x540) screen definition • 4.3” screen size, 262 ppi • Using Cell Matrix Parallax Barrier • Realtime 2D to 3D conversion • 3D depth slider • No fixe distance between the device and the tester’s eyes Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  11. Test description (6/6) – Subjective test • SAMVIQ (ITU-R BT 1788) method • Continuous scale (0-100) graded with following french quality items : Mauvais (Bad), Médiocre (Poor), Assez bon (Fair), Bon (Good), Excellent (Excellent) • Adapted method discriminating quality levels and providing accurate quality scores • 28 Observers • 4 to 9 rejected observers in the worst conditions according the thresholds used. See annex Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  12. Part 3 Tests results Orange Labs - Research & Development – High Speed Access Video profiles – 26th Jan. 10 interne Groupe France Télécom

  13. Subjective Test results (1/5) – FPS (82% of correlation) • Good quality • starts at 10fps for Cartoon and Interview • not before 16,6fps for Football • Only 19/28 of testers kept Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  14. Subjective Test results (2/5) – FPS (75% of correlation) • Results quite similar but overall quality slightly decrease. • Cartoon and Interview notations at 8,33fps are closer • 22/28 of testers kept Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  15. Subjective Test results (3/5) – DEF (82% of correlation) • Excellent quality • obtained from WVGA with Cartoon and Interview • only reached with the hidden reference for Football • 25/28 of testers kept Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  16. Subjective Test results (4/5) – DEF (82% of correlation) • Very low (<60) explicit reference notations deleted : • overall quality increased • better “trusty range” at 95% • 24/28 of testers kept Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  17. Subjective Test results (5/5) • FPS global analysis • Minimum FPS required for Cartoon and Interview is 10fps whereas Football needs at least 16,6fps • Football is more than an item of quality under the other contents until the 16,6fps profile. • Definition global analysis • Poor to fair quality with WQVGA and WHVGA profiles whatever contents used • Strong gap with WVGA profile to reach good to excellent quality. • Then other profiles (qHD and 720p) don’t improve quality that much except for football with 720p profile. Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  18. Part 4 Conclusion Orange Labs - Research & Development – High Speed Access Video profiles – 26th Jan. 10 interne Groupe France Télécom

  19. Conclusion – R&D recommendations • Tests conducted to assess the impact of frame rate and definition when using 3D side by side contents on 3D Smartphone. • The results of this test show that Cartoon and Interview contents can be associated in terms of frame rate and definition whereas Sport content need is own specifications. • Encoding framerate and definition for a good experience : • For Cartoon and Interview : 10fps and WVGA profile. • For Sport : 16,6fps and WVGA profile • For a better experience, an encoding profil using 16,6 fps is required. • Next Step : • 3D full quality per view • Evaluate 3D quality / encoding profile Orange Labs - Research & Development – 3D mobile video subjective tests – 28th Oct. 11

  20. Thank you Gilles TENIOU: gilles.teniou@orange.com Julien LIBOUBAN: julien.libouban@orange.com Jérôme BRIARD: jerome.briard@orange.com

  21. Annexes

  22. End-user perceived image quality • Main goal To be acquainted of the consumer (non-experts) opinion for a target service by evaluating the image quality displayed on their own receivers, knowing that image quality mainly depends on the compression and transmission effects. • How To carry out image quality subjective assessments, involving consumers • Procedure To express each observer (consumer) opinion by assigning a score to determine quality levels according to predefined items over a quality scale of votes

  23. Subjective method (1) • SAMVIQ (Subjective Assessment Methodology of Video Quality) • Context • Discrimination of quality for multimedia • Information on the method • Standardized at EBU and in progress at ITU-R • Continuous quality scale • Relatively short duration of tests • Global score per image sequence • Goal • Qualification of algorithms • Qualification and comparison of codecs • Potential of image quality of a network • Panel • 20 observers

  24. Subjective method (2) • SAMVIQ Main advantages • A few number of scenes and algorithms are required to get a discrimination of quality • Very good sensitivity • Steady and repetitive results • One score per sequence • Explicite reference image • Random access and short duration per sequence (10-15s) • SAMVIQ Main disadvantages • No information about temporal variation of quality inherent in global score results

More Related