Hybrid Bit-Stream Models (June 28 – July 2, Krakow)

1. Hybrid Bit-Stream Models (June 28 – July 2, Krakow)

2. Main Issues • Determining the working system • Ericsson/SQ: QVGA? • Yonsei: Supports all video formats. Needs a commercial codec for HD. • SRC/NDA • Validation data (ILG or ILG/Proponents) • Design of validation data: ILG or Proponents • Training data • Schedule

3. Main Issues: Working systms • Determining the working system • Ericsson/SQ: QVGA? • Yonsei: Supports all video formats. Needs a commercial codec for HD. • Decisions: Tentatively agreed to use the following working system provided it is functional. • Reference decoder: JM16.1 (modified by Ghent) • Encoder & Server: Xstreamer (written by Ghent, open source, available at http://xstreamer.atlantis.ugent.be) • Capturer (capturing & removing headers): Xstreamer • H264StreamGenerator(tracesplay): receives pcap files, remove headers and generate Annex.B bit stream, which can be decoded by JM16.1. Verification will be done in the following weeks. At least, one party needs to confirm that the working system is functional to finalize the working system (BT, NTT, Yonsei).

4. Main Issues: Test materials & Subjective tests • Multimedia & SD • License expired • Renewal • Use of the MM data • HD: HDTV project.

5. Main Issues • Validation test (ILG or ILG/Proponents) • Design of validation data: ILG or Proponents • Decision: (Jens)

6. Main Issues • Training data • Models will be prepared for all kinds of bit stream data generated by other proponents and ILGs. • Provides proponents clear understanding what kinds of impairments they should expect. • Decision: A limited number of SRCs will be used to generate a variety of PVSs and bit stream data.

7. Main Issues • Recommended SRCs (MP) www.cdvl.org “Angled stadium walkway, set B (some people) version 5” 1080i 59.94 fields-per-second. Find with key words: +stadium +walkway +B +5 www.cdvl.org “NTIA outdoor mall with tulips (3e)” 1080i 50 fields-per-second. Find with key words: +NTIA +outdoor +3e ftp://vqeg.its.bldrdoc.gov/HDTV/NTIA_source/ SnowMnt_8bit.avi 1080p 29.97 frames-per-second (We will need to use a scene from VQEG HDTV for 1080p 25 frames-per-second.  No such scene is currently available on CDVL, but I can upload one).

8. Main Issues • Schedule (Tentative) - Working system finalization: Oct. 2010 - Approval of testplan: next VQEG - Training data exchange: next VQEG + 3 months - Model submission: next VQEG + 6 (est. July 2011) - Final report (est. July 2012)

9. ILGs 5 • CRC 1 – not present • INTEL 0 • Acreo 1 • IRCCyN 1 • FUB (maybe) • NTIA 0 • AGH 1 • Ghent 1

10. Proponents: 6 OPTICOM 1 Yonsei 1 Symmetricom ? NTT 1SQ 1 DT 1 (maybe) BT 0 KDDI (maybe) 1 Psytechnics (maybe) ??: Lancaster Univ , Tektronix, Psytechnics Q: Collaboration or competition? (Hybrid, JEG)

11. Competition vs Collaboration • Pros • Cons

12. Hybrid Bit-Stream Models (January 25-29, Boulder)

13. Issues to be determined: Testplan • The testplan is revised using the outputs of the Berlin meeting. • Decision: Accepted

14. Issues to be determined Model types and input requirements

15. Issues to be determined: HRC p33 • Maximum transmission error? • Option: Any transmission errors will be allowed as long as the corresponding PVSs meet the calibration limits. • Decision: Any transmission errors will be allowed as long as the corresponding PVSs meet the calibration limits.

16. Issues to be determined: Calibration limits p30 • Spatial shifts • Gain/Offset • maximum allowable deviation in luminance gain is +/- 20% (Recommended is +/- 10%) • maximum allowable deviation in luminance offset is +/- 50 (Recommended is +/- 20) • maximum allowable Horizontal Shift is +/- 8 pixels for QVGA, +/- 16 pixels for SD/HDTV (Recommended is +/- 1) • maximum allowable Vertical Shift is +/- 5 lines (Recommended is +/- 1) • No PVS may have visibly obvious scaling. • The color space must appear to be correct (e.g., a red apple should not mistakenly rendered be rendered “blue” due to a swap of the Cb and Cr color planes). • No more than 1/2 of a PVS may consist of frozen frames or pure uni-color frames (e.g., from over-the-air broadcast lack of delivery). • ??Pure uni-color frames (e.g., from over-the-air broadcast lack of delivery) must not occur in the first 2-seconds or the last 2-seconds of any PVS. The reason for this constraint, is that the viewers may be confused and mistake the uni-color for the end of sequence. • Decision: We keep the required limits only. Some changes were made. Maximum Shift is +/- 8 pixels for QVGA, +/- 16 pixels for SD/HDTV

17. Issues to be determined: # of PVS P17 • How long will be the session length for 15s PVS? • How many PVSs are there per session for 16-24s PVS? • Decision: For 10s, 160; For 15s, 120; For 16-24 (ave 20), 90. The number of common set is changed to 24.

18. Issues to be determined p19 • Test design? Full matrix? • Decision: ILG will determine the test conditions.

19. Proponents & Preferred input types p22 (1) Hybrid models 8

20. ILGs 3 p22 • CRC 1 • INTEL 0 • Acreo 1 • IRCCyN 1 • FUB (maybe) • NTIA 0 • AGH 0

21. Remaining issues: subjective test p22 already written • Subjective test • Multimedia project approach: proven, but delays may happen (about 6 month delay after model submission). Easy to correct errors • HDTV project approach: Supposedly fast, but there can be impasse if serious PVS problems occur. Difficult to correct errors. • Subjective test • ILG only:5 • ILG and Proponents: 8

22. Issues to be determined: Test materials & Subjective tests • Multimedia & SD • License expired • Renewal • Use of the MM data • HD: HDTV project.

23. Target usage (PLR): TBD • SD/HD: (information from Broadband forum may be useful) • QCIF/VGA - internet: • mobile: 0-15% • DONE

24. Issues to be determined p34 (from Berlin) Encoders & Decoders • Decision • Number of reference decoders (for compatibility check): 1 reference decoder per codec. • Number of encoders: any encoders compatible with the reference decoder. It is preferred that more than one encoder is used. • Number of decoders (for subjective tests and inputs to hybrid models): any decoders compatible with the reference decoder. It is preferred that more than one decoder is used.

25. Working Systems • Ericsson System • JM decoder • Yonsei System • JM decoder • Decision: Tentatively, the JM decoder will be the reference decoder. Profiles and test conditions will be further investigated. Working group (OPTICOM, NTT, Symmetricom, Ericsson, Yonsei, KDDI, AGH). • Goal: finalize the working system by the next VQEG meeting.

26. 11 scale vs. 5 scale p11,85 • Quick Vote on the 5-scale vs. 11 scale. • Decision: • In favor of the 11 scale: 3 • In favor of the 5 scale: 12

27. Issues to be determined: Codec • We specify profiles. • Suggestions for H.264: • QVGA in baseline • SD in baseline and main • HD in main,. • Suggestions for MPEG-2 • SD in main and high profile. • Decision:

28. Data Analysis • Same as the multimedia project –tentatively agreed. • TBD: common set [TBD] Same as in the HDTV project with the same constraints and conditions. The common set will be used for data analysis for each test.

29. Schedule • Testplan • Decision:

30. Hybrid Bit-Stream Models (June 22-26, 2009 Berlin)

31. Issues to be determined: Testplan • The testplan is revised using the outputs of the San Jose meeting. • Decision: approved

32. Issues to be determined: Numerical values / p11 • Decision: Figure .2/P.910 will be used with numerical values displayed. Subjective scores should be inputted as integer numbers (0-10). The input methods for subjective scores include, but are not limited to, the following: • By checking one of 11 bins (computer or paper) • By entering an integer number (0-10) (computer or paper) • By moving a sliding bar which takes one of 11 discrete positions. (computer)

33. Issues to be determined: Numerical values

34. Issues to be determined: SD on HDTV monitor / p14 • For the HD/SD testing, the video will use the full screen dimensions and no background panel or black border will be present. If a HD LCD monitor is used for SDTV testing, the picture area should be centered and the non-picture area should be black or mid-level gray. • Decision: The following text is accepted: “If a HD LCD monitor is used for SDTV testing, the picture area should be centered and the non-picture area should be black or mid-level gray.”

35. Issues to be determined: common set p19 • The PVSs include both the common set of PVSs inserted in each experiment and the hidden reference (hidden SRCs) sequences, i.e. each hidden SRC is one PVS. The common set of PVSs will include the secret PVSs and secret source. • Decision: Common set will be used (30 PVSs)

36. Issues to be determined: Test materials • Multimedia & SD • License expired • Renewal • Use of the MM data • HD: HDTV project. • Results: Most of MM data can be renewed.

37. Remaining issues: (scope) Number of decoders p36 • For hybrid models, multiple decoders/players can be used to generate PVSs as long as the decoders can handle the bit-stream data which the reference decoder can decode. Bit-streams data can be generated by any encoder as long as the reference decoder can decode the bit stream data. (ACCEPTED) • For the case of bit-stream 1 & 2 models, PVSs should be generated using the reference decoder. • The multiple decoders should not generate PVSs which substantially differ in quality for the same bit-stream data. • Decision: The following text will be inserted: “For hybrid models, multiple decoders/players can be used to generate PVSs as long as the decoders can handle the bit-stream data which the reference decoder can decode. Bit-streams data can be generated by any encoder as long as the reference decoder can decode the bit stream data.”

38. Issues to be determined Model types and input requirements

39. Issues to be determined: Working System & Reference Decoder • The working system • Ericsson-SwissQual (San Jose, Jan. 2009): some part are missing • Yonsei system (Berlin, June 2009): complete (H.264) • Working group for develop and verify the working system? • Input format (files) • Output format • Target date: • Decision: Ericsson, SwissQual (to be confimed), DT and Yonsei will form a working group to develop and verify the working system, and report the candidate system (and the reference decoder) at the next meeting.

40. Issues to be determined: HRC • Maximum transmission error? • Option: Any transmission errors will be allowed as long as the corresponding PVSs meet the calibration limits. • Decision:

41. Remaining issues: Calibration limits • Temporal limit: decided at the last meeting. • currently same as HDTV & Multimedia (will be decided over the reflector)

42. Decision on temporal limits (10s)p34 • Max. length of freezing or skipping (10s PVS)? • 40% of the total length? • Decision: • maximum of freezing within 10s: 3s (mandatory 5s) • maximum of skipping within 10s: 3s (mandatory 5s) • max total loss (including beginning and end): 1s (mandatory 2s) • (max total loss=max loss at begin + max loss at end) • max total extra (including beginning and end): 1s (mandatory : the entire PVS must be contained in the SRC). • Anything can happen in-between (freezing with/without skipping, skipping as long they meet the aforementioned conditions).

43. Decision on temporal limits (15s) p34 • Max. length of freezing or skipping (15s PVS)? • 40% of the total length? • Decision: • maximum of freezing within 15s: 3s (mandatory 5s) • maximum of skipping within 15s: 3s (mandatory 5s) • max total loss (including beginning and end): 1s (mandatory 2s) • (max total loss=max loss at begin + max loss at end) • max total extra (including beginning and end): 1s (mandatory : the entire PVS must be contained in the SRC). • Anything can happen in-between (freezing with/without skipping, skipping as long they meet the aforementioned conditions

44. Issues to be determined: Calibration limits • Spatial shifts • Gain/Offset • maximum allowable deviation in luminance gain is +/- 20% (Recommended is +/- 10%) • maximum allowable deviation in luminance offset is +/- 50 (Recommended is +/- 20) • maximum allowable Horizontal Shift is +/- 5 pixels (Recommended is +/- 1) • maximum allowable Vertical Shift is +/- 5 lines (Recommended is +/- 1) • No PVS may have visibly obvious scaling. • The color space must appear to be correct (e.g., a red apple should not mistakenly rendered be rendered “blue” due to a swap of the Cb and Cr color planes). • No more than 1/2 of a PVS may consist of frozen frames or pure black frames (e.g., from over-the-air broadcast lack of delivery). • ??Pure black frames (e.g., from over-the-air broadcast lack of delivery) must not occur in the first 2-seconds or the last 4-seconds of any PVS. The reason for this constraint, is that the viewers may be confused and mistake the black for the end of sequence. • Decision:

45. Issues to be determined: Codec • We specify profiles. • Suggestions for H.264: • QVGA in baseline • SD in baseline and main • HD in main. • Suggestions for MPEG-2 • SD in main and high profile. • Decision:

46. Issues to be determined: # of PVS • How long will be the session length for 16-24s PVS? • How many PVSs are there per session for 16-24s PVS? • Decision:

47. Issues to be determined • Test design? Full matrix? • Decision: TBD

48. Data Analysis • Same as the multimedia project –tentatively agreed. • TBD: common set

49. Remaining issues: subjective test • Subjective test • Multimedia project approach: proven, but delays may happen (about 6 month delay after model submission). Easy to correct errors • HDTV project approach: Fast, but there can be impasse if serious PVS problems occur. Difficult to correct errors.

50. Issues to be determined p34 Encoders & Decoders • Decision • Number of reference decoders (for compatibility check): 1 reference decoder per codec. • Number of encoders: any encoders compatible with the reference decoder. It is preferred that more than one encoder is used. • Number of decoders (for subjective tests and inputs to hybrid models): any decoders compatible with the reference decoder. It is preferred that more than one decoder is used.