1 / 41

Spatio-Temporal Quincunx Sub-Sampling

Spatio-Temporal Quincunx Sub-Sampling. . . and how we get there David Lyon. Overview. Sampling in Television and Film The problems of aliasing Filtering requirements Conversion between differing formats Problems that can occur

shlomo
Download Presentation

Spatio-Temporal Quincunx Sub-Sampling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Spatio-Temporal Quincunx Sub-Sampling . . and how we get there David Lyon

  2. Overview • Sampling in Television and Film • The problems of aliasing • Filtering requirements • Conversion between differing formats • Problems that can occur • How we can mitigate some of the problems and maintain or improve quality

  3. Sampling Theory • Harry Nyquist – 1889 to 1976 • “The number of independent pulses that can be put through a telegraph channel per unit time is limited to twice the bandwidth of the channel”

  4. Sampling Theory • Harry Nyquist – 1889 to 1976 • “The number of independent pulses that can be put through a telegraph channel per unit time is limited to twice the bandwidth of the channel” • Later Nyquist-Shannon • “Exact reconstruction of a continuous-time baseband signal from its samples is possible if the signal is bandlimited and the sampling frequency is greater than twice the signal bandwidth”

  5. Amplitude Fs Frequency Sampling Theory

  6. Amplitude Fs Frequency Sampling Theory • Audio: • 20kHz bandwidth, Fs = 44.1kHz, 48kHz

  7. Amplitude Fs Frequency Sampling Theory • Audio: • 20kHz bandwidth, Fs = 44.1kHz, 48kHz • Video: • 5.75MHz bandwidth, Fs = 13.5MHz • 30MHz bandwidth, Fs = 74.25MHz

  8. Amplitude Fs Frequency Aliasing Nyquist Frequency

  9. Amplitude Fs Frequency Aliasing Nyquist Frequency • Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals

  10. Amplitude Fs Frequency Aliasing Nyquist Frequency • Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals • These signals CANNOT be removed afterwards

  11. Amplitude Fs Frequency Aliasing Nyquist Frequency • Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals • These signals CANNOT be removed afterwards • Filtering BEFORE sampling is needed

  12. Image Sampling Temporal – frames Vertical - lines Horizontal - pixels

  13. Image Sampling • Horizontal resolution • Sampling rate of 720, 1280, 1920 or 2048 samples/picture width • Resulting resolution of 360, 640, 960 or 1024 cycles/pw

  14. Image Sampling • Horizontal resolution • Sampling rate of 720, 1280, 1920 or 2048 samples/picture width • Resulting resolution of 360, 640, 960 or 1024 cycles/pw • Vertical resolution • Sampling rate of 480, 576, 720, 1080 samples/picture height • Resulting resolution of 240, 288, 360 or 540 cycles/ph

  15. Image Sampling • Horizontal resolution • Sampling rate of 720, 1280, 1920 or 2048 samples/picture width • Resulting resolution of 360, 640, 960 or 1024 cycles/pw • Vertical resolution • Sampling rate of 480, 576, 720, 1080 samples/picture height • Resulting resolution of 240, 288, 360 or 540 cycles/ph • Temporal resolution • Sampling rate of 24, 25, 30, 50, 60 . . . samples/second • Resulting resolution of 12, 15, 25, 30 cycles/sec

  16. Re-sampling • Image size changes are common

  17. 1080 Amplitude Vertical Frequency Potential Alias 480 Amplitude Vertical Frequency Re-sampling • Image size changes are common • Simple example of interpolating a 1080 picture to 480: • Input resolution is 540 cycles/ph • Output resolution is 240 cycles/ph (division by 2.25) Filter

  18. Re-sampling • Interpolation is only one part of the problem • Filtering is needed to control the signal spectrum and avoid the introduction of aliases • Simple interpolators are generally poor filters

  19. Re-sampling • Interpolation is only one part of the problem • Filtering is needed to control the signal spectrum and avoid the introduction of aliases • Simple interpolators are generally poor filters • Alias terms are “folded” about the Nyquist point • Inverted in frequency, inverted “movement” • Highly noticeable to the human eye, which references its own internal 3D model

  20. Re-sampling • Interpolation is only one part of the problem • Filtering is needed to control the signal spectrum and avoid the introduction of aliases • Simple interpolators are generally poor filters • Alias terms are “folded” about the Nyquist point • Inverted in frequency, inverted “movement” • Highly noticeable to the human eye, which references its own internal 3D model • Alias terms left in the image will be shifted again in any subsequent operations • Potentially cumulative problems

  21. Restricted by practical limitations Linked by aspect ratio and pixel shape 3D Sampling Temporal – frames Vertical - lines Horizontal - pixels

  22. Spatial Frequency No of Lines Potential alias Potential alias Frame Rate Temporal Frequency Spatio-Temporal Sampling Temporal – frames Spectrum Spatial - lines

  23. Spatial Frequency No of Lines Potential alias Potential alias Frame Rate Temporal Frequency Spatio-Temporal Sampling • Filtering: • Spatial – optical LPF and lens MTF Temporal – frames Spectrum Spatial - lines

  24. Spatial Frequency No of Lines Potential alias Potential alias Frame Rate Temporal Frequency Spatio-Temporal Sampling • Filtering: • Spatial – optical LPF and lens MTF • Temporal – integration time of sensor system Temporal – frames Spectrum Spatial - lines

  25. Potential alias Potential alias Spatio-Temporal Sub-Sampling Spatial Frequency • Where is the filter? No of Lines Temporal – frames Spectrum Frame Rate Spatial - lines Temporal Frequency

  26. Horizontal ? Up-conversion Spatial Frequency No of Lines Temporal Frame Rate Vertical Spectrum Temporal Frequency

  27. Horizontal ? Up-conversion Spatial Frequency • Adaptive filtering No of Lines Temporal Frame Rate Vertical Spectrum Temporal Frequency

  28. Horizontal ? Up-conversion Spatial Frequency • Adaptive filtering • Motion compensation No of Lines Temporal Frame Rate Vertical Spectrum Temporal Frequency

  29. Film 1080p 720p 480i 1080i 1080p (24) Format Interchange Spatial Frequency 500c/ph 250c/ph 0c/ph 0c/s 15c/s 30c/s Temporal Frequency

  30. Film 1080p 720p 480i 1080i 1080p (24) Format Interchange • Conversion between formats requires care Spatial Frequency 500c/ph 250c/ph 0c/ph 0c/s 15c/s 30c/s Temporal Frequency

  31. Film 1080p 720p 480i 1080i 1080p (24) Format Interchange • Conversion between formats requires care • Mixing formats such as film and video is to be avoided Spatial Frequency 500c/ph 250c/ph 0c/ph 0c/s 15c/s 30c/s Temporal Frequency

  32. Film 1080p 720p 480i 1080i 1080p (24) Format Interchange • Conversion between formats requires care • Mixing formats such as film and video is to be avoided • 1080p down-conversion might raise new challenges Spatial Frequency 500c/ph 250c/ph 0c/ph 0c/s 15c/s 30c/s Temporal Frequency

  33. 96 Amplitude Frequency 48 Amplitude Frequency Over-sampling • Commonly applied to audio – eg 96kHz down to 48kHz • Allows the use of a high performance digital filter: Filter

  34. Over-sampling • Commonly applied to audio – eg 96kHz down to 48kHz • Allows the use of a high performance digital filter:

  35. Over-sampling • Commonly applied to audio – eg 96kHz down to 48kHz • Allows the use of a high performance digital filter: • 1080p allows similar gains for outputs of 720p and 1080i • Good temporal filtering must introduce delay

  36. Over-sampling • Commonly applied to audio – eg 96kHz down to 48kHz • Allows the use of a high performance digital filter: • 1080p allows similar gains for outputs of 720p and 1080i • Good temporal filtering must introduce delay • Film sampling at >1080 lines/ph also allows controlled down-sampling

  37. Conclusion • Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time

  38. Conclusion • Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time • Modern cameras and processing can stress the format unless care is taken

  39. Conclusion • Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time • Modern cameras and processing can stress the format unless care is taken • Imprinted alias is difficult (or impossible) to remove • Camera integration is an important filter for interlace

  40. Conclusion • Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time • Modern cameras and processing can stress the format unless care is taken • Imprinted alias is difficult (or impossible) to remove • Camera integration is an important filter for interlace • Poor anti-alias filtering leads to additional compression concatenation artefacts

  41. Conclusion • Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time • Modern cameras and processing can stress the format unless care is taken • Imprinted alias is difficult (or impossible) to remove • Camera integration is an important filter for interlace • Poor anti-alias filtering leads to additional compression concatenation artefacts • 1080p down-conversion could make the stress worse

More Related