How to guarantee end-to-end delay with graceful quality degradation for

1. How to guarantee end-to-end delay with graceful quality degradation for chroma information in color images? Chroma interpolation using windowed kriging for color-image compression- by-network with guaranteed delay Mauritz Panggabean and Leif Arne Rønningen Department of Telematics, NTNU 17th International Conference on Digital Signal Processing (DSP2011) July 6-8, 2011, Corfu, Greece

2. Outline • Background and vision • Compression by network (CbN) • A solution for CbN: windowed kriging interpolation • Experiments and results • Conclusion

3. Begin with the end in mind. Stephen Covey Background and vision • Futuristic tele-immersive environment system for real-time artistic collaborations with near-natural quality • Arrays of auto-stereoscopic 3D displays and high-end cameras on all surfaces of the collaboration space • Low overall delay for good synchronization to 11.5ms • Challenge:guaranteeing end-to-end delay with graceful quality degradation? • For very high bitrate visual information of the envisioned system? • Best effort in the Internet and source coding approach • Our proposal: compression by network

4. If everyone is thinking alike, then somebody isn't thinking. George S. Patton Compression by network (CbN)

5. Have you got a problem? Do what you can where you are with what you've got. Theodore Roosevelt Solution for CbN? • Idea: directly drop pixel values and interpolate later? • How to interpolate 8 missing only from 1 with smallest error? • Solution: Kriging • A well known interpolation technique in geostatistics • Basic idea:model the spatial distribution of the data points and estimate values at unknown points using the distribution

6. Everything should be made as simple as possible, but not simpler. Albert Einstein Kriging interpolation illustrated 64x64 Lenna image, samples 1/9 of image

7. If we all worked on the assumption that what is accepted as true is really true, there would be little hope of advance. –Orville Wright Windowed kriging interpolation • Challenge: computationally expensive • Out of memory to interpolate 128x128 Lenna image with > 5% image using Matlab DACE toolbox* with 2.99GHz processor and 8GB RAM • A proposed solution: windowed kriging (WK) WK parameters - Width (W) and height (H): 6 - NxN pixel streams: N = 3 - Retained streams in k: k = [1,9], K = 2 - SxS block: S = 5 - DxD block, D/2 = S/2 + d: d = 2 *H.B. Nielsen, S.N. Lophaven and J. Søndergaard, DACE, A Matlab kriging toolbox, Technical University of Denmark, 2009.

8. If we knew what it was we were doing, it would not be called research, would it? Albert Einstein WK for CbN

9. It is an acknowledged truth in philosophy that a just theory will always be confirmed by experiment. Thomas Malthus WK for CbN on chroma • Chroma from four 768x512 color test images • Quality metrics*: BIKES PARROTS FACE CAPS * F. De Simone et al., ”A comparative study of color image compression standards using perceptually driven quality metrics”, in Proc. SPIE, 2008

10. It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong. -Richard P. Feynman Experiment setting • Exponential distribution in DACE selected for WK • Luminance from Matlab JPEG (quality 90/100) • Without entropy coding for WK (for further work) WK Parameter values • K = 1 (the worst case) where k = [1] • S = 32 pixels • d = 3 pixels • N from 2 to 10 => Compression ratio CR = N2/K

11. No amount of experimentation can ever prove me right; a single experiment can prove me wrong. Albert Einstein Results: Cb and Cr image quality Acceptable image quality even for N = 10 (CR = 100) Due to less edges in chroma than in luminance?

12. In much of society, research means to investigate something you do not know or understand. Neil Armstrong Results: chroma 4:2:0 sub-sampling PSNR Cb PSNR Cr MSSIM Cb MSSIM Cr WPSNR WPSNRMSE WPSNRPIX N = 2 (CR = 4), 4:2:0 sub-sampling using bicubic interpolation WK performs slightly better than 4:2:0 chroma sub-sampling using bicubic interpolation

13. Research is what I'm doing when I don't know what I'm doing. Wernher von Braun Results: final output image quality Small effect of chroma interpolation using WK to final output image quality (as expected)

14. Research is to see what everybody else has seen, and to think what nobody else has thought. Albert Szent-Gyorgyi Results: processing time • Two main operations in DACE: • Constructing DACE model for distribution from data • Predicting values in unknown pixels using the model • Prediction is obviously the most expensive operation • Image resolution a bigger factor than sample number

15. All truths are easy to understand once they are discovered; the point is to discover them. Galileo Galilei Results: actual visual quality Colors like red and yellow are very sensitive to artifact due to their dominant edges in Cb or Cr images Introducing color compressibility: the maximal CR of a color with unnoticeable artifact

16. The most exciting phrase to hear in science, the one that heralds the most discoveries, is not 'Eureka!' (I found it!) but 'That's funny‘ -Isaac Asimov Results: color segmentation The difference image as the main indicator of sensitive colors in interpolation using WK for CbN

17. An experiment disproving a prediction is a discovery. Enrico Fermi Results: color compressibility CR = 25 for segments with sensitive colors improves the final quality; CR = 100 for the rest

18. We must reinforce argument with results. Booker T. Washington Conclusions and further work Contributions • Windowed kriging for chroma interpolation in compression by network • Color compressibility Further work • Subjective quality assessment • Entropy coding and comparison with chroma sub-sampling • Temporal effects in sequences of images

19. The real voyage of discovery consists not in seeking new landscapes, but in having new eyes. - Marcel Proust Thank you