Image Scaling

1. Image Scaling Jackie Van Ryzin http://compsci.snc.edu/cs460/vanrjl

2. Outline • Project Description • Definition & Requirements • Solutions (what I did) • Exceptions (what I didn’t get to) • Methodology (how I did it) • Demonstration – Zooming fun! • Learning & Development Process • Strategies (what works for me) • Knowledge (useful CS concepts) • Extensions (where to from here) • Advice (for those yet to come) • Q & A

3. Project Description • Develop algorithms for real-time scaling that maintains quality • Study and implement smooth Bresenham nearest neighbor interpolation method • Look at ways to improve these methods • Develop new algorithms or alter existing • Implement a user interface that would allow someone to zoom in and out as they watch a streaming video

4. Overall Plan • Read, research, understand existing algorithms • Implement those algorithms using general arrays • Create program to read and alter images • Combine image program and algorithms • Develop interface to test and compare algorithms and variations of existing one • Alter programs & interface to scale real-time streaming video in addition to still images

5. In the beginning… • Thiadmer Riemersma was unhappy with: • Quality of fast image scaling algorithms • Slow processing for high quality scaling routines • He developed smooth Bresenham algorithm (Dr. Dobb’s Journal, May 2002) • Used concept of Bresenham’s line algorithm to scale a horizontal line of pixels

6. Background in Bresenham • Determine which points of an n-dimensional grid to plot to best approximate a straight line between two given points (x0,y0) and (x1,y1) • For each x between x0 and x1, choose row y that most nearly matches fcn value to plot pixel (x,y)

7. To determine y • General line formula is • Know column x so solve for row y as • Can pre-calculate constant • Since slope between 0 & 1 by assumption, after rounding we either use same y or add 1 to it

8. Tracking Error • User error value to track difference between current y value and exact y value for current x • As increment x++, increase error by slope • When error > 0.5, line is closer to next y value rather than current/previous • y++ and error--

9. Bresenham for Images • Pixels are picked up from discrete positions in source image and placed at discrete locations in the destination image • Nearest neighbor sampling • Magnifying: pixels duplicated • Minifying: pixels dropped • Both leave artifacts in destination image • More accurate sampling requires reading pixels from fractional positions in source image • Techniques exist that incorporate weighted averages – computation-intensive

10. A “Lightweight” Alternative • Compromise between linear interpolation & coarse Bresenham scaling • Destination pixel set to: • Value of closest source pixel • Unweighted average of neighbors • Based on proximity to source pixel Original 150% 75%

11. Smooth Bresenham (cont’d) • Merit: • Standard (unweighted) averages = simpler calculations • Increased performance speed – important for real-time applications • Dropped pixels & excessive “jaggies” of fast nearest-neighbor techniques (such as coarse Bresenham) are absent or kept to a minimum • Adjust this algorithm for 2D images • Nested for loops • Call horizontal line scaling algorithm • Average scaled lines with each other

12. How we use this… • OpenCV Platform • Images in IplImage format which has char* pointing to data • Data stored linearly as bytes as shown • Use CV’s split fcn to get 3 single-channel images from a 3-channel image • Run algorithm on each before remerging with CV’s merge fcn • CAREFUL with averaging!

13. Zooming! • At first could capture keys (used 1 & 2) with cvWaitKey for user to scale up or down • Discovered cvTrackBar • Attaches nicely to window • Auto updates the variable (scale factor) • Very user-friendly & visually appealing!

14. Analysis • First scale image using Bresenham, then “undo” the scaling by using Bresenham on scaled image to get back to original size • Compare with original image (calculate average difference between pixels of original and “un-scaled” images) • Do same with cvResize functions & compare results

15. I Wish I Could/Would Have... • Been better able to convert bytes to ints and back • Would have condensed processing time by calling algorithm only once (on 3-channel image rather than separately on 3 1-channel images) • Been able to implement more alterations to the algorithm for comparison and analysis • Looked into MIP mapping as mentioned in DDJ • Quick scaling by factor of 2 • Used actual calculations to determine speed rather than simply visual

16. Methods • Some mini-lectures/discussions with Dr. Pankratz & Dr. McVey in beginning to understand basic algorithm • Test algorithms on arrays of ints first! (it was a while before I tried images) • Tried using VB at first because Professor Blahnik provided code for reading from a webcam • Converting algorithm into VB (pointers) • DLL to leave it in C code • Picture box didn’t resize when image did

17. Methods (cont’d) • OpenCV • Started with sample program from developers • Strip it down to minimum with John’s help • Add features and functions in baby steps • Always had two versions • Still images using LoadImage • Webcam using Capture • Wrote functions in small test program to run on arrays of ints before adding to interfaces

18. Let’s Scale! • Real-time scrolling from webcam • HALLELUJAH!\scroll_webcam\scroll_webcam.dsw • Compare with CV resizing using still image • HALLELUJAH!\diff_cv\diff_cv.dsw • How well does it “undo” the scaling? • HALLELUJAH!\rescale_cmp\rescale_cmp.dsw • Does a different threshold matter? • HALLELUJAH!\cmp_threshold\cmp_threshold.dsw

19. Where did this stuff come from? • Dr. Dobb’s Journal • OpenCV manual • Dr. Pankratz & Dr. McVey • Forums and online tutorials for OpenCV • Practice, experimenting, trial-and-error • John, Ted, Ryan P

20. Important CS Concepts • Data Structures • Arrays, pointers, structures, chars vs. unsigned chars! • Dynamic memory allocation & de-allocation • Machine Org. & Assembly Language • Fast calculations for dividing & masking • Programming Languages • Designing abstractly without a language • All classes • General learning, experimenting, researching techniques

21. Where to From Here? • Be able to convert to array of ints & back to reduce calls to scaling algorithm • Test & compare speed using actual clock calculations • Find a better mathematical/computational method for comparing how “nice” a picture looks scaled • Alter algorithm in some way to improve quality and/or speed • Investigate MIP mapping and using a combination of MIP & Bresenham

22. For Those Who Follow • Meet with DCP and other profs OFTEN and right away • Ask questions of anyone and everyone • Someone may have a unique solution you wouldn’t think of • Could make you discover a solution with a simple comment about how chars in C aren’t fun • When you get stuck, admit it and ask for help • Write your journal! It keeps you accountable and encourages progress. • Paper can be useful! Draw images, diagrams, etc. It helps to see it before you program it • Baby steps…starting small is easier, and every little milestone is rewarding. It also makes testing & debugging easier.

23. Q & A Thanks to Dr. Pankratz and Dr. McVey for the guidance and ideas! Thank you to classmates for ideas, answers, and suggestions. http://compsci.snc.edu/cs460/vanrjl