Master’s Project Defense Bill Champlin University of Colorado at Colorado Springs Dept of Computer Science

1. Master’s Project DefenseBill ChamplinUniversity of Colorado at Colorado SpringsDept of Computer Science Java Quasi-Connected Components (JQCC) Tracking System April15, 2010 Advisor - Dr. Terrance Boult

2. Agenda • Overview/Scope • Approach • Activities • Analysis & Design • Re-host Effort • Port to Linux • Effort &Performance Metrics • Adapting QCC to a new Domain • Lessons Learned/Issues/Recommendations • JQCC Demonstration

3. Project Overview • JQCC re-hosts Quasi-Connected Components (QCC) portion of the Lehigh Omni-Directional Tracking System (LOTS) to Java • Object tracking software used to track snipers, ships, UAVs, etc • Dr. Boult wanted a version to post on the web • Means Java for maximum portability • Need to prove that by running different platforms • Would Java be fast enough? i.e. 30 FPS?

4. Approach • Options: write from scratch or port QCC portions of LOTS QTRACK? • From scratch, had these obstacles: • Had no track region or “cluster” association algorithm • Legacy had numerous special cases to handle tracking issues - would be missing those • Porting LOTS QTRACK daunting • QTRACK baseline ~ 96 KSLOC of C++ • “You won’t understand what a lot of what the code does” – Dr. Boult • Complex, undocumented and sparsely commented • Supports various input sources, architectures i.e. Intel MMX and camera types • A lot of dead code • Chose to port QTRACK • Knew it worked – just had to translate? • Not quite – a lot of architecture work first

5. Activities • Obtain LOTS s/w baseline (QTRACK) and get running • Extract core functionality (QCC) • Develop requirements and UML design • Port LOTS core functionality to Java (JQCC): • Verify execution under Windows and Linux platforms • Evaluate performance vs. legacy QTRACK • Adapt to tracking night sky images • Generate promised “artifacts” • SRS, Certification Specification, and User Handbook • Project Report • Project Website: http://jqcc.pbworks.com/

6. Analysis Model • Requirements broken out by JQCC Use Cases:

7. Design Model Concurrency diagram depicts top level process/thread allocation

8. QCC Technique (from 20K ft) QCC tracks targets by: • Building a background model • Blends input frames into primary and secondary “reference” images ref = (1-alpha)ref + alpha*ref // big alpha blends faster • Primary ref = ref closest to input frame pixel and will be used for thresholding – means a more accurate threshold • Thresholding / Connecting Thresholded Pixels • Identifies candidate target pixels exceeding dynamic high threshold • Sum them into a 4x4 reduced “Parent” image • Identify neighbors that exceed a lower threshold around target pixel • Sum into into Parent image (form small blobs or “regions”) • Adjust dynamic threshold for next time based on pixel count exceeding high threshold

9. QCC Technique-Con’t • Parent regions are grown and “connected” to those from last parent - get target ID (Label) from past parent if they overlap a region from last time • All pixels of the same region are “labeled” with the same target id • Regions are matched with previous overlapping and non-overlapping regions • Regions that are too sparse or to big (due to lighting shadows/etc) are “cleaned” (discarded) • Regions are assigned a “confidence” based on density, shape, speed, etc and classified “tracks” if they exceed confidence threshold • Every nth frame, reference and variance frames are updated

10. QCC Performance Boosters • Thresholding, swapping references done over column segments of each row (cords) • Reference frames only blended every 16th frame or so (configurable) • Input buffer “absorbs” periodic slow downs • Parent image reduces pixels to process • Only parent rows with used pixels are processed

11. Porting to Java Approach • Keep legacy track class structure where possible • Preprocessor statements conversion • #defines variables to static final or static variables • Groups of related #defines converted to Java enums • #define macros to functions • #if, #else, #endif converted to if, if-else blocks • Had to put debug in QTRACK to figure out what defines were “TRUE”

12. Porting to Java Approach – Con’t • Convert unsigned char to byte, etc. • Save memory over unsigned char to short or int • No unsigned primitive types in Java – use masks • Example: ucharval=0x8F;int result = 0xFF – val; Convert to: byte val=0x8F;int result = 0xFF – val; • Error: want - 0x70, get - 0x170 Java converts byte val to -113, then subtracts – logic error • Instead use: byte val=0x8F;int result=0xFF-val&0xFF; • Error prone – QTRACK uses a lot of compound expressions • Java right shift “>>” is a “signed” shift, shifts in 1’s if value being shifted has sign bit set • Use “>>>” instead to shift in 0’s from left to match C++

13. Porting to Java Approach – Con’t • No pass by reference “&” in Java – only by value • Convert reference or “out” arguments to “wrappers” public class ByteWrapper{ public byte val;} • Caller sets val , method gets val inside, method sets val before returning, caller retrieves val after method returns • Autobox types i.e. Integer, Double, etc. don’t work for this • No pointer arithmetic in Java • QTRACK uses pointer arithmetic extensively • Convert pointers to use indexed arrays and do math on the indexes • Error prone separating ptr math from “content” math • Convert static function variables to static class fields

14. Code Metrics

15. Performance Comparison

16. Port to Linux • Goal – a portable, ready-to-run solution that doesn’t force the user to download and install a lot of packages (preferably nothing but Java) • Biggest issue – getting movies played/injected on Linux using Java • JMF pure Java version advertised as runnable under Linux • Unable to run JMF MPEG Camera Simulator under RedHat Enterprise 5.0 • Examined other options: • XuggleXuggler (licensing issues / involved to build source on Linux) • FOBS4JMF (requires FFMPEG, other C libs) • Finally used Java Advanced Imaging (JAI) to load JPEG image sequences • Created a 2nd version of the Camera Simulator (JPEGCameraSimulator) • Same JQCC class files now run under Linux • Startup bash scripts only difference – can run same Java class files as MS Windows • Verified under both Ubuntu 9.10 and RedHat Enterprise 5.0

17. Adapting JQCC to new Domain • Domain chosen: Tracking objects in night sky images • Difficulty finding sample movies • Web searches not that productive • NASA JPL Search • Very short videos of Martian moons Phobos & Deimos • GALEX Space Telescope Debris Movie [5] • Ran across Kevin Fetter’s web site[3] www.kfetter.com • Movie camera telescope setup http://www.kfetter.com/setup/videosetup.htm • Had no images when I contacted him • Collected several sets of satellites for me moving at various directions and speeds (GEO, Near Earth, Blinker) (has collected a lot more since then) • Initially, JQCC would track only one of the fast satellites • Modified parameters to get it to work for most images

18. Lessons Learned • Java weaknesses • Movie support • No unsigned types • Garbage Coll. causes display updates to “pause” • Can’t avoid it totally – JVM / open source packages • GC runs longer the more memory there is allocated • Work-arounds • Use incremental “-Xinc” option to help GC “pauses” • Consider using direct memory allocation to allocate memory outside of JVM so GC doesn’t check it • Using signed types with masking is error prone • Predicates with final constants leads to “dead code” • Promise of Java Portability Holds • Write once, build once, run anywhere (with a JVM) works

19. Issues/Recommendations • Modify JPEG Camera Simulator to “read ahead” instead of loading all frames into memory • Find and fix possible dynamic thresholding issue • Examine tracking stability • Improve reusability - finish supporting all tuning parameters in calibration file and on GUI; remove scenario specific processing • Finish debugging rectangle files for “ignoring” regions and outside of a radius for omni-directional images • Re-factor into an OO design; add more / better comments • Improve display performance / fix “update” color artifacts • Implement remaining capabilities

20. Backup Slides

21. Demonstration ScriptSatellite images courtesy of [4], GALEX Debris courtesy of [5] • Low earth satellite • Rate Adjustment / Tracking Parameters • Vertical Object • MPEG GEO • Blinker • Linux low earth satellite • Side by side comparison • Frame Rates • GALEX Debris • Toolbag

22. Class Diagrams

23. Differencing For every “used” pixel i primaryDiff = |inim[i] - refim1[i]| secondaryDiff = |inim[i] - refim2[i]| // use closest difference if secondaryDiff < primaryDiff diff = primaryDiff swap(refim1[i], refim2[i]) // keep bkgrnd closest to last input as primary thresh = dynThresh + varim[i] if diff[i] > thresh // threshold difference frame outim[i] = diff else outim[i] = 0 //Build parent image (see next slide for what a parent image is) For pos in every parent pixel For j in every image pixel pos that maps to parent if outim[j] parent[pos]++ if outim[i]-delta > lowThresh parent[pos] += 256 // Increments count above low threshold

24. Differencing - Con’t • A parent is a smaller image that summarizes regions across the larger FOV • Each parent pixel maps to a 4x4 pixel in main image • Each parent pixel is a short which counts pixels above high threshold and low threshold at same time • Upper 8 bits count of pixels > High threshold • Lower 8 bits sums pixels > Low threshold • Smaller image = faster processing for subsequent processing i.e. growing regions

25. High Level Track Processing

26. Discards regions that are too small Track Processing Associates region with past targets that overlap in last update See Differencing slide Dynamic threshold adjusted up based on count of differenced pixels > threshold Associates region with past regions that have gaps between them Parent regions grown and matched to overlapping regions from past parent Associates region with regions older than last update Assigns a common label (integer value) to all parent pixels of a region Discards regions that are too big or sparse Reducing threshold around target improves tracking Regions sorted by confidence for next update comparison Confidences converted to priority which dictates next threshold reduction Input frame blended into ref. frames and variance frame updated to be slightly bigger than background

27. Metrics – Con’t

28. Metrics – Con’t

29. COCOMO II JQCC Effort EstimateUsing COSTAR[4] Tool

30. More Lessons Learned • Threading with SMP architectures improves performance • Forget about design and code “zealousies” • Do what’s simplest when translating code • Using Java Enum wasted time over constants • Had to create code and convert using code • Forget about making object oriented initially • Don’t disturb the existing design that works; refactor later

31. References • [1] T.E. Boult, R.J. Micheals, X. Gao, M. Eckmann, “Into the woods: visual surveillance of non-cooperative and camouflaged targets in complex outdoor settings”, in Proc. Of the IEEE, Oct. 2001 • [2] T.E. Boult, T. Zhang, R.C. Johnson, “Two threshold are better than one”, CVPR, pp.1-8, 2007 IEEE Conference on Computer Vision and Pattern Recognition • [3] K. Fetter, Astronomical Images, www.kfetter.com • [4] Softstar Systems, Costar Software Estimation Tool, www.softstarsystems.com • [5] GALEX Science Team - California Institute of Technology, GALEX Space Telescope Movie, www.galex.caltgech.edu