The Career of Lee Holeva

1. The Career of Lee Holeva Imaging Scientist November 11, 2005 Updated August 4, 2010

2. Selected Topics • Determining Interferometer Fringe Count Error • Orbital Smear Modeling • Contrast Matching of Images • Estimation of DSL Line Speed using Neural Networks • High Speed Sorting of Magazines using KIX codes • Range from Camera Blur using Regularization

3. Fringe Count Error Estimation • Fourier Transform interferometry measures scene spectra by computing the Fourier Transform of the interferogram of an imaged scene. • A shift of the interferogram implies a phase shift of the complex spectrum: • As the interferometer operates in the IR spectrum observed spectra will have both internal and external components.

4. A Phase Function • We need a phase function characterizing just the phase change due to fringe count error. • Define:

5. Phase to Fringe Count Error • Given the sign of the square root the phase is calculated from R: • The slope of the phase with respect to frequency shows the number of fringe count errors:

6. What about the sign of the Square Root? • The linear relationship of the phase slope to fringe count error assumes a long optical path length, OPD. The phase can vary greatly from a straight line at both the beginning and end of the range of the interferogram travel. • An adaptive algorithm is used to pick the best choice of square root sign.

7. Picking the sign using RLS • For each frequency k there are two possible phases, that found using a positive square root sign and that found using a negative square root sign: • The RLS algorithm may be used to adapt the square root sign by iterating over frequency:

8. The Recursion

9. Extrapolated Phase versus True Phase

10. Variation of Square Root Signs

11. Modeling Orbital Smear • Satellites typically employ line scan sensors with time delay integration, TDI • Due to the motion of the satellite, motion blur, orbital smear, occurs both along the array, the cross scan, and along the line of scan, the along scan direction. • The modeling of orbital smear requires knowledge of the satellite’s orbit, its attitude, and the orientation of the imaging array.

12. The Satellite in Orbit Eccentricity Constant angular momentum North Pole Orbit of the satellite Inclination Angle PQW frame in the plane of the orbit Angle of Perigee Line of Nodes (descending to ascending) ECI frame (at the Earth’s center) Right Ascension of the Ascending Node Vernal Equinox

13. A sequence of Frame Conversions • Use Kepler’s third law to convert time along orbit to anomaly • Use Kepler’s first law to get the position and velocity in the plane of the orbit • Apply a sequence of rotations to go to inertial coordinates • Take into account the rotation of the Earth to go to Earth Centered coordinates

14. Slew the Satellite • From a sequence of quaternions determine the attitude of the satellite • At each time step during the scan locate the target by intersecting the line of sight vector with the surface of the Earth

15. Cross Scan Smear is Proportional to the rate of change of GSD Y • Smear along the array is proportional to fractional rate of change of G • Assuming that the rotation rates are chosen so that the TDI stages remain perpendicular with the ground track, the smear decomposes into the sum of a range term, due to changes of slant range, and a zenith term, due to rotations about the zenith vector. Even without the parallel TDI assumption, a first order analysis indicates that the sum of range and zenith terms remains at least approximately valid. Z Unit Target Zenith I is the instantaneous field of view and s is the slant range

16. Along Scan Smear is proportional to changes of the rotation rates • Along scan smear, along the ground track, is proportional to changes of both the yaw and pitch rates. However, the if the condition of the TDI stages remaining perpendicular to the ground track is maintained then the along scan smear is minimized. Changes of terrain altitude will however induce changes of the rotation rates and consequentially nonzero along scan smear.

17. Conclusions • Two simulators were developed in Matlab, the first kept the satellite fixed in position allowing the variation of the free parameters of scanning to ascertain the variations of orbital smear. • The second simulator moved the satellite along its orbit producing smear images of detector position versus time. • Critical trade study questions may now be easily answered.

18. Setting Up a Delta NIIRS Image Quality Experiment Marker MTF Marker Enhancement Kernel Tonal transfer curves cm bm Marker Image Image simulator Wiener Filter Common Source Image Wiener Filter Test Image Image simulator bt ct Test Enhancement Kernel Test MTF Tonal transfer curves

19. The Marker and Test images must have the same Eye Scale and Contrast • Eye scale is easily matched by interpolation • To get the image contrast to agree requires adjustment of the the dynamic range adjustment adder and multiplier and the selection of the appropriate tonal transfer curve

20. Nonlinear Optimization used to Match Contrast • Key Observation: the tonal transfer curves are smooth and hence derivatives may be calculated (say by the use of a 2nd order Butterworth filter) • Setup the contrast matching problem as the minimization of:

21. Only a Few Iterations are needed for each Image pair • The well known method of Levenberg-Marqurdt may be used to perform the minimization • To both speed up the process and to desensitize the match to noise, perform a block average or median followed by a decimation

22. Conclusions • A dramatic increase in productivity results: A part of the experiment setup effort that previously took weeks, if not months, is now reduced to a few hours. • The contrast matching may be embedded into the shell scripts (it is possible to call Matlab commands directly from the UNIX command line without the use of a GUI) used to create the marker and test images.

23. Estimation of DSL Line Speed using Neural Networks • A Pair of Hybrid RBF/MLP neural networks are trained to predict the maximum downstream and upstream speeds of copper telephone lines • Lines are characterized by impedance and capacitance measurements made over frequency acquired from telephone lines of varying length with and without bridge taps. • The data tends to contain significant correlations. Techniques such as the Karhunen-Loeve transform are employed to truncate the data set into its most significant parts.

24. Hybrid RBF/MLP Neural Net

25. Training the Net: Conditioning the Features • Line speed is the linguistic variable. Different basis units for lines of differing speeds should make speed estimation easier. Condition the location of basis units by fuzzy sets:

26. Training the Net: Conditional Fuzzy Clustering • Classical K-means: • Fuzzy C-means: • Conditional Fuzzy C-means: • The conditioning terms, fk, come from the predefined triangular fuzzy sets

27. The following rule is suggested: features having a cluster membership near one should be reside near the basis unit center while features with very small cluster membership values should be reside far from the basis unit center. This idea is captured by the Mahalobis distance between the feature vector and the cluster center. The centers of the basis units and the center of the clusters coincide. Using Gaussian RBF units, the width of the basis units is characterized by the covariance matrix. For this work, off diagonal correlations are ignored. A number of ad-hoc algorithms are possible to produce basis units that mimic the trained clusters. Training the Net: Size and Location of the Basis Units

28. Training the Net: Training the Output Layer • Gaussian units form the basis layer: • The predicted speed is formed by a vector matrix vector multiply: • Either the standard Levenberg-Marquardt algorithm or a regularized version may be used to train the output matrix. The use of the full feature vector on the feed-forward path may result in over fitting.

29. Speed Predictions Plotted against the Actual Maximum Speed

30. High Speed Sorting of Magazines using KIX codes

31. The behavior of the sorting machine is Postal code Dependent • Magazines move down a conveyor belt at a rate of about 30 magazines per minute • The destination of the sort is dependent upon the postal codes • Each magazine may have up to three postal codes at any orientation or position • For adequate resolution the images may be need to be quite large, 3000 by 4000 pixels is typical • The only constraint is that a minimum background contrast is guaranteed.

32. An adaptive threshold is employed to binarize the image

33. A Hit-or-Miss Transform scans the image for Candidates • The Hit-or-Miss transform uses masking coupled with logical rules to detect regular features in the binary image. Scores from the Hit-or-Miss are then summed within a tiling of small windows.

34. Candidate Orientation by Performing an Eigenfit • Assuming that the position is known, say by the candidate’s centroid, the problem of determining the orientation may be stated as fitting a subspace. For the problem at hand, this means finding the line having minimum orthogonal distance to any point of the candidate. • To turn this idea into a real algorithm, setup the Lagrangian and take the derivative with respect to the subspace vector: u is the eigenvector associated with the smallest eigenvalue of the scatter matrix

35. Is the Candidate really a Postal Code? • Having rotated the candidate to the horizontal, setup a feature vector as the discrete fourier transform of one channel of wavelet coefficients • Assuming Gaussian statistics, a simple maximum likelihood classifier, with both the mean vector and covariance matrix estimated from positive postal code examples works very well.

36. Not a Postal Code

37. A Postal Code

38. Use Fuzzy Logic to Classify Bars

39. The Run-Time Architecture Two stage pipeline (2 magazines processed at a time)

40. Conclusions • After developing the imaging algorithms on a Sparc station the Hit-or-Miss operator was implemented in a Gate-Array and the C code ported to the C40 DSPs by a contractor. I then continued the development by debugging the delivered code. • The magazine sorter ran at full speed with a very small error rate. • Most problems were not with the imaging, but with mechanical aspects of the machine.

41. The Inverse Problem of Range from Camera Blur

42. Two Solutions • Frequency Domain, blur from the estimated transfer function: • Spatial Domain, blur from the estimated local point spread function:

43. Perceived Depth of a Block of Steps

44. Summary • I’ve been involved in a number of challenging projects which have included modeling and simulation, satellite system engineering, the development of signal and image processing algorithms, nonlinear estimation, optics, and real-time systems. • Also, a copy of my paper on mean field annealing, that appeared in the Journal of Electronic Imaging (April, 1996), may be found at (in two parts): • http://www.legiotricesima.org/netpaper1-15.pdf • http://www.legiotricesima.org/netpaper16-31.pdf