Ship Imaging and Classification

1. Ship Imaging and Classification – An Inverse SAR Application EECS 826: InSAR and Applications Tyler Herrmann April 16, 2009

2. Presentation Outline Inverse SAR (ISAR) Basics ISAR Introduction Typical Errors in ISAR ISAR Applications Ship Recognition using ISAR Introduction Ship Imaging Single Frame Processing and Feature Extraction Multi-Frame Processing Real World Examples AN/APS-137 – Presentation Outline

3. Inverse SAR Basics It is a technique to generate two-dimensional high resolution images of a target. ISAR requires that the target be moving while the radar itself is stationary. Typically a stationary radar does not have an adequate antenna beam width to generate high resolution images. ISAR uses the rotation / movement of the target to generate these high resolution images. Range resolution is obtained either using a short pulse or pulse compression. Cross-range resolution is obtained using the Doppler history of different parts of the target. ISAR generates a sequence of range-Doppler images. – ISAR Introduction

4. Inverse SAR Basics ISAR and SAR are related in that they both require a change in the aspect of a target. – ISAR Introduction: Comparison Between ISAR and SAR [1]

5. Inverse SAR Basics An example of ISAR imaging was done in a lab with the following setup. This imaging was done to model the properties of different scattering targets. – ISAR Introduction: ISAR Example [5]

6. Inverse SAR Basics Several objects were placed on a rotating fixture to produce ISAR images. A metallic model airplane A metallic plate A pair of scissors – ISAR Introduction: ISAR Example (Cont.) [2] [2] [5]

7. Inverse SAR Basics The ISAR Image process can result in errors which cause deterioration of the resulting images. Unknown target or antenna motion Unmodeled motion will cause defocusing of the image. Vertical nearfield errors Tall targets will defocus and move to incorrect locations. Integrated side lobe return Range and azimuth compression side lobes will cause the ISAR image quality to degrade. Frequency and azimuth sampling errors Aliased images and spurious targets can result from the selection of incorrect frequencies and aspect deltas. – Typical ISAR Errors

8. Inverse SAR Basics Antenna aberration Errors in the image happen when the antenna phase center is dependent upon the antenna aspect or RF frequency. Target Dispersion Targets that shift in position with RF frequency. Multipath Ghosting images can appear in ISAR images when there are multiple return reflections. – Typical ISAR Errors (Cont.)

9. Inverse SAR Basics ISAR is typically used in maritime surveillance. Ship Imaging and Classification / Recognition Coast Guard Search and Rescue ISAR can also be used in other applications Aircraft Imaging and Classification / Recognition Used to determine the RCS of different scattering targets. – ISAR Applications

10. Ship Recognition using ISAR ISAR is primarily used to image / classify ships at sea. By placing the ship in the antenna beam (search-lighting, spotlight mode), the radar can produce images with enough resolution for ship recognition. Typically a human operator analyzes the returned sequence of images to classify the target. It is desirable to provide the operator with a smooth sequence of ISAR images as the target’s motion changes . Automatic ship recognition from ISAR images is also possible. Limiting factor of quick recognition has been the performance of humans. – Introduction

11. Ship Recognition using ISAR Three different techniques are used for classification of the targets along with the intuitive recognition of the radar operator. Mensuration “…location of major target components (mast, superstructure breaks, cranes, weapons, etc.) along the bow-stern axis of the target are measure as a percentage of the range extent of the ship and compared with the locations of the same components on candidate ship classes.” [8] Feature Description “…is the process of labeling the components of candidate targets by their degree of match to descriptive templates (such as the stern: straight, curved, or rounded; or a mast: pole, lattice, or solid).” [8] Shape Correlation “… visually comparing the image to a wireframe model of one or more candidate ships the operator thinks might match the unknown image.” [8] – Introduction (Cont.)

12. Ship Recognition using ISAR The ship’s movement (pitch, roll, and yaw) with the ocean waves generates enough movement for ISAR images to be produced. – Introduction (Cont.)

13. Ship Recognition using ISAR Pitch A pitch motion causes the masts to have a higher velocity than the rest of the ship, generating a difference in Doppler shifts. This movement produces a vertical profile of the target in the length direction. Roll A roll motion causes a similar result as pitch but in the plane that includes the width of the target. This information does not play a major role because the width is small compared to the length of a ship. Yaw When the motion of the ship is a yaw motion, it produces a plan view of the target. Information of the ship height structure from pitch and roll is called “Profiling.” – Introduction (Cont.)

14. Ship Recognition using ISAR Ways that ISAR imaging can affect ship recognition To produce adequate images, it maybe necessary to wait as long as the ship’s motion period to get a favorable image aspect. The cross-range and projection-plane scales are always changing due to the ship’s motion. Different frames of the target separated by only a fraction of the motion period can reveal different target features. 3D images can be extracted directly from this. Since the images are changing radically with time, coherent integration can not be done. Multiframe processing might be required because not all frames have useful information contained in them. – Introduction (Cont.)

15. Ship Recognition using ISAR There are a few requirements when using ISAR to image ships. The range resolution of the radar must be much smaller than the dimensions of the target. The antenna beam must searchlight the target for usually longer than the ship’s roll period. The cells of the image must be range stabilized relative to the target. Meaning that one target point remains in the same range cell for at least the duration needed to obtain all the radar data of a single image frame. – Ship Imaging

16. Ship Recognition using ISAR Typical ISAR image sequence of a small commercial ship – Ship Imaging (Cont.) • The radar used to produce these images was an X-band radar with 100 MHz bandwidth and 2.5° antenna beamwidth. • Each frame is separated by approximately 0.6 seconds. • Between frame (a) and frame (g), the time elapsed was 3.7 seconds. [8]

17. Ship Recognition using ISAR Description of the frames in the ISAR image sequence – Ship Imaging (Cont.) • An inverted image that is predominately a vertical profile. • Also an inverted image but with less of a profile and more of plan view. • The vertical profile is minimized because of the very small roll and pitch rates. [8]

18. Ship Recognition using ISAR Description of the frames in the ISAR image sequence – Ship Imaging (Cont.) • The image has started to change vertical direction. • A defined vertical profile can be seen in the upward direction. • Again like (a) there is a sharply focused profile • Vertical profile in the upward direction with a significant plan view. [8]

19. Ship Recognition using ISAR Each image frame in a time sequence must be processed. Some frames will be of better image quality and include more target information. Processing of each frame consists of the following: Segmenting the targets Extracting features Frame selection criterion. Multiframe processing can also utilize these same techniques. – Single Frame Processing and Feature Extraction

20. Ship Recognition using ISAR Segmentation is the process of taking an image and partitioning it into different sets of pixels with certain visual characteristics. The end result of segmentation is to simplify the image into something more manageable and easier to analyze. Generally thought that a recognition scheme relying on a vigorous segmentation process are less successful. ISAR image characteristics make the segmentation process more reliable. – Single Frame Processing and Feature Extraction: Segmentation

21. Ship Recognition using ISAR When performing the segmentation algorithm it is important to keep the bright superstructure features well defined while maintaining the end-points of the target. This is difficult to achieve because of the large dynamic range of the different features of the target. Artifacts from Doppler sidelobes and focus errors can be eliminated by increasing the threshold but this removes information about the target’s endpoints. – Single Frame Processing and Feature Extraction: Segmentation (Cont.)

22. Ship Recognition using ISAR Description of the segmentation process – Single Frame Processing and Feature Extraction: Segmentation (Cont.) • A threshold is selected that will be applied later to remove the noise and clutter. • The vertical streaks in the original image are removed. A low-pass filter is also applied to remove noise spikes. [8]

23. Ship Recognition using ISAR Description of the segmentation process – Single Frame Processing and Feature Extraction: Segmentation (Cont.) • The threshold produced from step one is applied to the image in frame (c). • A morphological region-growing and region filling processes is applied to the remaining image. [8]

24. Ship Recognition using ISAR Description of the segmentation process. – Single Frame Processing and Feature Extraction: Segmentation (Cont.) • A geometric clustering process is used to eliminate the remaining non-target objects from the image. • The resulting clustered imaged is then overlaid on the original image to restore the target intensity information. [8]

25. Ship Recognition using ISAR Region-growing and region-filling process It is a simple process used in image segmentation which examines adjacent pixels of the initial “seed points.” Seeds points are locations in the image that are used to select a threshold and to determine where to grow the image from. If a pixel is determined to meet the threshold criteria then the pixel is connected to the seed point. This process continues until there are no more changes left in the image. – Single Frame Processing and Feature Extraction: Region Growing and Filling

26. Ship Recognition using ISAR Region-growing and region-filling process example – Single Frame Processing and Feature Extraction: Region Growing and Filling (Cont.) • Original image of a grey-scaled lightning strike. • Purpose is to leave only the lightning strike and remove the “noise.” • This image has values from 0 to 255 describing the different grey-scale levels. [9]

27. Ship Recognition using ISAR Region-growing and region-filling process example – Single Frame Processing and Feature Extraction: Region Growing and Filling (Cont.) • Seed points were selected and shown in the figure on the left. • The level of the seed points were selected to be 255. [9]

28. Ship Recognition using ISAR Region-growing and region-filling process example – Single Frame Processing and Feature Extraction: Region Growing and Filling (Cont.) • A range of grey-scale values were selected for the threshold. • Threshold values of 225 through 255 were used to produce this image. [9]

29. Ship Recognition using ISAR Region-growing and region-filling process example – Single Frame Processing and Feature Extraction: Region Growing and Filling (Cont.) [9] [9] Grey scale ranges of 190-255 Grey scale ranges of 155-255

30. Ship Recognition using ISAR Region-growing has it advantages and disadvantages Can correctly separate different regions based of the seed values. Can result in good segmentation results if the original image has clear edges. Easy concept that eliminates noise rather simply. It is time consuming to complete the computation process. Holes or over-segmentation can occur if there is a variation in intensity. Shading of real images may not be recognized. – Single Frame Processing and Feature Extraction: Region Growing and Filling (Cont.)

31. Ship Recognition using ISAR Geometric clustering process This process is used to remove any bright regions that are not located in image positions that are characteristics of the shape of the target. Geometric clustering performs region-labeling. These regions are measured using statistical labels such as size, boundaries, and center of mass. The process will either eliminate or preserve the region depending on its size and perpendicular distance to the estimated centerline of the target. – Single Frame Processing and Feature Extraction: Geometric Clustering

32. Ship Recognition using ISAR Feature extraction is used so operators can accomplish the process of shape-correlation. Different features included bow, stern, and the height of the bridge on the ship. These features are used to determine the projection plane of the image. This projection plane determines the physical projection of a 3D target model into a 2D image plane. – Single Frame Processing and Feature Extraction: Feature Extraction

33. Ship Recognition using ISAR Transforming the 3D image into a 2D plane allows for automated software to identify different features of the target. This results in an automatic adjustment of the wireframe model to align with the target image. – Single Frame Processing and Feature Extraction: Feature Extraction (Cont.) [8]

34. Ship Recognition using ISAR Centerline End-points Plan component Profile component – Single Frame Processing and Feature Extraction: Feature Extraction (Cont.) Features extracted used to determine the orientation • Width of plan component • Width of outline at either end • Superstructure breaks/extents • Major uprights [8]

35. Ship Recognition using ISAR A large amount of time is spent in determining the center line of the target. The information acquired by finding the centerline can also be used in the extraction of the other features. With automatic recognition, only an estimate of the center line is found. Using prior knowledge of the symmetry and the expected target shape. The centerline estimation is found by searching for the maximum peak of the Hough transform. – Single Frame Processing and Feature Extraction: Feature Extraction (Cont.)

36. Ship Recognition using ISAR The plan component is an important feature used in automatic ISAR processing. The plan component of the target is also estimated by using the Hough transform. Straight lines are fit to each deck edge then a search for the peaks at an angle parallel to the angle of the centerline. It is also estimated by using the Doppler separation of the two deck edges. – Single Frame Processing and Feature Extraction: Feature Extraction (Cont.)

37. Ship Recognition using ISAR By comparing the intensities near the centerline the target’s endpoints can be estimated. The width estimate of the target is also estimated by performing the Hough transform. Performed at angles 90 degrees from the centerline. A histogram process can be used to determine if the image is inverted or upright within the frame. – Single Frame Processing and Feature Extraction: Feature Extraction (Cont.)

38. Ship Recognition using ISAR If there is enough profile information, then other vertical features can be found. Using peak-decomposition on the profile histogram, mast locations and superstructure groups can be located. – Single Frame Processing and Feature Extraction: Feature Extraction (Cont.) [8]

39. Ship Recognition using ISAR The width measurements, superstructure and upright locations are used to determine which end is the bow. If the features make it unmistakable which end is the bow, then a single decision is made for that image frame. When the results are uncertain then both scenarios must be pursued. – Single Frame Processing and Feature Extraction: Feature Extraction (Cont.)

40. Ship Recognition using ISAR In shape correlation, a high quality ISAR image is compared to a wireframe model from a database of ships. For each wireframe model, the aspect and motion parameters are calculated from the features extracted mentioned in the previous slides. The wireframe is transformed to have the same projection as the image. – Single Frame Processing and Feature Extraction: Shape Correlation

41. Ship Recognition using ISAR Wireframe models were used because they are easily obtained and are generally easier to perform computations on. Even though wireframe models have their limitations, they have an advantage over a data-based approach of classification. Data-based classification requires a set of data samples for each target to be classified. – Single Frame Processing and Feature Extraction: Shape Correlation (Cont.)

42. Ship Recognition using ISAR The yaw rate is estimated using the Doppler differences between the bow and stern and the length of the ship. To estimate an effective pitch and roll rate, the Doppler is used on the on the deck edges of the height components. A real pitch and roll rate can not be calculated in a single frame image. – Single Frame Processing and Feature Extraction: Shape Correlation (Cont.)

43. Ship Recognition using ISAR With the wireframe model adjusted to fit the projection of the image the following is done: The wireframe model is solidified and the target image is overlaid onto the model. Then the reverse is applied; the model is overlaid onto the target image. The number of pixels that show through both situations are counted. The lower the number of pixels, the higher percentage of a correct match. – Single Frame Processing and Feature Extraction: Shape Correlation (Cont.)

44. Ship Recognition using ISAR – Single Frame Processing and Feature Extraction: Shape Correlation (Cont.) Example of the fitting a model to an ISAR image • Frame (a) is the obtained ISAR image. • Frame (b) is the best matching overlay to the ISAR image. • Frame (c) is the second best match to the original image. [8]

45. Ship Recognition using ISAR Due to the inherent nature of ISAR images, classifications based off of single frames can be unreliable. Multiframe processing has a couple steps that it performs to increase the probability of a correct match. Frame Selection Multiframe Feature Extraction – Multiframe Processing

46. Ship Recognition using ISAR Frame selection for multiframe processing is based on profile information. A specialized criteria is applied to each frame to determine if the profile component is present. The plan component is also estimated so that it can be determined if there is enough true profile height. Little or no plan component of the image is preferred. When determining the height, it is important to obtain an absolute estimate of the profile instead of just a relative height. – Multiframe Processing: Frame Selection

47. Ship Recognition using ISAR “The plan-side pixel count outside the plan boundary opposite the profile side should be less than a threshold.” [8] “Plan width must be less than a threshold.” [8] “The median height of the profile histogram must exceed a threshold.” [8] "The profile histogram standard deviation should exceed another threshold.” [8] – Multiframe Processing: Frame Selection (Cont.) A typical criteria was developed after an evaluation of the effectiveness of feature detection was performed.

48. Ship Recognition using ISAR Problems can occur when performing the frame selection process. The amount of plan and profile in the image vary independently from each other. This is because pitch and roll are not necessarily related to yaw. An image can have the best profile view but also incur significant plan view. It is very important to adjust the profile height for the plan view. – Multiframe Processing: Frame Selection (Cont.)

49. Ship Recognition using ISAR Multiframe feature extraction uses many image frames to make a determination of which features to use. Feature extraction does not occur until approximately 30 profile view frames have been processed. Each frame is processed individually to find the features then compared to one another. The consistency of each feature’s location from frame to frame is used to determine if that feature is used in further processing. – Multiframe Processing: Feature Extraction

50. Ship Recognition using ISAR Example of Multiframe Feature Extraction – Multiframe Processing: Feature Extraction (Cont.) • The top left image in the following figures is the original ISAR image with the features marked. • The middle left plot is a single frame profile that shows the height of the target in Doppler cells vs. range cells. • The remaining four plots are histograms representing the different features of the target. • The lower left image shows the width of the target at the bow and stern.