HSeg Review & Update

1. National Aeronautics and Space Administration HSeg Review & Update James C. Tilton Code 606.3 Computational & Information Sciences and Technology Office NASA Goddard Space Flight Center July 27, 2016 www.nasa.gov

2. HSeg Background HSegproduces a hierarchical set of image segmentations with the following characteristics: A set of segmentations that • consist of segmentations at different levels of detail, in which • the coarser segmentations can be produced from merges of regions from the finer segmentations, and • the region boundaries are maintained at the full image spatial resolution The HSeg algorithm is fully described in: James C. Tilton, YuliyaTarabalka, Paul M. Montesano and Emanuel Gofman, “Best Merge Region Growing Segmentation with Integrated Non-Adjacent Region Object Aggregation,” IEEE Transactions on Geoscience and Remote Sensing, Vol. 50, No. 11, Nov. 2012, pp. 4454-4467. GFSAD30 2

3. Incorporating Edge Information into HSWO/HSeg/RHSeg Edge information is incorporated at three different stages: • An initialization stage in which the edge information directs a fast first-merge region growing process (Muerle-Allen)to quickly merge connected areas with edge values <= Et (set by user), and • The normal HSWO/HSeg best merge region growing stage in which the edge information influences the best merge decisions. • In performing processing window artifact elimination in RHSeg. J. L. Muerle, D. C. Allen, “Experimental Evaluation of Techniques for Automatic Segmentation of Objects in a Complex Scene,” in G. C. Cheng, et al. (Eds.), Pictorial Pattern Recognition, Thompson, Washington, DC, pp. 3-13, 1968. GFSAD30 3

4. Frei-Chen Edge Difference Operator Result: A true color rendition of a 768x768 pixel section of Ikonos data from the Patterson Park/Inner Harbor area of Baltimore, MD. Frei-Chen Edge Difference Operator Result, maximum over spectral bands, thresholded at 0.07. GFSAD30 4

5. Utilizing HSeg in the GFSAD30 Project Some possibilities: • Use RHSeg/HSeg together with HSegLearn to perform computer assisted photointerpretation of high resolution imagery data (< 5m) to develop ground reference data. • Develop pre-processing techniques to “enhance” the imagery data prior to processing with HSeg. • Develop post-processing analysis approaches for automated classification. • Porting HSeg or hsegextract to GEE may facilitate the use of HSeg in the project. GFSAD30 5

6. RHSeg/HSeg together with HSegLearn • HSegLearn takes as input a hierarchical set of image segmentations such as produced by the HSeg best-merge region growing segmentation program. • Through HSegLearn, an analyst specifies a selected set of positive and negative examples of impervious land cover. • HSegLearn searches the hierarchical set of segmentations for the coarsest level of segmentation at which the selected positive examples do not conflict with negative example locations and labels the image accordingly. GFSAD30 6

7. RHSeg/HSegfollowed by hsegextract $ hsegextract -h -oparam -hseg_level -class_labels_map_ext -class_npix_map_ext -class_mean_map_ext -class_std_dev_map_ext -class_bpratio_map_ext -object_labels_map_ext -object_npix_map_ext -object_mean_map_ext -object_std_dev_map_ext -object_bpratio_map_ext GFSAD30 7

8. Post-processing of RHSeg/HSeg results: Mutlu found that the RHSeg/HSeg image segmentation results contained too many small region objects. He requested that I develop post-processing approaches to eliminate small region objects while maintaining the overall RHSeg/HSeg image segmentation quality. In response I developed two post-processing programs last year: • hsegnpixprune: selects a segmentation out of a segmentation hierarchy that has region objects no larger than a specified number of pixels, • Modified hswo: performs best-merge region growing by the HSWO method in which merges between pairs of regions that both are larger than a “min_map_unit” size are not allowed. Must be initialized with the segmentation output from hsegnpixprune or from a segmentation selected from an RHSeg/HSeg segmentation hierarchy. GFSAD30 8

9. Progress with RHSeg/HSeg Since January 2016: • Created a new version of RHSeg (labeled version 1.64) that completely eliminates processing window artifacts. This new version utilizes slightly overlapping processing windows. • Submitted a NASA New Technology Report (GSC-17660-1), “Elimination of Processing Window Artifacts by Utilizing Processing Window Overlap” on April 4, 2016. The submission includes an updated RHSeg/HSeg User’s Manual. • Started working with Mutlu and Aparna to produce image segmentation maps for their cropland mapping approach. GFSAD30 9

10. Version 1.64 of RHSeg/HSeg: The analysis flow of the RHSegalgorithm: Fig. 2. Lris determined as the number times the input image must be subdivided to achieve a small enough image size for efficient processing with HSeg. The analysis flow of the recursive function rhseg(L,X): Fig. 3. Nminis equal to ¼ the number of pixels in the subimage processed at the deepest level of recursion. GFSAD30 10

11. Version 1.64 of RHSeg/HSeg: Two key changes from previous versions: • Processing window subsections overlap by adding overlap_width (= 2 by default) pixels around perimeter (rows and columns outside the physical extent are masked out). • Region classes are selected for merging across adjoining subsections by • Creating a histogram of correspondence between the region labelings in the overlapping seams, and • Finding region pairs for which the histogram count is more than a specified value (MIN_HISTO_COUNT, default value 9). GFSAD30 11

12. Version 1.64 of RHSeg/HSeg: Side effect of processing window overlapping: The image pixels in the overlapping areas are counted twice in accumulating the feature values for each region. This side effect could be eliminated by a more complicated programming structure – but this is not necessary. One must realize that RHSegalready is an approximation of HSeg over the whole image, and this double counting should be considered to be another aspect of this approximation. Nonetheless, this double counting is eliminated at the final stage of RHSeg prior to when HSeg is run of the image as a whole. At this point, the processing window overlapping is undone by discarding the data values in the overlapped areas in each image subsections and reinitializing the region features based on the non-overlapping data subsections. GFSAD30 12

13. Progress with RHSeg/HSeg Since January 2016: • The NASA New Technology Report (GSC-17660-1) was approved: “Elimination of Processing Window Artifacts by Utilizing Processing Window Overlap.” (RHSeg/HSeg version 1.64) • Version 1.64 of RHSeg/HSeg installed on Goddard NCCS, Ames NEX, NAU cluster and UW cluster. • Processed 17 multi-date Landsat TM images for Mutlu and Aparna. • Revised the “plurality_vote” utility to better suit Mutlu’s analysis requirements. • Started working with Richard Massey/Teki and Jun Xiong in applying RHSeg to support their analysis approaches. Process some multi-date images for Richard (some were cloudy) and processed 20 2-date Landsat TM images for Jun. Also produced NDVI image segmentation outputs for Jun. GFSAD30 13

14. Open Question: Integration of RHSeg/HSeg with GEE The porting of RHSeg/HSeg to GEE does not seem feasible. However, perhaps it may be feasible to port hsegextract to GEE. We would have to do the following to do so: • We need to be able to read in a set of parameters from ASCII files (filenames, numbers and Boolean flags). • We need to be able to read image files (with filenames provided in the ASCII file in 1 above). • We need to be able to read binary data files based on parameters from 1 (skipping to a particular location in the file and reading records of specified sizes and types). • We need to be able perform some data manipulation based on the information read from the parameter files, image files and binary data files. • We need to be able to write image files. GFSAD30 14

15. SignOffPage GFSAD30 15