Tony Rees and Glenelg Smith Divisional Data Centre + Remote Sensing Facility

1. Application of c-squares spatial indexing to an archive of remotely sensed data Tony Rees and Glenelg Smith Divisional Data Centre + Remote Sensing Facility CSIRO Marine Research, Australia (Tony.Rees@csiro.au)

2. The starting point … • Archive of c.60,000 satellite AVHRR scenes, 1986-current, accumulating at another 15+ per day • Region covered varies for each scene • Metadata (satellite, orbit, time parameters) and computed polygon (e.g. 7 points per side) available for each scene • Determining which scenes include any given point, on-the-fly (to answer user’s request) is a standard requirement, but not trivial – available methods may be slow, or inaccurate, or both...

3. example in more detail ... x • Example – may need to query which out of these four scenes includes point “x” (in reality, stop/start points of scenes also vary, also other scenes cross at different angle/s)

4. Options for spatial indexing Bounding polygons • Require a GIS system (or similar) for spatial queries (polygon overlays); can be slow to test many thousands of scenes • Can be problems with footprints which cross the date line, or include a pole Bounding rectangles • Computationally cheap, rapid to query, no GIS required • Typically a poor fit to actual data footprints (typically these are not square, or even nearly so) – many “false positives” • Can be problems with footprints which cross the date line, or include a pole Gridded (tile based) representation • Potentially simple to query, no GIS required • May be poor fit to actual data footprints if tile size is large (smaller tiles best, however index gets larger – tradeoff required here) • Compromise approach – slightly inexact compared with polygon overlays, but should be better performance for real-time queries (most computation has been done in advance) • Problems with footprints which cross the date line, or include a pole, can be pre-handled

5. C-squares basics • Based on a tiled (gridded) representation of the earth’s surface, at choice of 10 x 10, 5 x 5, 1 x 1, 0.5 x 0.5 degree tiles, etc. (0.5 x 0.5 degree tiles are used in this example) • Every tile has a unique, hierarchical alphanumeric ID (incorporates the ID of all parent, grandparent, etc. in every child tile ID) • Dataset (=scene) extents are represented by a list of all the tiles included in, or intersected by the footprint • Spatial search comprises looking for one or more tile IDs in the set associated with any dataset (= simple text search). (more details – seewww.marine.csiro.au/csquares/ )

6. C-squares nomenclature – ten degree squares (follows WMO 10 x 10º square nomenclature)

7. 10 degree square: 3213:*** 5 degree square: 3214:1** 1 degree square: 3315:100 Adaptive square size – 10 x 10, 5 x 5, 1 x 1, 0.5 x 0.5 degrees (etc.) – produces quadtree-like efficiencies (“compaction”) (this example shows detail to 1-degree square level) - can reduce storage requirement by up to 90%, cf. constant-resolution encoding

8. 10 x 10 degree squares (28) (base level of hierarchy, cannot compact) 5 x 5 degree squares (99) = 36 after “compaction” 1 x 1 degree squares (1,982) = 515 after comp. 0.5 x 0.5 degree squares (7,691) = 704 after comp. 0.5 x 0.5 degree squares - detail A real-world exampleNOAA-12 .. 21 Jun 2003 06:23

9. Scene <> c-square pairs (600-1200 rows per scene) Metadata table concept – 2 “search” tables only Scene details (1 row per scene)

10. Clickable map interface

11. Example search result

12. SQL for spatial search(example for 0.5 degree search square) select distinct A.scene_id, B.satellite, B.scene_date_time, B.image_location from satdata_csq_all A, scene_info B where ( (sqrsize = 0.5 and (A.csquare = search_csq -- e.g. 3414:100:1 (0.5 degree square) or A.csquare = substr(search_csq,1,8)|| ':*' -- 1 level of compression or A.csquare = substr(search_csq,1,6)|| '**:*' -- 2 levels of compression or A.csquare = substr(search_csq,1,4)||':***:*') -- 3 levels of compression ) -- (other supported search square size options inserted here) ) and (startdate is null or B.scene_date_time >= startdate) and (enddate is null or B.scene_date_time <= enddate) and (sat = 'any' or B.satellite = sat) and A.scene_id = B.scene_id order by B.scene_date_time, B.satellite;

13. Summary of steps involved • (1) Encoding – using custom polygon-to-c-squares algorithm (first attempt: 3+ mins per scene; second attempt: 30 seconds per scene to encode) • (2) Storage – first attempt: all scene/c-square pairs in one large table, second attempt: partitioned by 10 x 10º “bins” (100 smaller tables, for faster searching) • (3) Querying – using standard SQL and text matches • (4) (optional) Visualising – generating footprints, using web call to multi-purpose c-squares mapper More information: c-squares website - http://www.marine.csiro.au/csquares/ CMR Satellite Data spatial search: http://www.marine.csiro.au/remotesensing/csq-chooser.htm