Federated Geospatial Community Cloud For NGA Analytics

1. Federated Geospatial Community Cloud For NGA Analytics Dr. Brand Niemann Director and Senior Data Scientist Semantic Community http://semanticommunity.info/ AOL Government Blogger http://breakinggov.com/author/brand-niemann/ September 12, 2013 http://semanticommunity.info/Network_Centricity/NGA_Demo

2. Purpose • The Network Centric Operations Industry Consortium (NCOIC) is leading a project for the National Geospatial-Intelligence Agency to develop and test a Geospatial Community Cloud (GCC) concept, It is designed to show interoperability and movement of data in an open, cloud-based infrastructure. The demonstration, September 12th, for the NGA will depict a global, multidisciplinary response similar to the one that took place after the 2010 Haitian earthquake.

3. Background • As a subject matter expert guest of the NCOIC, I helped form the NCOIC Cloud Computing Work Group out of a Federal Cloud Computing Community of Practice and developed this Geospatial Community Cloud Concept out of pilot I started for NGA Director Letitia Long several years ago and continued for the NCOIC and others. Jon Polay, then with GeoEye, and now with Voyager was very instrumental in helping bring advanced geospatial technology to that pilot and has now helped with bringing Voyager Search, which is already being used by NGA for enterprise search, to this pilot. I also helped bring the pattern that has become the NCOIC Emergency Services Operational Pattern to the NCOIC based on earlier work for the Federal CIO Council and the US Environmental Protection Agency.

4. Voyager • Voyager provides an easy-­‐use, out-­‐of-­‐the-­‐box, enterprise search solution that can support a wide variety of users around the globe. It is open and scalable, but with advanced security. Voyager can index a wide variety of content. But most importantly it is the only spatially enabled, enterprise search solution. • Voyager is like Google for GIS and is used by NGA for Enterprise Search - Data Discovery and the Air Force for GeoBase.

5. Spotfire • Spotfire 5.5 includes 30+ new in-database sources, adds powerful new visualization techniques and provides essential data governance for: • Expansive, Flexible, Managed Data Access • Quick, Powerful, Contextual Visualizations • Real-Time & Big Data Predictive Analytics • Today’s enterprise users need to make insightful decisions residing in disparate data sources, such as cubes or relational databases. Spotfire 5.5 compresses time to actionable insight by enabling enterprise users to make discoveries in traditional sources along with those uncovered from real-time sources and unstructured formats.

6. Phase 1 Pilot • The main problem in Phase 1 of the project was finding the data rapidly and organizing it in a catalog that could be used by analysts to both respond to current disasters and provision analytics for future disasters. I did this by Google searches and "word of mouth" in Phases 1 to identify some best practice data sources for the Haitian Earthquake of 2010. Now that is automated by Voyager Search in a very elegant way so I can now demonstrate in rapid sequence the following steps used in the Open Government Data work I have been doing recently: • Step 1 Get the Data • Step 2 Scrub the Data • Step 3 Build Something • Step 4 Distribute

7. Phase 2 Pilot • So the demo will capture these steps using Voyager Search and Spotfire in the Cloud on selected 2010 Haitian Earthquake data sets to distribute analytic and map results to desktop and mobile devices. This demo implements the Global Architecture Concepts of NGA's Todd Myers and the NCOIC Emergency Services Operational Pattern work of Rex Brooks.

8. Knowledge Base My Note: This is like the Analysts Notebook in the Secure Cloud for Documentation and Sharing. http://semanticommunity.info/Network_Centricity/NGA_Demo

9. Voyager Search A new approach to SDI: It's time to reimagine the spatial data infrastructure (SDI). While web mapping has exploded since 1993, the usage has languished, the data is undiscoverable and too much valuable content is inaccessible. Voyager writes about some fresh approaches in GeoConnexion International Magazine. http://voyagersearch.com/

10. Embrace Chaos: Lack of metadata foils many spatial data infrastructure initiatives • VoyagerGIS, a small software company, was born as a partnership between a development and program manager from ESRI and a lead developer in the open-source search community that built Lucene/Solr. Together they taught Lucene to speak spatial like a native. • Voyager, works by walking through the myriad array of GIS files on your C: drive, your server, your favorite web services and that secret data stash you keep on an external drive. It opens each one and extracts every bit of information it can about it, some of it from metadata, much of it not. • Creating an index of that data, Voyager can answer questions like how many times is that data source used and in what maps documents? Where are the broken links? Are you hosting duplicate data? • It allows users to search by keyword for, say, ‘roads’ and to use a map to only look in Italy for those roads. Most importantly, if the data changes, Voyager knows that it has, even if the metadata hasn’t been updated. Source: GeoConnexion International Magazine, June 2013.

11. Voyager Search: Haiti My Note: Showing All 12 of 12 Web Address

12. Voyager Search: Haitian EarthquakeSummary View My Note: Showing Only 25 of 48 Web Address

13. Voyager Search: Haitian EarthquakeGrid View My Note: Showing 48 of 48 Web Address

14. Voyager Search: Haitian EarthquakeTable View My Note: Showing 48 of 48 Web Address

15. Voyager Search: Haitian EarthquakeReport View My Note: See Next Slide Web Address

16. Voyager Search: Haitian EarthquakeExport List My Note: I added the links to the 48 data sets. http://semanticommunity.info/@api/deki/files/25982/VoyagerHaitiEarthquake.xlsx

17. Voyager Search: Haitian EarthquakeLink to Data http://demo.voyagergis.com/services/voyager/#/id=30a9d767d8b03066/disp=5c7e6218/

18. Voyager Search: Haitian EarthquakeExport Map http://hdds.usgs.gov/arcgis/rest/services/201001_Earthquake_Haiti/MapServer/export

19. Spotfire Desktop: NCOICNGAOGC 1

20. Spotfire Web Player: NCOICNGAOGC https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?NCOICNGAOGC-Spotfire

21. Spotfire iPad: NCOICNGAOGC http://itunes.apple.com/us/app/spotfire/id417436823?mt=8

22. Conclusions and Recommendations • Semantic Community Has Produced 2010 Haitian Earthquake Analytics in a Federated Geospatial Community Cloud for the NGA. • Three Advanced Secure Community Cloud Tools Were Used: • MindTouch Used by Many Organizations for Help Desk and Documentation with Content Analytics; • Voyager (like Google for GIS) Used by NGA For Enterprise Search - Data Discovery; and • Spotfire Used by the Intelligence Community For Enterprise Applications to Make Insightful Decisions Residing in Disparate Data Sources. • Recorded Future Used by the Intelligence Community as a Temporal Analytics Engine for Mining the Internet (not shown here). • Be Informed Used by the Dutch Government and the European Union for Dynamic Case Management (not shown here). • These Tools Used by Subject Matter Experts Support the NGA Mission of Putting GEOINT in the Hands of Users and Freeing the Time of Analysts to Do More Advanced Analytics. • These Results Also Support the New Open Government Data Policy and Implementation Guidance – Project Open Data.

23. Federation of Cloud Technologies Now, Federation of Data Next • After today's demonstration my comment was as follows: I liked what you did. As someone who spent years working on disaster response at the US EPA, my experience was that we analyzed the heck out of past disasters to find out what really happened and what we should have done about them, and because we did not have cloud technology at that time, we worked on federation of the databases, as a basis for eventually having a center of excellent for disaster response. • What I saw today was mostly what I would call federation of technologies in a cloud. There are lots of government and non-government organizations that have centers of excellence for disaster response with important databases (e.g. United Nations, NATO, Red Cross, Department of Homeland Security, etc.) that need to be federated in the cloud. So that would seem to be the next step here. • I will do another example of that using Voyager Search, who I asked to provide more information and they responded as follows: I have added information from the sources that you provided to us in the email from last week, and we have also added lots of additional information from our own sources.  Please have a look at what we have compiled and indexed on our public demo site, which can be found at http://voyagerdemo.com/haiti • I extracted 1755 geospatial metadata items into a spreadsheets and added the links to the 25 Shape files and imported them into Spotfire. See next slides

24. Voyager 1.9 Beta: Haiti Table My Note: Export Table to CSV, SHP, or XML. See Next Slide for CSV. http://voyagerdemo.com/haiti/voyager/#/view=table/

25. Voyager 1.9 Beta: Haiti CSV Export http://semanticommunity.info/@api/deki/files/25982/VoyagerHaitiEarthquake.xlsx

26. Voyager 1.9 Beta: Haiti Shape Files http://voyagerdemo.com/haiti/voyager/#/f.format=application%252Fvnd.esri.shapefile/

27. Spotfire Desktop: NCOICNGAOGC 5

28. Spotfire Desktop: NCOICNGAOGC 6

29. Voyager Demos http://voyagersearch.com/ex/demos

30. Voyager Demos: York Region http://findit.yorkmaps.ca/opendata/explore/#/disp=df68233d/

31. Voyager Export Results List My Note: I added the first four columns to give each data sets a well-define URL. http://semanticommunity.info/@api/deki/files/26383/voyager86567802384065691351.xlsx

32. Colour Shaded Relief, 2009 • Purpose - Colour shaded relief is used to represent topography on maps in a natural, aesthetic, and intuitive manner. • Overview - This colour shaded relief is used to represent the relief and orientation of physiographic features within York Region on maps in a natural, aesthetic, and intuitive manner. The desired effect is achieved by varying the shading and transparency of the colour shaded relief in the map. This colour shaded relief dataset was created using ArcGIS Spatial Analyst extension, Hills shade tool with the 2009 Ground Elevation (DEM) as the input surface. http://findit.yorkmaps.ca/opendata/explore/#/id=8afdba31863b70cc/disp=df68233d/

33. Spotfire Desktop:NGA Demo Federation of Data

34. Spotfire Web Player:NGA Demo Federation of Data https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?YorkRegionOpenData-Spotfire