A Look into the Apache OODT Ecosystem

1. A Look into the Apache OODT Ecosystem Chris A. MattmannNASA JPL/Univ. Southern California/ASF mattmann@apache.org November 9, 2011

2. And you are? • Senior Computer Scientist at NASA JPL in Pasadena, CA USA • Software Architecture/Engineering Prof at Univ. of Southern California • Apache Member involved in • OODT (VP, PMC), Tika (VP,PMC), Nutch (PMC), Incubator (PMC), SIS (Mentor), Lucy (Mentor) and Gora (Champion), MRUnit (Mentor), Airavata (Mentor)

3. Welcome to the Apache in Space! (OODT) Track

4. Agenda • Overview of OODT and its history • How we got it to Apache • How other projects can follow our model • Existing successful deployments of OODT • Pointers to papers, and more information including case studies

5. Lessons from 90’s era missions • Increasing data volumes (exponential growth) • Increasing complexity of instruments and algorithms • Increasing availability of proxy/sim/ancillary data • Increasing rate of technology refresh • … all of this while NASA Earth Mission funding was decreasing A data system framework based on a standard architecture and reusable software components for supporting all future missions.

6. Enter OODT Object Oriented Data Technology http://oodt.apache.org Funded initially in 1998 by NASA’s Office of Space Science Envisaged as a national software framework for sharing data across heterogeneous, distributed data repositories OODT is both an architecture and a reference implementation providing Data Production Data Discovery Data Distribution Data Access OODT is Open Source and available from the Apache Software Foundation

7. Apache OODT • Originally funded by NASA to focus on • distributed science data system environments • science data generation • data capture, end-to-end • Distributed access to science data repositories by the community • A set of building blocks/services to exploit common system patterns for reuse • Supports deployment based on a rich information model • Selected as a top level Apache Software Foundation project in January 2011 • Runner up for NASA Software of the Year • Used for a number of science data system activities in planetary, earth, biomedicine, astrophysics http://oodt.apache.org 7

8. Apache OODT Press

9. Why Apache and OODT? • OODT is meant to be a set of tools to help build data systems • It’s not meant to be “turn key” • It attempts to exploit the boundary between bringing in capability vs. being overly rigid in science • Each discipline/project extends • Apache is the elite open source community for software developers • Less than 100 projects have been promoted to top level (Apache Web Server, Tomcat, Solr, Hadoop) • Differs from other open source communities; it provides a governance and management structure

10. Governance Model+NASA=♥ • NASA and other government agencies have tons of process • They like that

11. Publicly accessible and searchable archives • http://svnsearch.org/svnsearch/repos/ASF/search?path=%2Foodt • http://mail-archives.apache.org/mod_mbox/oodt-dev/ • http://mail-archives.apache.org/mod_mbox/oodt-user/ • 100+ ML list subscriptions

12. Great Metrics and Insight • http://www.ohloh.net/p/oodt

13. Movement to the ASF • Meeting held June 15, 2007 at JPL with ASF President Justin Erenkrantz • Develop plan moving forward to bring first NASA project to Apache • Discuss obstacles, sponsorship • Discuss outlook

14. 2007: original goals • Come up with incubation proposal • Chris Mattmann was one of the principal contributors to the proposal for the Tika project, and to other Incubation activities (Apache SIS) • Send out emails to the Incubator mailing list • Look for mentors • Get sponsorship from ranking Apache PMC member or board member • Justin and others • Top-level project versus sub project outlook heading out of incubation

15. OODT Incubator Planning • Monthly Updates (for first 3 months, then quarterly) • Status • Progress • Community • Acceptance • Plan for exiting incubation • How to have a solid user base • How to operate as a unit in the Apache way • Maintenance of user interest and community going forward

16. OODT’s next steps circa 2007 • JPL to tackle legal issues • Is OODT releasable as an Apache product • http://www.apache.org/licenses/software-grant.txt • This needs to be signed by parties that be by JPL • Contributor License Agreement • Do we need a corporate one? • In parallel to this • Draft OODT incubation proposal • Start identifying who would initially be interested • More external, non-JPL people who are interested, the better • Justin to get slides from other incubator people

17. …2 years later • Worked it out with JPL legal • Turns out the ALv2 license is extremely friendly and is something that JPL (note not all of NASA) was amenable to • Developed OODT incubator proposal • http://wiki.apache.org/incubator/OODTProposal • Found willing Apache mentors besides Justin • Jean-Frederic Clere, Ross Gardler, Ian Holsman • …Put OODT at Apache!

18. Apache OODT Community • Includes PMC members from • NASA JPL, Univ. of Southern California, Google, Children’s Hospital Los Angeles (CHLA), Vdio, South African SKA Project • Projects that are deploying it operationally at • Decadal-survey recommended NASA Earth science missions, NIH, and NCI, CHLA, USC, South African SKA project • Use in the classroom • My graduate-level software architecture and seach engines courses

19. OODT Framework Andrew Hart and Emily Law will talk about these later OODT/Science Web Tools Archive Client Navigation Service OBJECT ORIENTED DATA TECHNOLOGY FRAMEWORK You’ll hear about this later today Catalog & Archive Service Bridge to External Services Other Service 1 Profile Service Product Service Query Service Other Service 2 Profile XML Data Data System 1 Data System 2 I’ll tell you about these now

20. Architectural Principles • Division of Labor • Don’t make one component the workhorse! • Technology Independence • Don’t get bitten in the rear when a software vendor decides to charge you a lot of $$$ for their previously low cost technology • Metadata as a first-class citizen • Descriptions of resources come in handy • Separation of software and data models • Allow each to evolve independently

21. OODT Architecture • Reference Architecture • Four pairs of component types • Product Client/Server, Profile Client/Server, Query Client/Server, Catalog and Archive Client/Server • Two connector types • Messaging layer discussed in http://sunset.usc.edu/~mattmann/pubs/ICSE.pdf • Handler connector (discussed in this presentation) • Instantiated for different domains using these fundamental building blocks

22. Web site Product Server (B) Product Server (C) MSSQL Product Client (B) Product Client (C) Product Client and Server • Deliver data from underlying data store • Accept uniform query structure that identifies 0 or more “products” (data items) to retrieve • Many-to-Many Product Server (A) RAIDDisk Product Client (A)

23. How about an example of a product?

24. Web site Profile Server (B) Profile Server (C) MSSQL Profile Client (B) Profile Client (C) Profile Client and Server • Deliver metadata from underlying metadata store • Metadata gives user enough information about where to find actual data • Housekeeping information • Resource information • Domain-specific information • Many-to-Many Profile Server (A) Oracle Profile Client (A)

25. How about an example of a profile? Attributes Relationships

26. Discovered Initial set Query Server (A) Query Client (A) Product Server (A) Product Server (B) Profile Server (B) Profile Server (A) Query Client (B) Query Client (C) Query Server (B) Query Server (C) Query Client and Server • Query Server seeded with initial set of pointers to Profile Servers • Profile Servers point to actual resources (Product Servers, even other Profile Servers) • Interactive (metadata returned)and non-interactive (data returned) • Many-to-Many

27. Archive Client (A) Archive Server (A) Profile Server (A) Product Server (A) Archive Server (B) Archive Client (B) Archive Client (C) Repository Registry Catalog and Archive Client and Server (CAS) • Ingest data into repository and metadata into registry • Run processing algorithms on data/metadata upon ingestion • Workflow support • Serve back Repository data with Product Server • Serve back Registry metadata with Profile Server • Many-to-Many

28. Some notes about CAS Credit: D. Woollard • All Core components implemented as web services • XML-RPC used to communicate between components • Servers implemented in Java • Clients implemented in Java, scripts, Python, PHP and web-apps • Service configuration implemented in ASCII and XML files

29. Web site Product/Profile Server DBMS Product Handler Flat File Product Handler Web Site Product Handler MSSQL MSSQL Product/Profile Server Handler Connectors • Encapsulate (meta-)data coordination and communication • Allow for dynamic addition and removal of different classes of back end metadata and data stores RAIDDisk

30. Example handler connectors • XMLPS • http://oodt.apache.org/components/maven/xmlps/ • XML config file specifies recipe for extracting records from an RDBMS and turning them into a NoSQL repository • PS • XML configurable profile server to unlock OPeNDAP datasets and pass them to OODT

31. So, how do you piece them together: NASA VODC • NASA’s Virtual Oceanographic Data Center (VODC) • http://vodc.jpl.nasa.gov • Information integration using OODT components • Profile, Product, Query, also uses Apache Solr, and Plone

32. So, how do you piece them together: JPL’s CDX • CDX = Climate Data Exchange • Provide comparison of remote sensing data and model outputs • Existing systems remain in place; services expose data and functions over the network; support the era of IPCC 5th assessment and distributed, petabytes of data

33. Who’s doing what? • Children’s Hospital Los Angeles • Improving upon XMLPS, and CAS (Andrew Hart + Ricky Nguyen will talk about this) • Supporting data analytics • Google • Brian Foster working on command line improvements and data protocol push/pull • SKA South Africa • Deploying file manager and crawler for use in KAT-7 pipeline ingestion • NIH/NCI • Maintaining the XMLPS components, and CAS components • Helping with user interfaces • Various JPL and NASA research projects • OPeNDAPps, XMLPS • Various NASA missions • Workflow, PCS, services, OPSui, other web apps

34. Latest release: 0.3 • First appearance of PCS • Core, Services (JAX-RS) • Web Applications • Balance (PHP), and Wicket (Java)-based apps for file management and workflow monitoring • First release deployed to Maven Central • We did backport 0.2 there after this • Over 60 issues fixed in JIRA • June 2011: recommended stable release

35. Working on: 0.4 • Operator Interface (OODT-157) • Andrew Hart and I will talk about this • Workflow2 integration (OODT-215) and all of its sub-issues • Global workflow conditions, dynamic workflows, parallel/sequential model, new workflow engine, etc. • OODT RADIX for super easy deployment (OODT-120) • Paul Ramirez and Cameron Goodale will discuss this • Solr sync with File Manager (OODT-326) • Improvements to XMLPS (OODT-333) and new crawler actions (OODT-33, OODT-34, OODT-35, OODT-36, OODT-37) • Over 48 issues currently resolved • Likely to come before end of Q4 2011

36. Using Apache OODT as a testbed for software process • Missions maintaintheir own local CMs • Local mission CMs contain forks of existing OSS software • Forks can be patchbased or CM based • Changes found particularly effectiveare discussed within the comm.And eventually brought before a CCB that reviews their generality, etc. Credit: D. Freeborn

37. Some Grand Challenges I’m interested in: OODT can help! • How do we handle 700 TB/sec of data coming off the wire when we actually have to keep it around? • Required by the Square Kilometre Array • Joe scientist says I’ve got an IDL or Matlab algorithm that I will not change and I need to run it on 10 years of data from the Colorado River Basin and store and disseminate the output products • Required by the Western Snow Hydrology project

38. Some Grand Challenges I’m interested in: OODT can help! • How do we compare petabytes of climate model output data in a variety of formats (HDF, NetCDF, Grib, etc.) with petabytes of remote sensing data to improve climate models for the next IPCC assessment? • Required by the 5th IPCC assessment and the Earth System Grid and NASA • How do we catalog all of NASA’s current planetary science data?

39. Key Takeaway OODT is already doing and/or preparing the world to handle all of these diverse use cases! It’s a constantly evolving and improving framework – join up and help. It’s free and open source from Apache and helping government demonstrate the public good

40. OODT Project Contact Info • Learn more and track our progress at: • http://oodt.apache.org • WIKI: https://cwiki.apache.org/OODT/ • JIRA: https://issues.apache.org/jira/browse/OODT • Join the mailing list: • dev@oodt.apache.org • Chat on IRC: • #oodt on irc.freenode.net • Acknowledgements • Key Members of the OODT teams: Chris Mattmann, Daniel J. Crichton, Steve Hughes, Andrew Hart, Sean Kelly, Sean Hardman, Paul Ramirez, David Woollard, Brian Foster, Dana Freeborn, Emily Law, Mike Cayanan, Luca Cinquini, Heather Kincaid • Projects, Sponsors, Collaborators: Planetary Data System, Early Detection Research Network, Climate Data Exchange, Virtual Pediatric Intensive Care Unit, NASA SMAP Mission, NASA OCO-2 Mission, NASA NPP Sounder Peate, NASA ACOS Mission, Earth System Grid Federation

41. Alright, I’ll shut up now • Any questions? • THANK YOU! • chris.a.mattmann@nasa.gov • @chrismattmann on Twitter