EdSkyQuery-G Overview

1. EdSkyQuery-G Overview Brian Hills, December 2004 www.edikt.org

2. Contents • Edikt • Motivation & Aims • Architecture • Current Status • Results • Future Outlook

3. Edikt • “e-Science Data, Information and Knowledge Transformation” • Bridge the gap between applications and computer science: • Produce robust tools… • …for real application science problems… • …test them under extreme science conditions… • …and keep an eye on the commercial possibilities. • Projects which may be of interest to astronomers: • BinX, Eldas & EdSkyquery-G. • Visit: www.edikt.org

4. Astronomy Requirements • Sky Surveys collect masses of data to be managed: • For example, Sloan Digital Sky Survey: 15TB. • 2 * 10GB databases to be used for the project. • Edikt will have access to a 155TB SAN. • Further research by leveraging data from different surveys: • Must identify same object from different catalogues. • Require a “federated” view: • Data is distributed, homogeneous, large scale. • Building one big data warehouse isn’t feasible. • Interoperable services to combine disparate data sources. Is the middleware up to the task?

5. EdSkyQuery-G: Motivation & Aims • Support the “Open SkyQuery Initiative” • Move from a .NET-specific implementation. • Enable similar functionality on other platforms. • Extensible framework for e-science: • Handle heterogeneous archives. • ‘Plug in’ algorithms e.g. Nearest Neighbour. • Interact with Astrogrid components & VO. • Leveraged for the BRIDGES project to perform simple joins. • Apply Eldas to large scale E-Science problems: • Test: functionality, scalability & performance. • Cross team collaboration: • Dr. Bob Mann (UoE, ROE), Edikt, EPCC, NeSC.

6. Client Portal SkyNode 1 SkyNode 2 SkyNode 3 SkyQuery: High Level Architecture 1. User submits query. 9. Returns results for display. 2. Client sends query to Portal. 3. Portal invokes SkyNode with an execution plan. 4. Recursive call to next SkyNode in the execution plan. 8. Performs cross-match and returns final set of results. 5. Recursive call to last SkyNode in the execution plan. 7. Performs cross-match and returns results. 6. Runs query and returns results.

7. EdSkyQuery-G: Architecture • Inspired by Greg Riccardi’s paper for DAIS-WG: • http://www.cs.fsu.edu/~riccardi/grid/skyquery.pdf • Discusses two approaches: • Access recipes for service interactions. • Retained state for service interactions. • Potential benefits of #2: • Scalability • Robustness • Usability

8. Query Builder Service Manager SkyNode 1 DB2 - XP SkyNode 2 DB2 - Linux EdSkyQuery-G: Architecture 1. User submits query. 9. Client may pull results back from SkyNode. 2. Query Builder sends query to Service Manager (SM). 8. SM returns a handle to the results. 5. SM invokes SkyNode 1 with an execution plan + results handle. 3. SM invokes SkyNode 1 with an execution plan. 7. Handle to results. 4. Handle to results. 6. Transfer results file.

9. getMetadata doQuery JBOSS Grid Service Layer Generic Parser Distributed Query Processor Optimize Split Execute EdSkyQuery-G: Service Manager Client Uses Service Client Uses Service Service Manager Call to Query service Call to Xmatch service Call to Info service

10. DB2 EdSkyQuery-G: SkyNode Architecture Invoked via the Service Manager Query XMatch Info EldasSkyNode Grid Services SkyNode Query & Stored Proc invocations

11. Results: .csv .xml Results: .csv .xml Results: .csv .xml EDR SSA EdSkyQuery-G: Stored Procedures Export Transport Import Export Export Transport Import Export Transport

12. Current Status • Pre-Alpha (internal release), November ‘04. • End-to-end invocation of all components: • Client->ServiceManager->SkyNode->Database • 2 * 10GB DB2 test databases: • SSA from SuperCosmos, hosted by NeSC. • EDR from SDSS, hosted by EPCC. • Limitations: • SM: No query parser/splitting. • Simple cross database join: not yet using XMatch. • No data transport, other than JDBC between databases. • Final results reside on database server.

13. Results: .csv .xml Results: .csv .xml EDR SSA EdSkyQuery-G: Pre-Alpha Stored Procedures Import Export Export JDBC Query Results Import Export

14. Results • Tested with 3 queries from ROE cookbook: • http://surveys.roe.ac.uk/ssa/sqlcookbook.html • Queries #16, #17, #19. • Results show: • Exporting data is quick. • Importing data is >10 * slower: • Should we use native DB calls rather than JDBC for import only? • Queries slow: • Need more indexes and database tuning?

15. Deliverables • Software (Internal): • Prototype client & GUI client. • Service Manager. • Eldas + Skynode Interface. • Database stored procedures (Java). • Documentation (some on NescForge): • Use Cases, Requirements, Design. • Performance Testing, Installation. • Papers • AHM 04 Poster & Paper: • http://www.allhands.org.uk/proceedings/papers/123.pdf • ADASS 04 Paper.

16. Short Term Focus • Enable science: • Compare different cross match algorithms. • Incorporate XMatch, CSIRO, ROE as stored procedures. • Data Transfer – SCP: • Between databases. • Pull results back to client. • Deliver to a third party. • Client & Service Manager enhancements: • Handle broader range of queries. • Performance: • Compare with pre-alpha benchmarks. • Improve test infrastructure.

17. Query Builder MySpace Client Service Manager Parser Registry SkyNode 1 DB2 - XP SkyNode 2 DB2 - Linux Longer Term Focus: AstroGrid

18. Longer Term Focus • Interaction with Astrogrid components: • Clients, Registry, ADQL Parser, MySpace. • OpenSkyQuery: • Compliance with interfaces. • Test with other DBMS and catalogues. • Query Builder GUI/Data Integration tool: • Lead user through choosing fields from different datasets. • GridFTP: • Currently unsuitable…. • NeSC course in Jan 05 may reveal more.

19. Thank you Questions?