Prototype Web Services Using SDSS DR1

1. Prototype Web ServicesUsing SDSS DR1 Alex Szalay, Tamas Budavari, Sam Carlisle, Jim Gray, Vivek Haridas, Nolan Li, Tanu Malik, Maria Nieto-Santisteban, Wil O’Mullane, Ani Thakar

2. NVO: How Will It Work? • Define commonly used ‘core’ services • Build higher level toolboxes/portals on top • We do not build ‘everything for everybody’ • Use the 90-10 rule: • Define the standards and interfaces • Build the framework • Build the 10% of services that are used by 90% • Let the users build the rest from the components

3. Using SDSS DR1 • SDSS DR1 (Data Release1) is now publicly available http://skyserver.pha.jhu.edu/dr1/ • About 1TB of catalog data • Using MS SQL Server 2000 • Complex schema (72 Tables) • About 80 million photometric objects • Two versions (TARGET/BEST) • Automated documentation • Raw data at FNAL file server with URL access

4. Loading DR1 • Automated table driven workflow system for loading • Included lots of verification code • Over 16K lines of SQL code • Loading process was extremely painful • Lack of systems engineering for the pipelines • Poor testing (lots of foreign key mismatch) • Detected data bugs even a month ago • Most of the time spent on scrubbing data • Fixing corrupted files (RAID5 disk errors) • Once data was clean, everything loaded in 3 days • Neighbors calculation took about 10 hours • Reorganization of data took about 1 week of experiments in partitioning/layouts

5. Reorganization • Introduced partitions and filegroups • Photo, Tag, Neighbors, Spectro, Frame, Other, Profiles • Keep partitions under 100GB • Vertical partitioning – tried and abandoned • Both partitioning and index build now table driven • Stored procedures to create/drop indices at various granularities • Tremendous improvement in performance when doing this on a large memory machine (24GB) • Also much better performance afterwards

6. Spatial Features • Precomputed Neighbors • All objects within 30” • Boundaries, Masks and Outlines • Stored as spatial polygons Time Domain: • Precomputed Match • All objects with 1”, observed at different times • Found duplicates due to telescope tracking errors • Manual fix, recorded in the database • MatchHead • The first observation of the linked list used as unique id to chain of observations of the same object

7. Spatial Algorithms • Updated HTM library • Automated depth for HTM_Cover • Output vertices • Simplify polygon • Boolean operations on regions • Part of VO data model (A. Rots) • Zones • Much better performance for bulk neighbors at a fixed radius • Footprint service in progress • Bool Contains(point) • Region Intersect(region)

8. Web Services in Progress • Registry • Harvesting and querying • Data Delivery • Query driven Queue management • Graphics and visualization • Query driven vs interactive • Show spatial objects (Chart/Navi/List) • Footprint/intersect • It is a “fractal” • Cross-matching • SkyQuery and SkyNode • Ferris-wheel • Distributed vs parallel

9. Registry: Easy Clients Just use SOAP toolkit (T. McGlynn & J. Lee have done Perl client). Easy in Java java org.apache.axis.wsdl.WSDL2Java "http://skyservice.pha.jhu.edu/devel/registry/registry.asmx?wsdl" • Gives set of Classes for accessing the service • Gives Classes for the XML which is returned (i.e. SimpleResource) Still need to write client like RegistryLocator loc = new RegistryLocator(); RegistrySoap reg = loc.getRegistrySoap(); ArrayOfSimpleResource reses = null; reses = reg.queryRegistry(args[0]); http://skyservice.pha.jhu.edu/devel/registry/index.aspx

10. Generic Catalog Access • After 2 years of SDSS EDR and 6 months of DR1 usage, access patterns start to emerge • Lots of small users, requiring instant response • 1/f distribution of request sizes (tail of the lognormal) • How to make everybody happy? • No clear business model… • We need a separate interactive and batch server • We also need access to full SQL with extensions • Users want to access services via browsers • Other services will need SOAP access

11. Data Formats • Different data formats requested: • HTML, CSV, FITS binary, VOTABLE, XML, graphics • Quick browsing and exploration • Small requests, need to be nicely rendered • Needs good random access performance • Also simple 2D scatter plots or density plots required • Heavy duty statistical use • Aggregate functions on complex joins, lots of scans but small output, mostly want CSV • Successive Data Filter • Multi-step non-indexed filtering of the whole database,mostly want FITS binary

12. Data Delivery • Small requests (<100MB) • Putting data on the stream • Medium requests (<1GB) • Use DIME attachments to SOAP messages • Large requests (>1GB) • Save data in scratch area and use asynch delivery • Only practical for large/long queries • Iterative requests • Save data in temp tables in user space • Let user manipulate via web browser • Paradox: if we use web browser to submit, users want immediate response from batch-size queries

13. How To Provide a UserDB • Goal: through several search/filter operations reduce data transfer to manageable sizes (1-100MB) • Today: people download tens of millions of rows, and then do their next filtering on client side, using F77 • Could be much better done in the database • But: users need to create/manage temporary tables • DOS attacks, fragmentation, who pays for it • Security, who can see my data (group access)? • Follow progress of long jobs • Who does the cleanup?

14. Query Managament Service • Enable fast, anonymous access to small requests • Enable large queries, with ability to manage • Enable creation of temporary tables in user space • Create multiple ways to get query output • Needs to support multiple mirrors/load balancing • Do all this without logging in to Windows • Need also support of machine clients • Web Service: http://skyservice.pha.jhu.edu/devel/CasJobs/ • Two request categories: • Quick • Batch

15. Queue Management • Need to register batch ‘power users’ • Query output goes to ‘MyDB’ • Can be joined with source database • Results are materialized from MyDB upon request • Users can do: • Insert, Drop, Create, Select Into, Functions, Procedures • Publish their tables to a group area • Data delivery via the CASService (C# WS) • http://skyservice.pha.jhu.edu/devel/CasService/CasService.asmx

16. Graphics Tools • Simple xy plotshttp://skyservice.pha.jhu.edu/nli/wplot/ • Density plothttp://skyservice.pha.jhu.edu/devel/DensityMap/AllSkyView.aspxhttp://skyservice.pha.jhu.edu/devel/DensityMap/PlotQuery.aspx • Chart/Navi/Listhttp://skyservice.pha.jhu.edu/dr1/imgcutout/getjpeg.asmx • Can be built into various applications

17. Archive Footprint • Footprint is a ‘fractal’ • Result depends on context • all sky, degree scale, pixel scale • Translate to web services • Footprint()returns single region that contains the archive • Intersection(region, tolerance)feed a region and returns the intersection with archive footprint • Contains(point)returns yes/no (maybe fuzzy) if point is inside archive footprint

18. Cross-Matching • SkyQuery – SkyNode • Currently lots of proprietary features • Data transmitted via .NET DataSet => VOTable • Query plan written in MS T-SQL => ADQL • Spatial operator restricted to a cone =>VORegion • Made up metadata delivery => VORegistry • Data delivery in XML/HTML => VOTable • Catalogs in the near future • SDSS DR1, FIRST, 2MASS, INT • POSS-1, GSC-2, HST, ROSAT, 2dF • GALEX, IRAS, PSCZ

19. Spatial Cross-Match • For small area HTM is close to optimal, but needs more speed • For all-sky surveys the zone algorithm is best • Current heuristic is a linear chain of all nodes • Easy to generalize to include precomputed neighbors • But, for all sky queries very large numberof random reads instead of sequential

20. SDSS Portal Ferris-Wheel • Sky split into buckets/zones • All archives scan in sync • Queries enter at bottom • Results come back afterfull circle • Only sequential access=> buckets get into cache,then queries processed

21. Utilitites • FITSLIB 1.10C# library around the CFITSIO packagehttp://www.cs.jhu.edu/~haridas/tech/Fits/ • MIRAGEJava wrapper around Mirage, can directly access the VORegistry, and ConeSearchhttp://skyservice.pha.jhu.edu/develop/vo/mirage/mirage.html • HTM2.0Updated HTM library, conforming to the new Region specificationhttp://www.sdss.jhu.edu/htm/ • ADQLPrototype service to convert back and forth between ADQL and SQLhttp://skyservice.pha.jhu.edu/vivek/msdev/AstroDql/ws/http://skyservice.pha.jhu.edu/vivek/msdev/AstroDql/ws/Archive.asmx • SDSSQAJava application, emulating MS Query Analyzer

22. Summary • Web Services have been remarkably easy to use • Now different platforms are interoperable • We have invested a lot of energy to develop various interface libraries (FITS, VOTable) • Integrating graphics into web services was very easy • Next: • Parallel queries • Finish query queue management • Upgrade SkyQuery • Bring in more archives • Ferris-Wheel experiment • On-demand database creation • 100TB parallel data access layer

23. http://skyservice.pha.jhu.edu/develop/