1 / 37

Data-Intensive Science at Johns Hopkins University

Data-Intensive Science at Johns Hopkins University. Jordan Raddick. Institute for Data-Intensive Engineering and Science (IDIES) Johns Hopkins University. Case Study: Astronomy. In the “old days” astronomers took photos. New instruments are digital (< 100 GB/night)

callia
Download Presentation

Data-Intensive Science at Johns Hopkins University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data-Intensive Science atJohns Hopkins University Jordan Raddick Institute for Data-Intensive Engineering and Science (IDIES) Johns Hopkins University

  2. Case Study: Astronomy • In the “old days” astronomers took photos. • New instruments are digital (< 100 GB/night) • Detectors are following Moore’s law. • Data avalanche: double every 2 years Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

  3. Why astronomy? • Because “it’s worthless” (Jim Gray) • No privacy restrictions • No intellectual property • No one wants to sell you data • Because it’s intrinsically interesting • And because there’s a lot of it

  4. Astronomy Data • Astronomers have a few Petabytes now • Data doubles every 2 years • Data is public after 2 years • So, 50% of the data is public • But… how do I get at that 50% of the data?

  5. Science is breaking • You can… • search 1 MB in 1 second, transfer for < 1¢ • search 1 GB in 1 minute, transfer for $1 • search 1 TB in 2 days, transfer for $1,000 • search 1 PB in 3 years, transfer for $1,000,000 • …and 1 PB is 5,000 disks • “Data avalanche” in all fields of science • New approach needed

  6. Data Intensive Science Tools • Databases instead of files (DBMSs) • Data integrity ensured • Optimized data access (DB indices) • Ability to define methods on data • Do the science inside the database! • Bring the analysis to the data, not vice-versa • Scalable (parallel) data access • High-speed transport protocols • Move the data rapidly when necessary • Asynchronous Web services • Web browsers cannot handle data volumes

  7. Gray’s Laws of Data Engineering • Jim Gray • Scientific computing is revolving around data • Need scale-out solution for analysis • Take the analysis to the data! • Start with “20 queries” use cases

  8. Virtual Observatory • Premise: Most data is (or could be) online • So, the Internet is the world’s best telescope: • It has data on every part of the sky • In every measured spectral band: optical, x-ray, radio.. • As deep as the best instruments (2 years ago). • It is up when you are up.The observing conditions are always great (no working at night, no clouds no moons no..). • It’s a smart telescope: links objects and data to literature on them.

  9. Sloan Digital Sky Survey • A map of the universe • Telescope at Apache Point Observatory (Sunspot, NM) • 2.5 meter reflector • ~3 degree field of view • Drift scanning (telescope stationary, sky appears to move)

  10. An Ambitious Survey • Info content > US Library of Congress • Before SDSS, total number of galaxies with measured parameters ~ 200k • With SDSS, we already have detailed parameters for over 200 Million galaxies!! A thousand-fold increase in the amount of data!

  11. SDSS Data Overview Catalog Archive Server (CAS) Science parameters extracted to catalogs Stuffed into relational DBMS (SQL Server) Heavily indexed, optimized Online access via SkyServer Several levels of access, query tools Data Archive Server (DAS) FITS files (raw data) Images, spectra, corrected frames, atlas images, binned images, masks Online form-based access Rsync and wget file retrieval SDSS Data Release cas.sdss.org skyserver.sdss.org www.sdss.org das.sdss.org/DRx-cgi-bin/DAS

  12. The CAS Databases SDSS Pipelines • Processed data is stuffed into a commercial relational DBMS • Microsoft SQL Server 2000 • Allows fast exploration and analysis - Data Mining • Two versions of the sky: Best and Target • Target is version of sky on which spectroscopic targets were chosen • Best is latest, greatest processing of the data • 2 DBs for each release, e.g. BestDR3 and TargDR3 • Heavily indexed to speed up access – HTM + DB Indices • Short queries can run interactively. • Long queries (> 1 hr) require a custom Batch Query System.

  13. SkyServer Web Access • Supports several levels of SQL access • Novice/casual users • Radial (cone) and Rectangular search • Intermediate/astro users • Imaging and Spectro Form Query • Expert • Free-form SQL, Object Crossid (upload RA/dec list) • CasJobs workbench environment (MyDB) • Visual tools • ImageCutout service • Finding Chart, Navigate/browse images, image lists • Explore tool • Detailed info for each image object, including spectrum • Downloadable interfaces • Emacs, Java tool (sdssQA), sqlcl (command-line) http://cas.sdss.org/, http://skyserver.sdss.org/

  14. Query execution time follows power law Vast majority of queries under a minute Separate short and long queries Execute long queries in batch mode Short and long queues (short=1min, long=8hrs) Strictly limit time of query in a queue Provide user workspace DB – MyDB Reduce network traffic from repeat and intermediate results Allow sharing of query results between user groups CAS Workload Management

  15. CasJobs and MyDB • Batch Query Workbench for SDSS CAS • Queries are queued for batch execution • Load balancing – queues on multiple servers • Limit of 2 simultaneous queries per server • Short (1 minute) queue for immediate mode • Query aborted after 1 minute • MyDB personal database • 1 GB (more on demand) SQL DB for each user • Long queries write to MyDB table by default • User can extract output (download) when ready • Ability to share MyDB tables with other users via group visibility

  16. SkyServer • Entire database of the Sloan Digital Sky Survey • Free to anyone • 4 TB of data • 287 million sky objects • 1.2 million spectra

  17. Why data to the public? • All SDSS data available to general public • Commitment to data sharing • People can look at any star or galaxy • Inquiry learning known to be effective (learners answer a question themselves, with their own design)

  18. Browse Data

  19. Explore Data

  20. Search for Data

  21. From Access to Learning • Not enough just to give public access to data • How are they using it? • Are they learning anything from it? • Do they understand what they’re doing and why?* *Understanding = long-term memory, transfer to new situations (How People Learn)

  22. Formal Education: Projects

  23. SkyServer projects • Three levels • Kids (K-8) • Basic (high school or Astro 101) • Advanced (skilled and motivated students) • Research challenges • Independent research (open inquiry) with data • Activities created by users

  24. SkyServer projects

  25. Use of educational projects • 2 of top 20 IPs to hit entire site are K-12 districts • Orlando, FL & Los Angeles

  26. Citizen science: Galaxy Zoo • Background: galaxies come in different shapes • Classifying is hard for computers, easy for people • So get the public to help! spiral elliptical

  27. Galaxy Zoo

  28. Galaxy Zoo Discovery • But if you mirror galaxies, it’s still 48/52! • Counter-clockwise excess due to perception • Unintentional psychology discovery! clockwise (48%) counter-clockwise (52%)

  29. Galaxy Zoo Discovery • “Hanny’s Voorwerp” (“Voorwerp” = object) • Strange blue thing found by Dutch teacher Hanny van Arkel • Most likely “light echo” from a long-dead quasar • Follow-up time on HST • Galaxy Zoo great for finding the unexpected!

  30. Galaxy Zoo and learning • Even people who love science often don’t understand peer review, proposals, etc. • Want to use Galaxy Zoo to show process of science • As we write science papers, we share results with volunteers

  31. Galaxy Zoo blog

  32. Other Science: Sensors • Small, self-contained devices to measure soil temperature, etc. • Deployed around Maryland • Study carbon cycle, nesting of turtles, etc. • Wide range of applications

  33. Other science: Turbulence • Large-scale simulation of isotropic turbulence in fluid • Imagine shaking a box of water • FORTRAN code • 1024 x 1024 x 1024 mesh • 10,240 timesteps • Every 10th timestep written to SQL database

  34. Other science: Turbulence • Database accessible online through web services • Can call database from FORTRANor C code • Like having a supercomputer simulation on your laptop!

  35. Graywulf • Data-intensive computing architecture • 40 worker nodes, 1440 MB/s each • 6 head nodes, 2100 MB/s each • DDR InfiniBand interconnect • Deployed mid-2008 • Architecture for future projects

  36. Graywulf at SC08

  37. Contact information Contact Information Jordan Raddick 410-516-8889 raddick@jhu.edu

More Related