1 / 25

Pan-STARRS PS1 Published Science Products Subsystem

Pan-STARRS PS1 Published Science Products Subsystem. Presentation to the PS1 Science Council August 1, 2007. What is PSPS?. Responsible for managing the catalogs of digital data PS1 PSPS will not receive image files, which are retained by IPP Three significant PS1 I/O threads:

feo
Download Presentation

Pan-STARRS PS1 Published Science Products Subsystem

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pan-STARRS PS1Published Science Products Subsystem Presentation to the PS1 Science Council August 1, 2007

  2. What is PSPS? • Responsible for managing the catalogs of digital data • PS1 PSPS will not receive image files, which are retained by IPP • Three significant PS1 I/O threads: • Ingest of detections and initial celestial object data from IPP • Ingest of moving object data from MOPS • User queries of detection/object data records

  3. DRL WBI IPP MOPS Other S/W Client Human Other DM What is PSPS? • Web Based Interface (WBI) – the “link” with the human • Data Retrieval Layer (DRL) – the “gate-keeper” of the data collections • PS1 data collection managers • Object Data Manager (ODM) • Solar System Data Manager (SSDM) • Provide the connection protocol for other (future/PS4) data collection managers; e.g., • “Postage stamp” cutouts • Complete Metadata database • Cumulative sky image server • Filtered transient database (and other special clients) ODM SSDM

  4. PSPS ComponentsOverview/Terminology • DRL: Data Retrieval Layer • Software clients, not humans, are PDCs • Connects to DMs • PDC: Published Data Client • WBI: Web Based Interface • External PDCs (non-PSPS) • DM: Data Manager (generic) • ODM: Object Data Manager • SSDM: Solar System Data Manager

  5. PSPS Development Status • DRL - a Request For Proposals has been issued to software developers to code the DRL designed by SAIC. This layer includes APIs to connect to the web clients and the databases. • ODM - a cooperative agreement is being developed with Johns Hopkins University’s Department of Physics & Astronomy to develop the ODM, leveraging their experience from the Sloan Digital Sky Survey database work. • SSDM - will be a working clone of the MOPS science client database (and a hot spare for the MOPS system).

  6. PSPS Development Status • WBI - Web clients to access the ODM and SSDM will include those already developed for the MOPS, the “Gator” clone developed at IfA, and a port of the SDSS “CasJobs” client. These will use the new DRL API being developed in the lead item above. • End-to-end testing of the PSPS structure can be accomplished using the DRL, the ported MOPS web client, and a MOPS clone on the backend. This can be done while the ODM is still under development.

  7. The Object Data Manager • The ODM is the major component of the PSPS, both in terms of size and complexity. It’s more than a simple archive. • The ODM will hold & provide user access to: • Catalogs of all individual focal plane (P2) detections. • Catalogs of detections from all stacked images. • Catalogs of all derived objects. • Catalogs of high-significance detections in difference images (when they become available). • “Blobs” of low-significance detections from difference images. • Sufficient metadata to allow the user to determine the provenance of any observation.

  8. ODM - Not Your Traditional Astronomical Database! • Unlike SDSS or 2MASS, we are not waiting until the project is over to generate the database, we’ll publish data as we go! • Data releases? The concept doesn’t apply here! We will probably keep monthly snapshots of the object catalog as the project proceeds. • Our logical data structure will allow the user to track how an object’s properties change as new (better) information is added over time. (It’s possible but not necessarily easy!)

  9. ODM Prototyping Goals The prototyping effort now underway at JHU is intended to demonstrate: • Data ingest (primarily detection to object correlation) • Scalability (physical data schema) - aka partitioning • Publishing (moving data from ingest pipeline to query side storage) in a way that has minimal impact on queries

  10. Prototype ODM Structure

  11. Existing Components (from SDSS) The prototype will utilize the following existing SDSS components: • Data Loading Pipeline (sqlLoader) • Self-extracting Documentation & Diagnostics • SQL Query Workbench (CasJobs) • Spatial Library (Spherical/HTM)

  12. Functionality Under Development New components for the prototype include: • Data Transformation Layer (input to loader) • Simulated Data (SDSS data & simulated galactic plane) • Sample Queries (verify query performance) • Cross-Match Functionality (detection-object correlation) • Data Partitioning Procedures (partition across muti-mode cluster for parallel data access)

  13. PS1 Logical Data Schema

  14. PS1 Data Tables & Sizes

  15. User Interfaces • The DRL authenticates “users” on a per machine basis. • Our initial implementation will be via a secure web server providing access to the following clients: • A port of CasJobs from SDSS to access the ODM • A “Gator” like menu driven tool to access the ODM • Perl tools (developed by MOPS) to access the SSDM • Machine access (PDCs) will be configured to attach to the DRL directly.

  16. System Expansion • Addition of other data collections, e.g., value added products are accommodated within our PSPS design. • The basic PSPS design provides well-defined APIs to the outside (WBI and PDCs) and inside (databases) • Data collections need not be Relational Database Management Systems (RDMS), but must obey the DRL-DM API • Databases need not all be the same type, e.g., ODM will use MSSQL and SSDM will be built on MySQL. • Although not part of the original PSPS design, we can provide intra-database communications (below the DRL) via well-defined mechanisms (e.g., ODBC, JDBC) to allow queries that cross the data collections hosted by the PSPS. These would be limited to read operations.

  17. Development Schedule • Award DRL development contract - August 2007 • ODM Prototyping through end of September 2007 • Critical Design Review - end of October 2007 • Hire PSPS Software Engineers (IfA & JHU) - October 2007 • Complete DRL development and perform integration & end-to-end tests using MOPS DB and web interface - April 2008 • Complete integration of the ODM from JHU into the PSPS & full subsystem testing of the system - August 2008

More Related