1 / 10

Ingesting and Serving WISE Data in IRSA

Ingesting and Serving WISE Data in IRSA. Scott Terek Ramón Rey GRITS – June 11, 2010. Ingesting and Serving WISE Data in IRSA. What are we doing, and why are we doing it? Challenges we faced Our solution Current usage Future plans. What are we doing?.

kara
Download Presentation

Ingesting and Serving WISE Data in IRSA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ingesting and Serving WISE Data in IRSA Scott Terek Ramón Rey GRITS – June 11, 2010

  2. Ingesting and Serving WISE Data in IRSA • What are we doing, and why are we doing it? • Challenges we faced • Our solution • Current usage • Future plans Ingesting and Serving WISE data in IRSA

  3. What are we doing? • IRSA will serve WISE data during the mission • IRSA has the tools • Gator • isisql • IRSA has the experience • IRSA already serves a lot of other large catalogs • Personnel have experience with large databases in general Ingesting and Serving WISE data in IRSA

  4. Why is this different? • Larger than our usual catalogs • Many small individual loads instead of being delivered all at once • Need access to all previous data as new data are being ingested Ingesting and Serving WISE data in IRSA

  5. Our solution • Fragmentation • Fragment data into 50 equal buckets on disk • Fragmented by spatial index (spt_ind) • Fragment spt_ind index to match • Multiple tables • New table for each source delivery • Create “view” uniting all tables into one virtual table Ingesting and Serving WISE data in IRSA

  6. wise_pass1_i1bs_psd_view wise_pass1_i1bs_psd_1 wise_pass1_i1bs_psd_2 wise_pass1_i1bs_psd_3 … … … Disk chunks (1-50) Disk chunks (1-50) Disk chunks (1-50) wise_pass1_i1bs_psd_n wise_pass1_i1bs_psd_4 . . . … … Disk chunks (1-50) Disk chunks (1-50) Ingesting and Serving WISE data in IRSA

  7. Proof of concept / testing • Palomar Transit Factory (PTF) • Very similar issues • Large dataset • Multiple deliveries • Performance testing • Single table vs. multi-table view • Multi-table view as number of tables increases Ingesting and Serving WISE data in IRSA

  8. Tradeoffs • Inability to do aggregates • COUNT(), MAX(), MIN(), etc. • Considered writing functions to perform those tasks • Decided it would be too much work for too little use • Increased maintenance costs • Extra software to perform view management, fragment data, etc. Ingesting and Serving WISE data in IRSA

  9. Current operations • Began receiving data14 January • 19 provisional data deliveries ingested • 17 level 1b source deliveries • 4 level 3 source deliveries • 40 pass1 data deliveries ingested (as of 8 June) • 20 level 1b source deliveries • 20level 3 source deliveries Ingesting and Serving WISE data in IRSA

  10. Future plans • Realign data at end of mission • Revert to single table • Regain ability to do aggregates • Re-fragment according to final spatial index distribution • Keep parallelization • More evenly distribute data Ingesting and Serving WISE data in IRSA

More Related