1 / 7

Recap of Data Access Methods See Chapter 1 of Data Analysis Guide and youtube videos!

Recap of Data Access Methods See Chapter 1 of Data Analysis Guide and youtube videos!. David R. Ardila User Support coordinator / NHSC Archive Scientist. Data Management. D ata management in Herschel i s complicated by the size of datasets

maxine
Download Presentation

Recap of Data Access Methods See Chapter 1 of Data Analysis Guide and youtube videos!

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Recap of Data Access MethodsSee Chapter 1 of Data Analysis Guide andyoutube videos! David R. Ardila User Support coordinator / NHSC Archive Scientist

  2. Data Management • Data management in Herschel is complicated by the size of datasets • Everything is about making data management efficient • Do you really need big data sets in your disk?

  3. For most people, this is all you need. • Gets references to the data in HIPE and downloads data as needed • Useful for • Quick inspection • Data reduction • Does not save, unless you do it explicitly* • *of course, there is a way to override this! HSA getObservation(…,useHsa=True) Product Browser “Send to External Application” HIPE

  4. HSA getObservation(…,useHsa=True) Product Browser “Send to External Application” “Retrieve Products” HIPE Tar file in local disk getObservation() Use Navigator untar Directory in local disk • HSA to local disk • May be slow • But then you have the data • It allows you to bypass HIPE

  5. Local Pools (working directories in the local disk) saveObservation() saveProduct() Save Products GUI getObservation(…,useHsa=True) Product Browser HSA getObservation() Product Browser “Send to External Application” “Retrieve Products” HIPE HIPE Tar file in local disk getObservation() Use Navigator untar • Pools are good if: • You have a lot of data • That needs reprocessing Directory in local disk

  6. Local Pools (working directories in the local disk) saveObservation() saveProduct() Save Products GUI HSA getObservation(…,useHsa=True) Product Browser getObservation() Product Browser “Send to External Application” “Retrieve Products” HIPE HIPE Tar file in local disk getObservation(…, useHsa=True, save=True) getObservation() Use Navigator untar getObservation() Product Browser myHSA A type of pool. Unmodified data (or links) from HSA Directory in local disk

  7. Local Pools (working directories in the local disk) saveObservation() saveProduct() Save Products GUI HSA getObservation(…,useHsa=True) Product Browser “Send to External Application” getObservation() Product Browser “Retrieve Products” HIPE HIPE Tar file in local disk getObservation(…, useHsa=True, save=True) getObservation() Use Navigator untar getObservation() Product Browser myHSA A type of pool. Unmodified data (or links) from HSA Directory in local disk

More Related