1 / 46

Introduction to HIPE and the HSA Applet

Introduction to HIPE and the HSA Applet. David Shupe, NHSC on behalf of the Herschel Science Ground Segment Consortium http://herschel.esac.esa.int/DpHipeContributors.shtml. This presentation is intended as an overview and “quick start”. It sets the stage for hands-on work

qamar
Download Presentation

Introduction to HIPE and the HSA Applet

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Introduction to HIPE and the HSA Applet David Shupe, NHSC on behalf of the Herschel Science Ground Segment Consortiumhttp://herschel.esac.esa.int/DpHipeContributors.shtml

  2. This presentation is intended asan overview and “quick start” • It sets the stage for hands-on work • Some features will not be covered • Read also the “HIPE Owners Manual” • Included with your installation

  3. HIPE is designed to easily handleyour Herschel data It handles Herschel data types It includes routines to go from raw data to publishable results It places the official pipeline software on your desktop It is modern and actively developed

  4. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Help and Documentation • A First Script – Unpacking the Workshop Data

  5. Introduction to HIPE • Key Data Concepts • Objects and Data Products • Contexts • Memory management • A Visual Tour of the HIPE Interface • Help and Documentation • A First Script – Unpacking the Workshop Data

  6. Wrapping data as objectslets HIPE do more for you • Sensible defaults for “double-click” • Usually a viewer • A module or function works on any object of a given class • Use the same methods or interfaces • The system can suggest the right task or option • Less for you to remember

  7. Contexts allow data to be organizedin tree structures • Members can be “named” or simply “numbered” • Data items are loaded only when they are accessed(“lazy loading”) • Access is the same,wherever the data are

  8. The Observation Context consolidatesall the different types of data ObservationContext Auxiliary logObsContext Calibration Level0(Raw Data) Points to everything that was usedfor processing Easy to choose what to look ator ignore

  9. Computer memory is automaticallymanaged for you • The system allocates memory whenever a new object is created • The system runs “Garbage Collection” as needed • Reclaims memory • Bar in lower right cornershows status

  10. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Views and Perspectives • The Welcome! Perspective • The Work Bench Perspective • The Product Browser Perspective • The Herschel Science Archive Perspective & Applet • Help and Documentation • A First Script – Unpacking the Workshop Data

  11. Views are windows or regionswith specialized functions Views can be resized or minimized Views can be endlessly rearranged Views don’t shut down when closed Views are accessible by menu

  12. A Perspective is a specific collectionof view windows Welcome! Product Browser HSA Work Bench Reset Perspectives are pre-defined Your re-arrangement of views is “sticky” You can reset the arrangement to the default

  13. The Welcome! perspective is a map to the major parts of HIPE

  14. Most of the action takes placein the Work Bench perspective

  15. The Variables view provideseasy access to each data item Delete all variables Delete selected variable Double-click for default (usually a viewer) Right-click or control-click for a quick list of options

  16. The Outline view showsdetails and structure of a data item Red = not loaded yet Black = loaded in session For any item: Double-click for default (usually a viewer) Right-click or control-click for a quick list of options

  17. Jython commands are executedin the Console window Log window contains system messages History tab keeps a record of your commands

  18. The Editor view containsscripts, viewers, and task interfaces Script controls (see tooltips) Script window

  19. The Observation Viewer breaks outall the pieces of your observations

  20. The Tasks view enables quick startupof applicable modules simpleFitsWriter task • Double-click to launch • Drag-and-drop variables into parameter slots • The “Applicable” tab shows all the available tasks for a selected variable

  21. The Navigator view enablesbrowsing of your filesystem • Double-click to load a script or a FITS file • Refresh • Show Hidden • Create “user area”

  22. The Product Browser perspectiveprovides powerful search and retrieval

  23. Interact directly with the archiveusing the HSA perspective Log in using your Herschel credentials Start the HSA Applet

  24. Query the archive by obsid,target name, instrument, public status If OBSID is known… Set to “Public” to browsereleased data Search by name Specify instrument

  25. The HSA Applet can send datadirectly to your HIPE session Select “Send to External Application” then “All”

  26. For multiple downloads,use “Retrieve” to receive tarfiles Select “Retrieve” then “All”

  27. The Shopping Basket will collectseveral observations …or reprocess with last released version You can retrieve thecontents at one time…

  28. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Help and Documentation • Starting the Help system • User Guides, Tutorials and How-Tos • Detailed Developer Documentation • A First Script – Unpacking the Workshop Data

  29. The Help and Documentationare accessed in your web browser • Start the help system by 1 of 3 ways: • Menu “Help” -> Help Contents • “show_help” in the app directory of HIPE • Right-click on variable • Help in URM (Users Ref. Manual) • Help in DRM (Developers Ref. Manual) • Only one help system can be running

  30. The Help system includesuser guides, tutorials and how-tos

  31. The Help system includes allthe detailed developer documentation

  32. Introduction to HIPE • Key Data Concepts • A Visual Tour of the HIPE Interface • Help and Documentation • A First Script – Unpacking the Workshop Data • Quick introduction to scripting • Where your local data are stored • Edit and run the UnpackWorkshopData.py script

  33. Variables are simply namesthat point to data items No ‘type’ or declaration needed Assignment creates the variable:a = 1b = 2 Strings can use single or double quotes:c = “hello world”e = ‘hi there’ Reference: Scripting and Data Mining, Sec 1.2

  34. Comments, continuations, and printinghave a simple syntax The comment character is the pound sign# this is a comment The continuation character is the backslashx = a + b + \ c * d * e A formatted string uses C-style format characters and the percent signprint “integer = %d, real = %f” %(j,x)

  35. if-then-else blocksare denoted by indentation Reference: Scripting and Data Mining, Sec 1.8.1 Syntax:if condition1:block1elif condition2:block2else:block3 Notice that blocks are denoted by indentation only Example:if (0 <= x <= 10): y = y - 10print “Value is in range [0,10]”elif (10 < x < 20): print “Value is in range [10,20]”else: print “Value not in range [0,20]”

  36. for loops are not just for integers Reference: Scripting and Data Mining, Sec. 1.8.2 Syntax of a for loop:for var in sequence:block The sequence can be any list, array, etc. Examples:for petin [“cat”, “dog”, “bird”]: print petfor i in range(10): # [0, 1, …, 9] print i # prints numbers 0-9 Therangefunction returns a list of integers. In general range(start,end,stepsize)where start defaults to 0 and stepsize to 1.print range(5)# [0, 1, 2, 3, 4]

  37. Your “local pools” are stored in the “lstore” directory • Defaults: • Macs /Users/[userid]/.hcss/lstore • Linux /home/[userid]/.hcss/lstore • Windows C:\Documents and Settings\[userid]\.hcss\lstore • You can change this using “Edit” and then “Preferences” then “Local Store”

  38. The UnpackWorkshopData.py scriptwill unpack the workshop data • Get the script from agenda webpage • Follow link at top of PACS section • Unpacks all .tgz files in a directory • …which you downloaded previously • These are special compressed pools • Not the format of HSA tarballs

  39. Edit Line 6 of UnpackWorkshopData.pyand run script to unpack …then put cursor in between “”on Line 6, and ctrl-V or cmd-V to paste HIPE tip: Select data directory, then copy with ctrl-C or cmd-C… Navigator Editor Console output:

  40. Backup: Alternate ways of unpacking .tgz datasets to your lstore Courtesy of Phil Appleton

  41. Data downloads DOWNLOADS OF gzipped-tarred (tgz) example datasets should already have been done before coming to the workshop If you forgot its https://nhscsci.ipac.caltech.edu/sc/index.php/Workshops/Feb2011WorkshopDownloads for the full instructions and https://nhscsci.ipac.caltech.edu/workshop/Feb_2011_NHSC_DPWorkShop_Repository/ DataDownload/ for the data repository

  42. Data Pools are stored in a lstore directory on your computer: It lives in the .hcss directory Macs /Users/[userid]/.hcss/lstore Linux /home/[userid]/.hcss/lstore windows GOTO folder C:\Documents and Settings\[userid]\.hcss\lstore (to make it : then pulldown file>>new>>folder lstore)

  43. OLD WAY OF COPYING FROMCOMMAND LINE (Script given earlier is best)

  44. Safest to move the pool to the .hcss/lstore BEFORE UNCOMPRESSING .../.hcss/lstore Data (pool) (already copied from WebDav) DO NOT UNTAR or UNZIP YET 1342184579.tgz move to lstore without uncompressing yet Once in lstore you can untar or unzip .../.hcss/lstore/1342184579 THIS IS A POOL

  45. On the Mac the .hcss directory is "hidden" so best do move with a command line instruction For UNIX users (Mac or LINUX) * move the dataset to the HIPE local store location >mv dataset.gz ~/.hcss/lstore THEN UNCOMPRESS For WINDOWS * the easiest thing is to drag and drop the subdirectory dataset.gz or zip into the location of the .hcss/lstore directory located for example in C:\Documents and Setting\[user]\.hcss\lstore THEN UNCOMPRESS

  46. How do you know if it worked? >cd ~/.hcss/lstore (e.g. /Users/phil/.hcss/lstore on Mac) >ls 1342184579 >cd 1342184579 (This directory contains many sub-directories) that look like this >ls herschel.ia.dataset.Product herschel.ia.dataset.image.SimpleImage herschel.ia.obs.ObservationContext herschel.ia.obs.QPLog herschel.ia.obs.auxiliary.AuxiliaryContext herschel.ia.obs.auxiliary.missingtm.MissingTmProduct herschel.ia.obs.auxiliary.ool.OolProduct .......

More Related