1 / 31

Content

The PEACE Database: Maintaining your SDDAS database access, data export and data visualisation tools R.J. Wilson and A.N. Fazakerley Mullard Space Science Laboratory Spring 2006 Version 1.1. Content. Introduction Overview of the activities Recommended approach Graphic “how to” guide

beate
Download Presentation

Content

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The PEACE Database: Maintaining your SDDAS database access, data export and data visualisation toolsR.J. Wilson and A.N. Fazakerley Mullard Space Science LaboratorySpring 2006Version 1.1

  2. Content • Introduction • Overview of the activities • Recommended approach • Graphic “how to” guide • Explanation of each activity Note:This document tells you how to update your own local SDDAS installation, on your own machine.If you use msslAB to run SDDAS then all this is done for you.

  3. Introduction (1) The PEACE Database uses IDFS format files, and a suite of software known as SDDAS is freely available which enables you to work with the database, export data and visualise the data The Database changes with time as new data is added and as calibrations change Processed data in the Database (SPINPAD, moments, spacecraft potential estimates etc.) may be reprocessed from time to time Tools discussed here allow you to check whether your local copy of the database is up to date, and to make updates when necessary Other tools allow you to update the SDDAS software We also provide suggestions on maintenance of your local copy of the database

  4. Introduction (2) The Master Server for all PEACE data is at MSSL (UK) Its IP number is currently 128.40.70.117 A mirror copy of the PEACE database is at SwRI (USA) Its IP number is currently 129.162.154.69 It is intended to be regularly updated to track changes at MSSL

  5. Overview of the activities • Sync With Server • Update your local copy of the Hierarchy from root server i.e. Tells your computer what products are available • MegaMetaPromote • Using your local copy of the Hierarchy, update your local copy of the data catalogue, from which you may promote data. • Download Beta AppsGet the latest versions of SDDAS applications to plot/export data. • Clearing out your promoted data areaStop your local Hard Disk filling up, and stops you using older data.

  6. Recommended approach to maintenance • Proposed Schedule and proposed order or execution: • Clearing out your promoted data area:At least monthly for Cluster, weekly for Double Star(Weekly) • Download Beta Apps:At least monthly(Monthly) • Sync With Server:Monthly, or if you hear a new virtual has been added.(Monthly) • MegaMetaPromote (always after Sync with Server):Weekly at least.If you’re working with very recent data consider daily.(Daily) • For reference, the main MSSL group computer is updated at the frequency in blue.

  7. Caring for SDDAS…in pictures… Select Admin Tools • Sync with Server • MegaMetaPromote • Download Beta Apps

  8. Select Admin Tools • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Config Files Editor” • Select Pull Down Menu:“Action > Sync With Server…” • From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”. • If files are already in sync it will tell you, otherwise you get: • You can select just parts the you want – but easier to just select “Select All” and then “OK” • You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

  9. Select Admin Tools • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Config Files Editor” • Select Pull Down Menu:“Action > Sync With Server…” • From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”. • If files are already in sync it will tell you, otherwise you get: • You can select just parts the you want – but easier to just select “Select All” and then “OK” • You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

  10. Sync with Server • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Config Files Editor” • Select Pull Down Menu:“Action > Sync With Server…” • From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”. • If files are already in sync it will tell you, otherwise you get: • You can select just parts the you want – but easier to just select “Select All” and then “OK” • You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

  11. Sync with Server • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Config Files Editor” • Select Pull Down Menu:“Action > Sync With Server…” • From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.71.151Mars Express 129.162.154.22Then select “Ok”. • If files are already in sync it will tell you, otherwise you get: • You can select just parts the you want – but easier to just select “Select All” and then “OK” • You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

  12. Sync with Server • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Config Files Editor” • Select Pull Down Menu:“Action > Sync With Server…” • From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.70.117 • Then select “Ok”. • If files are already in sync it will tell you, otherwise you get: • You can select just parts the you want – but easier to just select “Select All” and then “OK” • You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

  13. Sync with Server Material that is not up to date on your computer • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Config Files Editor” • Select Pull Down Menu:“Action > Sync With Server…” • From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.70.117 • Then select “Ok”. • If files are already in sync it will tell you, otherwise you get a GUI like the one shown here: • You can highlight just parts the you want to update, and press “OK” – or just select “Select All” and then “OK” • You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”DON’T FORGET TO SAVE! (and make sure you have permissions to save the files) Material that is no longer on the master server,so can be deleted from your local system.

  14. Sync with Server • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Config Files Editor” • Select Pull Down Menu:“Action > Sync With Server…” • From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left.Cluster/Double Star 128.40.70.117 • Then select “Ok”. • If files are already in sync it will tell you, otherwise you get a GUI like the one shown here: • You can highlight just parts the you want to update, and press “OK” – or just select “Select All” and then “OK” • You are returned to the earlier screen. Now you MUST use the pull down menu:“File > Save”DON’T FORGET TO SAVE! (and make sure you have permissions to save the files) When saving, it is normal to see a pop-up box asking you to confirm that you want to overwrite the current .cfg files

  15. Always do this after, not before, Sync with Server! MegaMetaPromote • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Promote Meta Data” • Select “Source” • You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII… • At his point just click “Ok” as you want all the Cluster spacecraft. • This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.]

  16. Always do this after, not before, Sync with Server! MegaMetaPromote • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Promote Meta Data” • Select “Source” • You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII… • At his point just click “Ok” as you want all the Cluster spacecraft. • This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.]

  17. Always do this after, not before, Sync with Server! MegaMetaPromote • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Promote Meta Data” • Select “Source” • You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) • At this point just click “Ok” as you want all the Cluster spacecraft. • This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.]

  18. Always do this after, not before, Sync with Server! MegaMetaPromote • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Promote Meta Data” • Select “Source” • You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) • At this point just click “Ok” as you want all the Cluster II spacecraft. • This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.]

  19. Always do this after, not before, Sync with Server! MegaMetaPromote • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Promote Meta Data” • Select “Source” • You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) • At this point just click “Ok” as you want all the Cluster II spacecraft. • This returns you to earlier window, with CLUSTERII in the Source at the top.Now just click “Go!” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.]

  20. Download Beta Apps • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Download BETA Apps” • Select “Go” • Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

  21. Download Beta Apps • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Download BETA Apps” • Either select “Go”, or… • Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

  22. Download Beta Apps • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Download BETA Apps” • Either select “Go”, or… • Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want.

  23. Download Beta Apps • At console prompt, type:sdcontrol & • Select “Administration Tools” • Select “Download BETA Apps” • Either select “Go”, or… • Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want. • To finish now select “Quit”.

  24. Clearing out your promoted data area • When you promote data (or VIDFs, PIDFs) they go in to the hierarchy subdirectories such as those beneath: /home/sddas/data/CLUSTERII (for Project CLUSTERII) /home/sddas/data/DOUBLESTAR (for Project DOUBLESTAR) • These files stay there and take up room on your hard disk. • i.e. a single sensor Cluster PEACE data file (*D) from PAD may be 10 Mb in size… If want both HEEA & LEEA for a short period (hours), over all 4 spacecraft that’s 80 Mb. • A single day Double Star PEACE 3DF data file (*D) can easily be 500 Mb,If you want both TC-1 and TC-2 then that’s 1000Mb, or 1 Gb. • You local Hard Disk will quickly become full if you do not delete these files after use.This can then stop your entire computer working properly due to lack of disk space. • Delete files you no longer need!!! Files to delete are…

  25. Clearing out your promoted data area • PIDFS: Only small files (~80k), relatively few of them. • VIDFS: Only small files (~80k) but many. • Header files: Small-ish files (up to a few Mb) but many. • Data files: LARGE files (10s to100s of Mb) and many of them. • Why delete the small (or large) files at all? Maybe they’ve been updated on the master server with new calibrations or re-processed. Deleting them ensures you promoted the latest version next time you use them. • A brute force method of deleting all promoted files, for project CLUSTERII, assuming your local data directory is in /home/sddas/data/ :rm /home/sddas/data/CLUSTERII/*/*/*/*.pidf.v2 - delete PIDFsrm /home/sddas/data/CLUSTERII/*/*/*/*V.v3 - delete version 3 VIDFsrm /home/sddas/data/CLUSTERII/*/*/*/*[IV] - delete version 2 VIDFs, both file typesrm /home/sddas/data/CLUSTERII/*/*/*/*[HD] - delete Header and Data filesSimilar for projects DOUBLESTAR or MARS. Do not delete anything from the Database directories, i.e. leave /home/sddas/data/CLUSTERII/*/*/Database alone.The use of wildcards illustrated above is safe as no files in the Database directory match the ending letters above. (…but if you do after all delete from the Database directories you can recover using MegaMetaPromote)

  26. Explanation of the Activities

  27. Anatomy of IDFS 1 • In IDFS you have a hierarchyfrom mission down to instrument: Project > Mission > Experiment > Instrument i.e. CLUSTERII > CLUSTER-1 > PEACE > 3DF • At the Instrument level you have “virtual instruments”, which are subsetsof the instruments data. i.e. CP3DFH and CP3DFL • “virtuals” have a set naming convention: (Project, experiment, description) CP3DFH = Cluster PEACE 3DFHEEA DP3DF = DoubleStar PEACE 3DF (single sensor) CFUNIT = Cluster FGM UNIT vector DSPRIME = DoubleStar Staff PRIME parameters Name must be unique and no more than 8 characters. DHIAPRIM = DoubleStar HIAPRIME parameters DHIDPRIM = DoubleStar HIDPRIME parameters (Exception to convention: SPINAD with virtuals such as HSPAD & LSPAD)

  28. Anatomy of IDFS 2 There are 4 types of file, VIDF, PIDF, Header and Data • PIDF: Plot Interface Description File. • Just one per virtual, contains labels and unit choices. Filenames of: {virtual}.pidf.v2 - text file that you can read i.e. CPPADH.pidf.v2 • VIDF: Virtual Instrument Description File. • Contains calibrations and conversions for units. • Can be many VIDFs per virtual, each file contains a start and stop time inside it. • Filename contains start time (to nearest minute). • VIDFS must not overlap in period. {virtual}{start-time}V - version2 VIDF text file (for human use) {virtual}{start-time}I - version2 VIDF binary file (for SDDAS use) {virtual}{start-time}V.v3 – version3 VIDF text file (for human & SDDAS use) {Start-time} in format: (year)(doy)hhmm, ie. 20050381400 for 2005-Feb-07 14:00 i.e. CPPADH20050381400V.v3 • HEADER and DATA file: they come as a pair, contain the actual data. • Header file is tiny, data file is BIG. • Can be up to 3 different file-pairs per day for Cluster, 1 file-pair per day for Double Star. {virtual}{start-time}H - binary file (for SDDAS use) {virtual}{start-time}D - binary file (for SDDAS use) i.e. CPPADH20050381400H & CPPADH20050381400D

  29. Sync With Server • Updates your machine with the root server that contains the latest version of the Hierarchy. i.e. including any new virtuals recently added. Modifies: • /usr/local/sddas/config/DB.cfgContains list of what servers you can promote from for each Project. (IP addresses listed rather than server names.) • /usr/local/sddas/config/PROJECTS.cfgContains list of full hierarchy for each Project. • Doing a Sync with Server updates these files for you.It shows differences only between your current copy and the master server.Makes sure you have permissions to write/save these files, if not use the chmod command or run as superuser. • You need to know the IP address of the server you wish to Sync to: CLUSTERII/DOUBLESTAR IP at MSSL 128.40.70.117

  30. MegaMetaPromote • In your local SDDAS Data directory (e.g. here we assume it is/home/sddas/data/ ?) you have your Hierarchy. Under each experiment you will have a Database directory, i.e. /home/sddas/data/CLUSTERII/CLUSTER-1/PEACE/Database • Within the Database directory, each virtual has 4 files in two pairs: {virtual}.HD.DBF {virtual}.HD.NDX - Header & Data list {virtual}.I.DBF {virtual}.I.NDX - VIDF list • These contain what your computer thinks is available for promotion,as opposed to what is really available for promotion from the master server. • Do a MegaMetaPromote to update this list from the master server.Makes sure you have permissions to write/save these files in *ALL* the Database directories, if not use the chmod command or run as superuser. • This is essential as you can only promote the data that your computer thinks is available...

  31. Download Beta Apps • SDDAS is made up from a number of separate programs ( ~80!) • To ensure you have the latest version of each you Download Beta Apps • Download Beta Apps updates older versions of programs with newer ones from Southwest Research Institute (SwRI). • Older versions are copied to /usr/local/sddas/backups rather than being over-written. Should the new version be worse (i.e. new bugs) than the last you can then restore the old one from the backups directory.Makes sure you have permissions to write/save these files in the /usr/local/sddas/backups directory, if not use the chmod command or run as superuser. • Do not use “Download Applications” – this is rarely updated by SWRI,Always use the Beta Apps.

More Related