1 / 22

AMESD training On-the-job Training CICOS – Mars 2011 M.Clerici (JRC)

AMESD training On-the-job Training CICOS – Mars 2011 M.Clerici (JRC). Programme de la semaine. Monday 14/3: Review of the eStation basic concepts (summary of week 1). Updates Installation: concepts and practice (v. 1.1). Installation of historical archives (v 1.1). Tuesday 15/3:

flynn
Download Presentation

AMESD training On-the-job Training CICOS – Mars 2011 M.Clerici (JRC)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AMESD training On-the-job TrainingCICOS – Mars 2011M.Clerici (JRC)

  2. Programme de la semaine • Monday 14/3: • Review of the eStation basic concepts (summary of week 1). • Updates Installation: concepts and practice (v. 1.1). • Installation of historical archives (v 1.1). • Tuesday 15/3: • Version 1.1 of the ‘system’: new/modified functionalities. • Data visualization with EMMA. • Environmental reports editing using Reporter.

  3. Programme de la semaine • Wednesday 16/3: • Analysis of the products available on the eStation. • Detailed presentation of products of interest for CICOS (precipitations, ET). • Thursday17/3 and Friday 18/3: • Derived product generation: • a. make_derived mechanism • b. Example: Cumulated precipitations (RFE), ET. • Timeseries: • a. time-series mechanism (templates) • b. Examples: cumulated precipitations.

  4. Lundi 14/03 Installation of updates • Guidelines: • There are 5 packages to be installed: SYSTEM, PS, EMMA, PS-DATABASE and the static_data. • ‘SYSTEM’ is the customization of the Ubuntu menus and the web interface (http://estation-ps/ and http://estation-emma/). • PS and EMMA, the main components, are installed on both PC2 and PC3: they can be activated in Recovery Mode (mode de secours). • PS-DATABASE is an upgrade of PostGreSQL database, with new tables, for instance for the ‘Reporter’ component. • Static_data are files like layers (shapefiles) used by EMMA. • NOTE: PS-database and Static_data have to be installed ONLY on PC2 (PS), and will be copied to PC3 through the ‘synchronization’.

  5. Lundi 14/03 Installation of updates Procedure: On PS: stop the synchronisation with EMMA (PC3). Stop the services (get,ingest,make_derived). Open the Gdebi package, select the update debian file.

  6. Lundi 14/03 Installation of updates Procedure - 2: Navigate to the location of the debian file and select it. Look at the file description, details and included files:

  7. Lundi 14/03 Installation of updates Procedure - 3: Click on ‘Install’ package and follow the operations in ‘Terminal’.

  8. Lundi 14/03 Installation of updates Note: To verify the installation of the packages you can type in a terminal: dpkg –l | grep amesd It will show the various version of the AMESD paskages installed:

  9. Lundi 14/03 Installation of updates Procedure - 4: at the end of the installation on PC2 and PC3 Do a manual synchronisation with EMMA (PC3). Re-start the services (get,ingest,make_derived). Re-activate the Synchronization. Important: on the two machines (PS and EMMA) open firefox and go to: ‘Outils -> Effacer mes traces’ Select all switches and execute, to clean the cookies, temporary web pages, ..

  10. Lundi 14/03 Installation of Historical Archives 1.1 Description and distribution of the updated Release-Note 744-EN. Description of the Archive Structure (file .xml, .sql and tif files in sub-directories) Procedure for the installation: Open the tool from the menu (see figure). B. Select the Archive on the disk C. Follow the execution: Log file ingest_archive.log Directory where data are copied

  11. Lundi 14/03 Creation of Archive of products • This operation can be done to provide data to other Users in the Region. • Procedure: • A. Open the tool from the menu. • B. Select 1 product • C. Select the sub-products: only Ingested, only derived, both of them or manually • D. Select ‘All dates’ (time frame • does not work in 1.1) • E.Select the target directory. • F. Follow the execution, looking at the • contents of the target directory • NOTE: Creation of an archive is faster • than its ingestion.

  12. Lundi 14/03 Ingestion of newly created Archive • The operation is done exactly as for the Historical Archives. • Procedure: • A. Copy the archive (i.e. the directory like fewsnet_rfe_15032011) to local machine. • B. Verify the permissions !! They should be 755, so that the ingestuser can access the directory. • C. If permissions are not ok, use the command: • sudo chmod –R 755 <ingest_archive_dir> • Where <ingest_archive_dir> can be something like: ~/Desktop/fewsnet_rfe_15032011 • D. Follow the same procedure as for Historical Archives.

  13. Mardi 15/03 Analysis of functionalities in SYSTEM 1.1 From the menu Applications -> PS -> Administration

  14. Mardi 15/03 Analysis of EMMA/Reporter functionalies • Main functionalities of EMMA are shown on the PC3: • Display of images and timeseries. • Navigation panel, additional layers and outmask. • GUI Admin: creation of new Users, creation of a new legend (FEWSNET-REF) on the basis of the colorbar in the product doc. • GUI Admin: assign the newly created legend to the product. • Reporter: very quick overview. • few bugs reported.

  15. Mercredi 16/03 Copy of data from RVF to CICOS station • Introduction:we have verified that the RVF station reception is OK, a part from the power cut. Therefore a procedure has been prepared to transfer data from RVF to CICOS station. • On RVF station: • Open a terminal on the Receiving Station and: • cd /data-space/data/AMESD_archive/2011/ • In this dir there are subdirs named after the day (e.g. 074) • Open the file copy_data.sh: first lines are: • #!/bin/bash • date=$(date +%Y%m%d) • selection_jour=0[789]? • target=adminuser@192.168.0.5:/media/Elements/RVF_data • The script creates archives for ET, MPE, VGT4Africa products and copy them to the CICOS HD, mounted on the PC2 (PS).

  16. Mercredi 16/03 Copy of data from RVF to CICOS station • 4. Modify, if needed, the <selection_jour> variable; if you are missing only days • 076, 077 and 078 you can change to: • selection_jour=07[678] • If the CICOS disk is NOT mounted on PS Station (IP address 192.168.0.5), or you would copy to another destination, change the <target> definition. For example, to copy to the homedir of adminuser on PS change to: • target=adminuser@192.168.0.5:/home/adminuser • 6.Execute the script (./copy_data.sh), wait until completion, the unmount the HD.

  17. Mercredi 16/03 Copy of data from RVF to CICOS station • On CICOS station: • On PS, open a terminal and: • cd PS/scripts/ • Verify the script import_rvf.sh is there, than execute it with a command like: • ./import_rvf.sh /media/Elements/RVF_data/VGT4Africa_archive_20110316.tgz • Where the argument is the filepath of the archive generated at RVF • 3. Data are extracted from the archive and copied to /data/ingest/ (verify)

  18. Jeudi 17/03 Verification of the RVF Station • The RVF station was stopped by power cut yesterday: • On the Receving Station, the Signal Level is ~ 87, but SNR ~ 98 (much higher than at CICOS). • The eStation Services (get/ingest) automatically re-started at the machine re-boot. • We verified that the Manual Synchro takes now only few minutes (ok). • The ingestion of historical data has been launched.

  19. Jeudi 17/03 Product Description • Presentation and discussion of the AMESD Product Description document (French – document distributed as hard-copy). • Earth Observation general concepts: geo-stationary and polar (sun-synchronous) satellites, general structure of a processing chain, taking as example the VGT4Africa Burnt Areas product (files passed to Jean). • Presentation on Fire Products: MSG-FIRA, VGTAfrica and MODIS. • Presentation of Precipitation Products: TAMSAT RFE, FEWSNET RFE, MSG MPE. • Note: can CICOS contribute to TAMSAT product by providing to Met Office additional gauge measurements for calibration ?

  20. Vendredi 18/03 RVF Station • Verify the status of the Station (reception, ingestion of historical data). • Re-activate the services: get/ingest/make_derived/synchro. • Produce a report of the PC2/3 (system_report.sh). • Export data from receiving station (copy_data.sh) and import to CICOS.

  21. Vendredi 18/03 Activity at CICOS: make_derived • Overview of the make_derived mechanism: libraries (python/bash), make_derived.sh and .xml, crontab. Supporting document: Developer Handbook (in hard-copy and distributed to the trainees). • Example of LSASAF_ET: • derived_mosaic.sh • derived_10d30min_avget.sh • derived_10dcumet.sh • Analysis of the results and preparation of legends (ToBeDone).

  22. Vendredi 18/03 Activity at CICOS:LSASAF_ET • A PS (partial) station is prepared on one of the Trainees’ computer (Olivier), including: • The up-to-date code of the processing in /opt/facilities/PS/processing/lsasaf_et (this code is NOT YET on the AMESD eStation). • LSASAF_ET HDF5 data (only 1 day) to show the ingestion. • LSASAF_ET mosaic files for 2011 (to show the make_derived). • On the basis of the test dataset: • ingest_lsasaf.sh is executed • derived_mosaic.sh (manually). • derived_10d30min_avget.sh (manually). • derived_10dcum.sh (manually) • And output analyzed. NOTE: there is a scaling factor of 10.000 in the dataset.

More Related