1 / 32

The IRIS Data Management System The View From the Engine Room: Data and Metadata

The IRIS Data Management System The View From the Engine Room: Data and Metadata. Rick Benson IRIS Consortium. Assembled Data. Non-SEED format data: CSS, IMS, AH, SEGY, SEG2,SAC, MATLAB,”RAW”. 235 unique data sets currently: From 1966 (Borovoye….Apollo Lunar…) to 2005 (SAFOD).

sachi
Download Presentation

The IRIS Data Management System The View From the Engine Room: Data and Metadata

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The IRIS Data Management System The View From the Engine Room: Data and Metadata Rick Benson IRIS Consortium IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  2. Assembled Data • Non-SEED format data: • CSS, IMS, AH, SEGY, SEG2,SAC, MATLAB,”RAW” • 235 unique data sets currently: • From 1966 (Borovoye….Apollo Lunar…) to 2005 (SAFOD) • Easy access to reports, descriptions, detail • Managed within Oracle, so you can select by • Time • Location • Name • Type (USGS, OBSIP, PASSCAL, UTEP, OTHER) http://www.iris.washington.edu/SeismiQuery/assembled.phtml IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  3. Assembled Data throughSeismiQuery, DemoShows location of reports, mechanism to “search”. IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  4. Look here Assembled Data Requests- Option 1 IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  5. Assembled Data Requests- Option 2 Assembled requests are formatted like a breq_req except: .Assembled in the 1st field entry The .LABEL field determines the data Example: .Assembled .NAME Frederik Tilmann .LABEL TOBA_96-015 .EMAIL tilmann@esc.cam.ac.uk .INST Univ of Cambridge .MAIL Bullard Lboratories Department of Earth Sciences Madingley Road, Cambridge Cambridgeshire, CB3 0EZ, UK .PHONE .FAX .MEDIA TYPE ftp .ALT MEDIA TYPE .END IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  6. SeismiQuery demo Example: Where are STS-2 sensors operating in 2006 located? Make map IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  7. Historical Data: a.k.a.SeismoArchives, demo IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  8. Why is metadata important? They are the Rosetta Stone- making it possible to decode information well into the future • Increases accessibility, “where are stations, what are • station names, what types of instruents”, etc • Retention of context- who operates instruments, etc • Carefully created, they create both short and long term • benefits by capturing “tribal knowledge” when it’s • still available, and ultimately creates an accurate history of true ground motion IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  9. Metadata In(di)gestion • Functionally, we use “technical” metadata, in that it is used to • describe how the seismic system behaves, and limited • software description (like compression, etc) • Complete description of waveform data for perpetual archive, • so that users can make FULL use of data • DMC must have metadata representing all holdings in order to • service user requests • DMC gets metadata from network operators in dataless SEED, • and makes metadata “portable” amongst heterogeneous • systems • Whenever anything changes at a site, new mdata gets exchanged IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  10. How can I access DMC Metadata? • Five ways, choose between interactive or passive • SeismiQuery http://www.iris.edu/SeismiQuery/index.html • Anonymous ftp: ftp.iris.washington.edu:/pub/RESPONSES/DATALESS_SEEDS • Metadata Aggregatorhttp:/www.iris.edu/mda • “FetchResp” client using DHI • Wget: wget -o /dev/null -O - --post-data=“web=‘http://www.iris.edu/pub/RESPONSES/DATALESS_SEEDS’”/ 6. Email a breq_fast request to dataless@iris.washington.edu IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  11. SeismiQuery Click on tab labeled “Responses” IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  12. How can I access DMC Metadata? • Five ways, choose between interactive or passive • SeismiQuery http://www.iris.edu/SeismiQuery/index.html • Anonymous ftp: ftp.iris.washington.edu:/pub/RESPONSES/DATALESS_SEEDS • Metadata Aggregatorhttp:/www.iris.edu/mda • “FetchResp” client using DHI • Wget: wget -o /dev/null -O - --post-data=“web=‘http://www.iris.edu/pub/RESPONSES/DATALESS_SEEDS’”/ • Email a breq_fast request to dataless@iris.washington.edu IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  13. pub/RESPONSES These are updated after each modification (new dataless SEED loaded) http://www.iris.washington.edu/pub/RESPONSES/ IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  14. How can I access DMC Metadata? • Five ways, choose between interactive or passive • SeismiQuery http://www.iris.edu/SeismiQuery/index.html • Anonymous ftp: ftp.iris.washington.edu:/pub/RESPONSES/DATALESS_SEEDS • Metadata Aggregatorhttp:/www.iris.edu/mda • “FetchResp” client using DHI • Wget: wget -o /dev/null -O - --post-data=“web=‘http://www.iris.edu/pub/RESPONSES/DATALESS_SEEDS’”/ • Email a breq_fast request to dataless@iris.washington.edu IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  15. Metadata Aggregator Step 1: Select Network IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  16. Begin Drill-down Step 2: Click on any station IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  17. {More information…. } Click on “More Information” IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  18. Check basic information, responses.. IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  19. How can I access DMC Metadata? • Five ways, choose between interactive or passive • SeismiQuery http://www.iris.edu/SeismiQuery/index.html • Anonymous ftp: ftp.iris.washington.edu:/pub/RESPONSES/DATALESS_SEEDS • Metadata Aggregatorhttp:/www.iris.edu/mda • “FetchResp” client using DHI • Wget: wget -o /dev/null -O - --post-data=“web=‘http://www.iris.edu/pub/RESPONSES/DATALESS_SEEDS’”/ • Email a breq_fast request to dataless@iris.washington.edu IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  20. FetchResp Usage • Get FetchRespRel.tar bundle Contact chris@iris.washington.edu or rick@iris.washington.edu • Cd to your un-tarred FetchResp directory • Edit the file runFetchResp to reflect your $HOME location export HOME=/users/rick/Desktop/fetchResp • Edit the file query.txt which is used to generate requested info. • Just type “./runFetchResp” in source directory. IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  21. How can I access DMC Metadata? • Five ways, choose between interactive or passive • SeismiQuery http://www.iris.edu/SeismiQuery/index.html • Anonymous ftp: ftp.iris.washington.edu:/pub/RESPONSES/DATALESS_SEEDS • Metadata Aggregatorhttp:/www.iris.edu/mda • “FetchResp” client using DHI • Wget: wget -o /dev/null -O - --post-data=“web=‘http://www.iris.edu/pub/RESPONSES/DATALESS_SEEDS’”/ • Email a breq_fastrequest to dataless@iris.washington.edu IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  22. Email a dataless@iris.washington.edu breq_fast request: Edit an ascii text file like this .NAME Rick Benson .INST IRIS DMC .MAIL 1408 NE 45th St, Suite 201, Seattle, WA 98105 .EMAIL rick@iris.washington.edu .PHONE .FAX .MEDIA ftp .LABEL DATALESS.NE .END ? NE 2005 01 01 00 00 00 2006 10 02 00 00 00 1 ??? IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  23. miniSEED example =============================================================================== STATION LOCATION CHANNEL NETWORK TIME KSM BHE MY 2006,149,00:00:00.0195 [1] (R) # samples in record: 354 sample_rate: 32760 multiplier: -1638 activity flags: I/O and clock flags: data quality flags: # of blockettes: 2 time correction: 0 begin data offset: 128 begin blkette offset: 48 BLOCKETTE 1000: encoding format: STEIM 2 Compression (val:11) word order: 68000/SPARC word order data record length: 9 reserved: 0 BLOCKETTE 100: actual sample rate: 20.0000 flags: to be defined - 0000 0000 reserved[0-2]: 0 0 0 IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  24. Working with miniSEED Submit a breq_fast request to miniseed@iris.washington.edu If I request/download miniSEED: setenv ALT_RESPONSE_FILE “local.dataless 2. Run rdseed 3. Rdseed now uses the Alternate headers to Emulate fullSEED Goal is to enable the use of “updated” metadata without having to exchange waveform data again. IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  25. Request Processing- “Engine Room” View the mail queue at http://www.iris.edu/data/data.htm IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  26. Your request is processed FIFO, more or less What takes time? (In May 2006, we received 25,872 requests, (!not including spam) percolating down to 18,952 shipments) Harvest is run, assembling headers (blockettes) and timeseries info (which Station/Day files to “fetch”) GT LPAZ % 20000725123226 20000725133226 BH% GT PLCA % 20000725123226 20000725133226 BH% Found: GT PLCA BHE 2000,207,00:00:00 2000,208,00:00:00 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Found: GT PLCA BHN 2000,207,00:00:00 2000,208,00:00:00 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Found: GT PLCA BHZ 2000,207,00:00:00 2000,208,00:00:00 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 GT SBA % 20000725123226 20000725133226 BH% GT VNDA % 20000725123226 20000725133226 BH% Found: GT VNDA BHE 2000,207,05:43:58 2000,207,14:11:01 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Found: GT VNDA BHE 2000,207,05:43:58 2000,208,00:00:00 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Found: GT VNDA BHN 2000,207,05:43:58 2000,207,14:12:40 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Found: GT VNDA BHN 2000,207,05:43:58 2000,208,00:00:00 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Found: GT VNDA BHZ 2000,207,05:43:58 2000,207,14:12:28 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Found: GT VNDA BHZ 2000,207,05:43:58 2000,208,00:00:00 D reqtime: 2000,207,12:32:26 2000,207,13:32:26 Harvest finished. Finish harvest Sun Jun 4 16:26:55 PDT 2006 IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  27. Building SEED, cont’d Go to the mass store for data: Isilon, Jbod, then silo last, running time_window to extract delimited data to nearest record Run POD: [dbserv1:[96] more pod.out set CREATOR IRIS DMC set SEED_VERSION 2.4 set SEED_LABEL 20000725_123226.LON-34_-16LAT-63_-50.BH set HEADER_PATH /seed/Wen-Che_Yu.8/HAR000 set DATA_PATH /seed/Wen-Che_Yu.8/data_files pod /seed/Wen-Che_Yu.8/seed /seed/Wen-Che_Yu.8/h.* 4096 32768 999,365 999,365 Flushing output volume POD terminating normally Run verseed, seedsniff for statistics Ship data to ftp, tape, DVD, etc. IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  28. Jweed Tutorial Download the InstallAnywhere version from http://www.iris.washington.edu/manuals/#1 View a Quicktime tutorial at http://www.iris.washington.edu/tutorials/jweed/ IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  29. JWeed intro, cont’d IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  30. SAC-DHI tutorial Get java code DHI_Access.jar and shared library libdhi.so files ported to your machine Read the SAC-DHI help file IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  31. in the shell you are using set two env variables: (assuming csh) # tells sac where to look for external shared objects (SO), in this case libdhi.so setenv SACSOLIST /the/path/to/where/you/saved/the/sofile/libdhi.so # tells the java code where the access classes are setenv ALT_CLASSPATH /the/path/to/where/you/saved/the/jarfile/DHI_Access.jar here is an example: run sac SAC> load read_dhi # loads the library (examines the env variable SACSOLIST) # you can get all arguments by entering read_dhi. 'read_dhi' will print out its usage. SAC> read_dhi usage: read_dhi List_serversusage read_dhi available_data net stn loc chn start_time end_time example: read_dhi available_data IU * 00 BH* 2005/11/30 09:00:00 2005/11/30 10:00:00 Note: use "--" for blank location code. You can use wildcards for network, station, location or channel entries. SAC-DHI cont’d IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

  32. Data Problem Reports (DPR’s) http://www.iris.edu/data/dpr.htm Network operators make observations: **************************************************************************** DATA PROBLEM REPORT ASL2005:115 2005/08/01 ASL DCC Valerie Peyton ASL SDV IU */* 2005,212,21:29:00 Problem Description INCORRECT TIME SEED COMMENT REFERENCE S 454 Timing MAY be in error, as GPS system is not operating consistently. The GPS system at SDV was not able to obtain a solid lock on the GPS satellites during this time. This caused the GPS system to not operate consistently, and produced timing jumps. Data should be used with caution. PROBLEM RESOLUTION REPORT ASL2005:115 2005/08/05 ASL DCC Valerie Peyton Description of Problem Resolution RESOLUTION The SDV GPS timing system recovered at 2005, 214, 11:41. Timing quality returns to normal at this time. END **************************************************************************** IRIS Data Users Shortcourse Westward Look Resort Tucson, Arizona

More Related