1 / 24

DSARCH OVERVIEW

DSARCH OVERVIEW. Dataset Archiving Utility Overview By Zaihua Ji. Outline. Definitions and how DSARCH fits into RDA functions Purpose of DSARCH Introduction of DSARCH Usage Procedures of using DSARCH to archive data Convert existing datasets Publish MSS/RDA Server data filelists.

aarmas
Download Presentation

DSARCH OVERVIEW

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DSARCH OVERVIEW Dataset Archiving Utility Overview By Zaihua Ji

  2. Outline • Definitions and how DSARCH fits into RDA functions • Purpose of DSARCH • Introduction of DSARCH Usage • Procedures of using DSARCH to archive data • Convert existing datasets • Publish MSS/RDA Server data filelists

  3. RDA Components

  4. Definition of Metadata -- Metadata that summarizes a dataset, such as author, title, summary, etc. -- Metadata that defines external properties of RDA files, such as file locations on MSS/RDA Server, sizes, file packaging (tar, COS block, etc.), dataset sub-groups of files, archive file type (P, B, W, …), etc. -- Metadata that defines internal properties of RDA files, such as data format (GRIB, ASCII, etc.), variables, spatial & temporal ranges, internal metrics (eg., number of grids or stations), etc.

  5. Metadata Coverage

  6. Current Data Archive Flow

  7. Dsarch Work Flow for Existing dataset

  8. Dsarch Work Flow for New Dataset

  9. What DSARCH For • Archives/Retrieves data files to/from MSS and RDA server • Records/Retrieves dataset metadata (mainly VSN metadata) to/from RDADB • Organizes archived data files for one dataset into sub-datasets, called groups

  10. What DSARCH Does • Defines group and dataset information in RDADB • Automatically selects MSS VSN by default, other options are available • Archives data files on the MSS and/or RDA server and saves transaction records into the RDADB • Retrieves dataset/group/file information from the RDADB • Modifies/corrects information in the RDADB • Copies MSS files back to local computers (RDA Server, bison, etc) • Moves MSS/Web data files from one dataset/group to another • Removes files from MSS or RDA server • RDADB Maintenance Functions • Backups dataset/group/file information from RDADB into CVS Archive • Restores dataset/group/file information into RDADB from CVS Archive

  11. Advantage of Using DSARCH

  12. General DSARCH Usage dsarch [[-DS] dsnnn.n] [Action Option] [Mode Options] [Information Options] • Quotes [] indicate optional • Three Option categories: Action, Mode, and Information (Info for short) Options. • Action options - specify what tasks this utility to execute • Mode options - modify behaviors of given actions • Info options - pass information, one or multiple values, to run DSARCH • An option is given in either short or long names, eg. -DS or -Dataset • All Action and Mode options, as well as Info option -IF (-InputFile), must be given on command line; all other info options can be given either on command line or in one or multiple input files specified by option -IF

  13. Categories of Action Options • Dataset Actions - create, modify and retrieve dataset information in RDADB • Group Actions - create, delete, modify and retrieve dataset group information in RDADB • File Actions • archive files onto MSS and RDA server • move and delete files on MSS and RDA server • create, delete, modify and retrieve information about data files in RDADB • File-Name Actions - generate, release, retrieve and archive MSS file names (VSN - Volume Serial Number) per RDADB card bank • Info Actions - create, modify and retrieve all information for datasets, groups in datasets, on MSS and RDA Server • Backup Actions - archive, restore, and check CVS backup history information for datasets, groups, and MSS and RDA Server files

  14. Procedure for Creating a New Dataset • Create SCCS dataset archive file with ‘Search & Discovery Metadata’ only (for dataset main webpage) • Create initial dataset record in RDADB per utility ‘filldataset’ • Set flag for using RDADB to ‘Y’ and modify the dataset info • Create group information if needed • Archive local files onto MSS and/or RDA Server • Set flag for using RDADB to ‘P’ or ‘I’, and publish dataset file lists per utility ‘publish_filelist’

  15. Procedure to Convert Existing Dataset • Set flag of using RDADB to ‘Y’ • Edit VSN section from the SCCS dataset document to create a file named ‘dsnnn.n.sccs’ • Use ‘myconvert’ utility to reformat ‘dsnnn.n.sccs’ into ‘dsnnn.n.mss’, which is an input file designed for RDADB • Use DSARCH to enter the input file ‘dsnnn.n.mss’ into RDADB • Set flag of using RDADB to ‘P’ or ‘I’, and use ‘publish_filelist’ utility to publish dataset file lists

  16. Convert SCCS Metadata for An Existing Dataset myconvert dsnnn.n.sccs > dsnnn.n.mss • dsnnn.n.sccs - MSS file information modified from VSN file section of SCCS dataset metadata file, by inserting conversion control keys for information about the dataset, groups and files. See examples later • dsnnn.n.mss - DSARCH input file holding dataset, group, and/or MSS file information

  17. Conversion Key Categories • Dataset Keys - mark public and/or internal MSS dataset notes • Group Keys - build up group information, such as group index, group name (ID), group titles, and public and/or internal MSS group notes • MSS File Keys - setup MSS file information, such as format key to specify what information should be collected from the description part of a file line, and keys for data and file formats • A special key LB, which can be used to turn on (LB<=><br>, for example) or off (LB<=>) of a html line-break symbol for multiple line notes

  18. Dataset Keys • DM - public MSS dataset note; description lines following ‘DM<=>’ are collected, including empty lines • DI - internal MSS dataset note; description lines following ‘DI<=>’ are collected, including empty lines

  19. Group Keys • GN - group name or group ID, up to 20 characters. It is given in format of GN<=>GroupID, and is optional if GI is present, eg. GN<=>List-A • GI - group index number. It is automatically assigned in order, if GN is present. It is given format of GI<=>GroupIndex, eg. GI<=>5

  20. MSS File Keys • DE - file note or description • SD - shared note for multiple files. Insert a line of 'SD<=>' to record description lines as a shared note • LF - local file name for a MSS VSN file name • FF - file format, up to 10 characters, eg. 'FF<=>BI.TAR' means the following files are binary COS-blocked and then tarred • TF - data format, up to 10 characters, eg. 'TF<=>ASC.IMMA' means the following files hold data files in both ASCII and IMMA formats • RL - length of each record in a file • RN - number (count) of records in a file • FMT - format for description part of file information lines; eg. 'FMT<=>LF,,DE' means that description part of each line is split into three columns by delimiter ','; the first column is local file name, the second is ignored and the third is file note

  21. Example of ds540.1.sccs p- TF<=>BINARY FF<=>TAR FMT<=>LF, GN<=>MSGSTD2 p- MSG, Standard Statistics, 2x2, 1800-1997 py61279 stdg3.1800.1809.tar, less than .5 MB py61281 stdg3.1810.1819.tar, less than .5 MB …. FF<=>Z.TAR GN<=>MSGSTD2_R2.2 p- MSG, Standard Statistics, 2x2, release 2.2, 1998-2004 p- Release 2.2 is exclusively new data for 1998-2004. For convenience, data from p- Group ID <a href="#MSGSTD2">MSGSTD2</a> for 1990-1997 have been included in these files. py88809 msg_2deg/stdg3.1990.1999.tar, 33.983 MB by88810 b/u y88809 py88811 msg_2deg/stdg4.1990.1999.tar,34.098 MB by88812 b/u y88811 ….

  22. Snapshots of Web Display ….. …..

  23. Publish MSS/Web Filelists Publish_filelist [-t] dsnnn.n • Set flag of using RDADB to P or I before publishing a dataset (Y is fine for option -t to publish test filelists) • Generates html file of MSS public filelist to root directory of given dataset • Generates html file of MSS internal filelist to internal dataset directory • Generates html index files of RDA Server data in data and sub-data directories of given dataset, unless manually created index html files exist already • All html filelist files will be built dynamically and cached

  24. Update Dataset With New Data • Run DSARCH to archive new data files onto MSS and/or RDA server • No need republish filelists • Filelists including the information of new archived data files will be recached automatically when the filelist web pages are accessed by users

More Related