1 / 11

Moving Archive Data to the EMC Storage Array March 14, 2003

Review and implementation plan for migrating archive data to the EMC storage array, improving availability, throughput, and reducing downtime.

nbarbee
Download Presentation

Moving Archive Data to the EMC Storage Array March 14, 2003

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Moving Archive Data to the EMC Storage ArrayMarch 14, 2003 Faith Abney Implementation Review

  2. Implementation Review

  3. Motivation for Migration • EMC has higher availability and throughput • Less downtime • Data gets to users quicker • More online processing space available on EMC • Current processing areas not large enough to process all retrieval requests when they are submitted • Requests are routinely queued and held until processing space is available Implementation Review

  4. Portion of EMC array to be used for faster access to Archive data • Currently all data is on MO platters in jukeboxes • Limited by number of platters mounted • Slow read/write access • Greatly affected by hardware downtime • Moving data to the EMC array will eliminate these bottlenecks which are due to data being in a jukebox Implementation Review

  5. Portion of EMC array to be used for faster access to Archive data • Calling this area the “Data Depot” to avoid confusion with other staging areas • DADS will check this area first (DADS 10.1) • if files are available will use them • if not, it will get them from MO • Ingest software will be updated to write appropriate data to the depot after write to MO (DADS 10.1) • Data to depot will come from read of MO platter, verifying the write to MO Implementation Review

  6. Data going to the Depot • All POD files (raw science data) for instruments with OTFR – 1.5 TB • ACS, NICMOS, STIS, WFPC2 • All legacy data - .5TB • GHRS, FOS, FOC, WFPC, HSP, AST • Calibration reference files - .15TB • FUSE data - .3 TB • Jitter files, Metadata - < .1TB • Mission scheduling data < 50 GB • SMS, MTL, MSC, ORB Implementation Review

  7. Data not going to the Depot • Some data is rarely requested and does not need faster access: • Calibrated data for which OTFR is available • Raw Engineering data • Log and Ancillary data Implementation Review

  8. Methodology: Requirements • Any movement of Data from MO platter to EMC will compete with both Ingest and Distribution for Jukebox access. • Migration should not prevent operations from meeting performance goals. • Migration of data off jukebox can proceed as background task • Task becomes high priority if jukebox failure rate begins to impact performance goals in operations. • Migration should allow for selection of highly requested data classes to be segregated and moved first. • POD files (raw science) for OTFR prime candidate to reduce overall retrieval times. Implementation Review

  9. Methodology: Options • MMM server within NSA • Used to populate MO from older Sony media • Required manual intervention for simultaneous efficient operations and transfer performance • Well understood operationally • Bulk copy using primary platters • Same copy mechanism used for other mirror archives (CADC, ECF) • Not a routine operational procedure, manual intervention • DADS 10.* beta test environment • V280R loading uncertain • Rapid Test and Build cycle for beta test environment • Copy from safe store platters using DADS 9.* test environment. • Highly manual (requires platter loads) • No operational impact (impacts testing schedule only) Implementation Review

  10. Methodology: Prescription • Perform trade studies to evaluate the quickest and least intrusive method. • Develop contingencies to accelerate transfer should jukebox failures become more frequent. • Initiate transfer and develop a good understanding of maximum transfer rate before predicting a completion date. • Lessons learned from MO migration • Note that DADS 10.1 delivery makes transfer progress non linear! Implementation Review

  11. Schedule • DADS 10.1 – scheduled for Spring – contains: • Software to populate Depot on ingest • Software to be able to read from the Depot • Tools to migrate existing data to Depot already written • Migration can start as soon as EMC Depot is connected to operational system • Migration completion estimate will be prepared after we complete acceptance testing and get actual performance numbers using the EMC and our tools Implementation Review

More Related