1 / 21

MAST Status

October 10, 2003. FUSE FOAC Meeting. 2. MAST Processing. HST and FUSE data are ingested and distributed with the same system utilizing the same hardware and softwareFUSE data has a separate tool to get the data to us, the rest of the process is identical MAST performance has been dismal for users

brinly
Download Presentation

MAST Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. October 10, 2003 FUSE FOAC Meeting 1 MAST Status Faith Abney Archive Branch, STScI

    2. October 10, 2003 FUSE FOAC Meeting 2 MAST Processing HST and FUSE data are ingested and distributed with the same system utilizing the same hardware and software FUSE data has a separate tool to get the data to us, the rest of the process is identical MAST performance has been dismal for users getting ALL data, this is not a problem specific to FUSE.

    3. October 10, 2003 FUSE FOAC Meeting 3

    4. October 10, 2003 FUSE FOAC Meeting 4 MAST Performance – What happened? ACS data overwhelmed our current infrastructure Distribution rates tripled Ingest rates tripled Software was not capable of handling massive dataset sizes and volume of requests Hardware reached capacity

    5. October 10, 2003 FUSE FOAC Meeting 5

    6. October 10, 2003 FUSE FOAC Meeting 6

    7. October 10, 2003 FUSE FOAC Meeting 7 Planned Changes for ACS Before ACS was placed on HST we knew: Data volumes would increase Data retrievals would increase in number and in size Planned for a redesign of our DADS Distribution and Ingest software Current software was written with smaller, legacy instruments in mind Planned for upgrading existing hardware but maintaining current architecture

    8. October 10, 2003 FUSE FOAC Meeting 8 Infrastructure Changes New Distribution and Ingest software was severely delayed for several reasons Distribution software scheduled for delivery by the end of the year We realized current hardware environment was not reliable or robust, needed to change the architecture of the hardware as well This added delays as we determined best products, purchased and implemented them Spent a considerable amount of money to make sure this infrastructure is scalable and reliable

    9. October 10, 2003 FUSE FOAC Meeting 9 Infrastructure Changes – Hardware DADS runs on a mixed system of VMS, Unix and Tru64 machines Many problems interfacing between the machines Inefficient, files got passed between the system many times before completing a request Extensive complexity keeping all systems configured properly Machines running at maximum capacity and no longer upgradeable Jukeboxes prone to many hardware failures due to extensive use

    10. October 10, 2003 FUSE FOAC Meeting 10 Infrastructure Changes – Hardware Simplify Environment with powerful machines - Processing Purchased and installed a SunFire 15k processing machine 68 UltraSparc III CPUs, 272 GB RAM, 7 Domains Will house all the processing power for DADS distribution, ingest, databases and data processing Test, Development and Operational systems all running in the same environment Able to dynamically allocate more resources during peak archive usage times

    11. October 10, 2003 FUSE FOAC Meeting 11 Infrastructure Changes – Hardware Simplify Environment with powerful machines - Storage Purchased and installed an EMC Symmetrix 8830 32 TB of usable space, all fully mirrored Will allow us to move all of our data onto spinning disks with immediate availability Primary retrieval of data has been from MO platters housed in jukeboxes Started populating on August 12th, all FUSE data ingested since then is on this spinning disk Eliminates need to get data from jukeboxes Complete migration of all FUSE data planned for early next year

    12. October 10, 2003 FUSE FOAC Meeting 12 Infrastructure Changes – Bandwidth Mars Outreach pictures, GOODS data distribution and DADS distribution maxed out our internet bandwidth Combination of those very successful projects brought our delivery of data to a trickle Successfully negotiated with Goddard and our ISP to increase our bandwidth temporarily while we decide on final solution Now average time for distribution has been around 12 hours Exploring options for data distribution such as External hosting of static datasets Using more mirror sites Distributing some datasets on DVD

    13. October 10, 2003 FUSE FOAC Meeting 13 FUSE ingest specific problems Tool used to push data to MAST for ingest developed intermittent failures Limited testing abilities necessitated a couple attempts to correct cause of intermittent failure Fix now implemented that appears to be consistently working Manual daily check for completeness as part of routine operations procedures

    14. October 10, 2003 FUSE FOAC Meeting 14 FUSE ingest specific problems Disks filling up FUSE data for ingest was sent to same disk being used to stage ACS data for media production Filled up frequently Just too many things going on with that disk despite our efforts to increase its size Now have a large disk (~150 GB) used almost exclusively for FUSE data ingest

    15. October 10, 2003 FUSE FOAC Meeting 15

    16. October 10, 2003 FUSE FOAC Meeting 16 Summary/Progress of Changes All test and development systems are running on SunFire 15k with EMC storage Portions of operational MAST are running on SunFire 15k and EMC storage Tru64 has been completely replaced Distribution will be completely on SunFire 15k by the end of the year – no more VMS interfacing Ingest requirements and design are complete, currently being coded, scheduled for May 2004 delivery Then all VMS goes away

    17. October 10, 2003 FUSE FOAC Meeting 17 Summary/Progress of Changes About 1 TB of data has been moved to the EMC storage for primary retrieval All will be moved by early next year Continue to write the data to the MO platters for safe storage A few local disks still remain for ingest, those will move to the EMC within the next few months Purchased an additional spare jukebox Can be swapped in to replace a broken one

    18. October 10, 2003 FUSE FOAC Meeting 18

    19. October 10, 2003 FUSE FOAC Meeting 19

    20. October 10, 2003 FUSE FOAC Meeting 20 Secure Delivery of Data DADS 10.2 (scheduled for delivery late November) has the ability to use Secure FTP Users must be running an SSH daemon on their system that supports SSHv2 DADS 10.3 will support staging of proprietary data Users can ftp data from the stage using their DADS username and password

    21. October 10, 2003 FUSE FOAC Meeting 21 Lessons Learned Better communication with users Sensitivity needed during critical times for FUSE project Routine meetings between designated FUSE and MAST staff recommended

More Related