Consolidating Data Management with Enterprise Storage and Backup Systems

1. Consolidating Data Management with Enterprise Storage and Backup Systems Barry Ribbeck – Director, Systems, Architecture & Infrastructure, Rice University Hubert Daugherty – Manager, Digital Media Services, Rice University Susie McClure – Systems Analyst, Rice University

2. Rice University Carnegie R1 Designation Ranked in the top 25 schools in US ~ 5000 Students ~ 1100 Faculty ~ 1500 Staff

3. Enterprise Storage and Backups • Enterprise - a distributed resource provided centrally • Storage - a place to put your stuff • Backup - Describes what you are doing when you are not moving forward. When it works, it can be incorporated into your career progressions plan.

4. A Large New Network Project • Provided a funding opportunity • Defined storage as part of the critical need • Provided the required network infrastructure for enterprise storage (Data can live anywhere) • A fresh look at technologies for enterprise storage and backups

5. Data Assessment Findings • Many distributed resources managed inconsistently • No consistent backup strategy • Few economies of scale • Data management plan – buy more disks • Growth Rate of digital storage needs was increasing • No differentiation of data retention or DR requirements • Old, outgrown and dying backup hardware

6. The Available Resources • Very talented and knowledgeable staff • Funding • Support from administration • Understanding from the campus

7. The Challenges • Changing EVERYTHING at once • In order to transition, we would have to keep 2 systems operational with the same amount of staff who are already over taxed with all of the other changes • New skill sets required to maintain new resources

8. Goals • If possible, the changes MUST have a positive visible affect on the customers • Storage must be able to scale (1/2 PB in 7 years) with no step function costs. • Support for Information Lifecycle Management (ILM) • Must support a Multi-tiered storage model • Must be a model capable of integrating multi vendor storage • Support costs defined for 5 years • Staged backups with only full backups to Tape

9. RFP to Storage Vendors • SAN with NAS Gateway design defined • Invited 7 vendors to respond to the RFP • Very wide dispersion of cost vs benefit • Bluearc Titan carried the day with a big iron model and 50 TB of initial storage

10. Defined Storage Usage • Email Mailbox storage (NO QUOTAS!) • Client Home directories and shares • 5000 students, 1000 faculty, 1500 staff, 1000+ guests (with quotas) • Database storage – Oracle, MSQL, Postgress (Sakai, Coldfusion,etc..) • Video Archives • Research Data Scratch space

11. Backups • Remember Katrina & Rita? • Tape is like Listerine • We needed a MAID to clean up our backup processes • What about Disruptive Technologies? • Hardware choices • Copan Systems & Sun/StorageTek • Software choice • Commvault

12. Storage Details • San Storage components • Positives • Standards Based • Multiple vendors enable competition • Negatives • Difficult to modify SAN architecture • Expensive network components • Nas Gateway • Provides storage abstraction and management

13. Systemic Performance Which System Elements Need Additional Optimization?

14. Systemic Performance Surprise, it’s the regular stuff!

15. Systemic Performance Dual 6509 frames with 2 dedicated 10G interlinks. One for layer 2 and one for layer 3.

16. Physical / Logical Architecture

17. What we Learned • The integration of a lot of new technology took longer than anticipated even though all of the vendors supported well understood standards like NDMP. • Outsourcing the integration would have been a mistake due to the complexity and the need for in house knowledge.

18. Bluearc Lessons and Vendor Report • Clients can’t compete • Outages MUST be coordinated with everything that uses the system • Clustered Titan nodes are a requirement • Vendor • Great support – very responsive, good technical people • Work well with other vendors • Problems we have worked through • NDMP, Replication, LDAP, still some to things to address, getting them to understand Enterprise in HE

19. Copan Lessons and Vendor Report • Good Product very reliable • Good support team, works well with other vendors • Good hardware, very robust, are meeting their claims. 1 drive and 1 shelf failure in 1 year. • Software is solid for what we use it for

20. Sun Lessons and Vendor Report • Very good support • Very good product. • Robot and disk drives failed and were replaced and were quickly replaced under support contract.

21. Commvault Lessons and Vendor report • Software difficult to get installed even with their professional services • Support is lacking in depth but they are very willing to work with us. • Windows backups work very well • Support works well with other vendors • Integration was very difficult and time consuming • Software patches were very frequent and problematic • Support tracking system is terrible • NDMP support is very challenging • When the product works, it works well • Documentation was poor

22. Contact Information & References • Barry Ribbeck Barry.R.Ribbeck@rice.edu • Hubert Daugherty hd@rice.edu • Susie McClure smcclure@rice.edu • www.bluearc.com • www.copansystems.com • www.sun.com (Storagetek) • www.commvault.com • www.inphase-technologies.com