1 / 82

Backup Methods For a Hot Site

Backup Methods For a Hot Site. Dieter W. Storr Los Angeles Times 23 August 2005. B/R Methods. Existing Backup Method Experiences Mirroring or Replicating Fast Copy of Data Proposals and Costs Future Technology Lessons learned. Existing Backup Method. From disk (databases) Copy to

huy
Download Presentation

Backup Methods For a Hot Site

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Backup Methods For a Hot Site Dieter W. Storr Los Angeles Times 23 August 2005

  2. B/R Methods • Existing Backup Method • Experiences • Mirroring or Replicating • Fast Copy of Data • Proposals and Costs • Future Technology • Lessons learned Dieter W. Storr -- www.storrconsulting.com

  3. Existing Backup Method • From disk (databases) • Copy to • 3490 / 3590-1 / VTS • Then, copy to • 3590-1 (cartridge) Dieter W. Storr -- www.storrconsulting.com

  4. ADABAS 6.2.2 Back-up at LA Times Dieter W. Storr -- www.storrconsulting.com

  5. B/R Methods “Companies that relied on tape or on third-party provider found in many cases they had difficulty meeting their recovery time objectives.” Source: http://www.drj.com/articles/spr02/1502-07.html Dieter W. Storr -- www.storrconsulting.com

  6. B/R Methods “Flaws in tape-based data backup may beleaving enterprises without key information and could lead to legal exposure under emerging laws such as Sarbanes-Oxley, say data backup and recovery experts. “ Source: 15 Apr 2004 | SearchSecurity.com Dieter W. Storr -- www.storrconsulting.com

  7. B/R Methods • In a survey of 500 IT departments completed … found that as many as 20% of routine, nightly backups fail to capture all data. • 40% of IT managers had been unable to recover data from a tape when they needed it • More than 23% sought to use data stored on tape backups more than 20 times in a year Source: 15 Apr 2004 | SearchSecurity.com Dieter W. Storr -- www.storrconsulting.com

  8. B/R Methods Are tapes really so bad? LA Times experiences? Dieter W. Storr -- www.storrconsulting.com

  9. Tape Problems 1 November 2002: • Six tape drive errors • Delay Dieter W. Storr -- www.storrconsulting.com

  10. Tape Problems 24 March 2003: • Only two channel paths per tape controller were provided • Slow restore time Dieter W. Storr -- www.storrconsulting.com

  11. Tape Problems 5 October 2003: • 3590 tape drives were not defined to DFSMS (SMS) • ADABAS restore and application test cancelled Dieter W. Storr -- www.storrconsulting.com

  12. Tape Problems 6 December 2003: • VTS problems with GDG datasets • End-user functions couldn’t be tested Dieter W. Storr -- www.storrconsulting.com

  13. Tape Problems 5 August 2004: • Restore jobs had to wait for an input tape that was being used by another restore job • Delay Dieter W. Storr -- www.storrconsulting.com

  14. Tape Problems 30 October 2004: • Packages didn’t arrive in time, due to a thunderstorm that affected FedEx delivery • Major delay Dieter W. Storr -- www.storrconsulting.com

  15. Tape Problems 30 October 2004: • Automated tape library experienced unit address problems during the restore process • Delay Dieter W. Storr -- www.storrconsulting.com

  16. Tape Problems 30 October 2004: • VTS logical tapes were not shipped to Wood Dale (HSM level 2, SAR level 2) • Delay Dieter W. Storr -- www.storrconsulting.com

  17. Tape Problems 30 October 2004: • Confusion about when to load DRP1 and DRP2 tapes, before or after IPL • Delay Dieter W. Storr -- www.storrconsulting.com

  18. Tape Problems 30 October 2004: • ICIS libraries were not backed up to tape • Application tests were not possible Dieter W. Storr -- www.storrconsulting.com

  19. Tape Problems 8 December 2004: • Load problems • Tapes were loaded before IPL and not after IPL • Major delay Dieter W. Storr -- www.storrconsulting.com

  20. Tape Problems 8 December 2004: • Experienced problems when trying to restore MIG1 data, e.G. DRADABC0 job • Major delay Dieter W. Storr -- www.storrconsulting.com

  21. Tape Problems 8 December 2004: • Recall sent by FedEx tapes to SunGard • One damaged package arrived without tapes • Restored DATA one generation back (-1)System was generation (0) Dieter W. Storr -- www.storrconsulting.com

  22. Tape Problems 21 March 2005: • Level 2 tapes for VTS not being sent off-site (but have been on the list) • Application team couldn’t test all data Dieter W. Storr -- www.storrconsulting.com

  23. Tape Problems 5 August 2005: • 3590-1 cartridges ejected, not found • DSS8370W - TMS SHOWS TAPE N00318 OUT OF AREA “DRP1”,SLOT 00031 • Delay Dieter W. Storr -- www.storrconsulting.com

  24. Time Warner employee data missing May 2, 2005: 5:51 PM EDT NEW YORK (CNN) - Time Warner Inc. said Monday that data on 600,000 current and former employees stored on computer backup tapes was lost by an outside storage company and that the Secret Service is now investigating. Dieter W. Storr -- www.storrconsulting.com

  25. Lost Backup Tape Held Ameritrade Client Data Wednesday, April 20, 2005 - LA Times … package was damaged during shipping between vendors ….. fourth tape is still missing…… The tapes may have included customers’ Social Security numbers ….. Dieter W. Storr -- www.storrconsulting.com

  26. Info On 3.9M Citigroup Customers Lost Monday, June 6, 2005 – CNN.COM Citigroup, the nation's biggest financial services company, said that UPS lost the tapes while shipping them to a credit bureau in Texas. Dieter W. Storr -- www.storrconsulting.com

  27. Costs for Tape Backups • SunGard recovery services • Offsite tape storage • Tape handling • Shipping per test • Special extra pick-ups Yearly $150,000 Dieter W. Storr -- www.storrconsulting.com

  28. Costs • Not capable to restore one day • $$ ??? • Last December: 2 weeks to rebuild manually (?) customer tables • Does it make sense to restore more than 2 days back ?? Dieter W. Storr -- www.storrconsulting.com

  29. Costs Example: 20 employees x $140 per day x 10 days = $28,000 And they couldn’t work on other projects $140 is based on $51,100 yearly income Dieter W. Storr -- www.storrconsulting.com

  30. Quantitative Risk Analysis Single Loss Expectancy • SLE = Single Loss Expectancy • EF = Exposure Factor, for example 50% or .50 • AV = Asset Value, for example $1,000,000 SLE = EF * AV SLE = .5 x $1,000,000 = $500,000 Dieter W. Storr -- www.storrconsulting.com

  31. B/R Methods Reducing tapes Dieter W. Storr -- www.storrconsulting.com

  32. B/R Methods Reducing tapes • Stacking datasets to 3590-1 cartridges • Using Delta Save Facility from ADABAS Dieter W. Storr -- www.storrconsulting.com

  33. B/R Methods Reducing tapes • Using Forward Index Compression (FIC) from ADABAS • Using larger block size for 3590 tapes = 256K, supported by ADABAS Dieter W. Storr -- www.storrconsulting.com

  34. Delta Save Facility (DSF) Dieter W. Storr -- www.storrconsulting.com

  35. Delta Save Facility Dieter W. Storr -- www.storrconsulting.com

  36. Within an index block the part of the index value that is identical to the forward part of the previous index value is suppressed. B/R Methods Forward Index Compression Rochester Gas & Electric Space savings: • Normal Index: 37% - 55% • Upper Index: 21% - 69% Dieter W. Storr -- www.storrconsulting.com

  37. B/R Methods IBM Magstar 3494 / Virtual Tape Server (VTS) SunGard LA Times Dieter W. Storr -- www.storrconsulting.com

  38. B/R Methods VTS problems LA Times: • Completion code A78 RC 18 • We switched from VTS to 3590-1 cartridges Dieter W. Storr -- www.storrconsulting.com

  39. B/R Methods VTS problems Virginia Information Technologies Agency: • Ran 2003/2004 into the same problem system completion code A78 RC 18 • We … converted … to 3490/3590 physical tapes • Problem solved Dieter W. Storr -- www.storrconsulting.com

  40. B/R Methods Disk to Disk • Mirroring • Hardware • Software • Replicating • Software Dieter W. Storr -- www.storrconsulting.com

  41. B/R Methods – Enterprise Server Enterprise Server NT / 2000 / XP Hot Site UNIX Dieter W. Storr -- www.storrconsulting.com

  42. B/R Methods – Open System Hot Site Dieter W. Storr -- www.storrconsulting.com

  43. B/R Methods Marty Stewart Disaster Recovery Manager AnMed Health: “…we’d rather have a server that’s running slower than having no server at all.” Dieter W. Storr -- www.storrconsulting.com

  44. Disk Mirroring ASSO Benefits • Asynchronous disk mirroring can provide better physical protection by supporting extended physical distances. • No loss of committed transactions in synchronous storage (mirroring/RAID) on a CPU failure DATA ASSO DATA Dieter W. Storr -- www.storrconsulting.com

  45. Disk Mirroring ASSO Limitations • No protection from data corruption • Secondary site is not guaranteed to betransitionally consistent, in the case of asynchronous mirroring. • Client application must be re-started after failure and need to be aware of failure DATA ASSO DATA Dieter W. Storr -- www.storrconsulting.com

  46. Disk Mirroring ASSO Limitations • Synchronous mirroring and RAID devices can add overhead to application performance. • Redundant/specialized high availability hardware/software can be expensive and restricted to use for backup purposes only. DATA ASSO DATA Dieter W. Storr -- www.storrconsulting.com

  47. Disk Mirroring ASSO Limitations • Secondary copy of data is not available for use – low hardware utilization. • Need to replicate everything on disk, no selectivity of data replication DATA ASSO DATA Dieter W. Storr -- www.storrconsulting.com

  48. Example For Disk Mirroring Back Up / Hot Site S/390 UNIX EMC 5700 SRDF remote mirroredsynchronized OC-3 link SRDF remote mirroredsynchronized 12-15 miles EMC 5700 S/390 UNIX Main Platform Dieter W. Storr -- www.storrconsulting.com

  49. B/R Methods • Can we buy used Enterprise Servers? • Yes…..and inexpensive • OP system is free for D/R Search for “selling used mainframes,” for example: http://www.used-line.com/fdc3236-find-dealer.htm http://www.azure.co.uk/ etc. Dieter W. Storr -- www.storrconsulting.com

  50. Dedicated line broadband speeds and prices • T-1 - 1.544 megabits per second (24 DS0 lines) Ave. cost $400.-$650./mo. • T-3 - 43.232 megabits per second (28 T1s) Ave. cost $6,000.-$16,000./mo. • OC-3 - 155 megabits per second (100 T1s)Ave. cost $20,000.-$45,000./mo. • OC-12 - 622 megabits per second (4 OC3s) no price • OC-48 - 2.5 gigabits per seconds (4 OC12s) no price • OC-192 - 9.6 gigabits per second (4 OC48s) no price Source: http://www.infobahn.com/research-information.htm prices updated: 12 May 2005 Dieter W. Storr -- www.storrconsulting.com

More Related