CIT 470: Advanced Network and System Administration

1. CIT 470: Advanced Network and System Administration Backups CIT 470: Advanced Network and System Administration

2. Topics • Backup Decisions • Types of Backups • Backup Hardware • Backup Software • Snapshots and CDP • Cron • Backup Security CIT 470: Advanced Network and System Administration

3. Backup Decisions Why? Why are you backing up data? What would happen if you lost data and didn’t back up? What types of data do you have? What? What to back up—entire system, or specific filesystems? What OS to backup? What other things to backup—MBR, LVM? When? When is the best time to backup? How often? Where? Where will backup occur? Where to store backup volumes? Who? Who is going to provide backup system? Who will do backups? How? How are you going to do backups? Tape, mirrors, off-site, etc. CIT 470: Advanced Network and System Administration

4. Why Backups? • Accidental deletions. • Hardware failures. • Data corruption. • Security incidents. • Plan for the worst. • System catches fire. • Fire spreads to replicated systems. • Sprinklers destroy backup system in data ctr. CIT 470: Advanced Network and System Administration

5. What to Backup? Backups for your backups • Be able to restore your backup server (software + backup volume db) in case it’s down. Which peripherals? • How many drives per server? • What is the capacity of each drive? How were they partitioned? • Drive partitions must be same as before disaster for restore from backup to work. • fdisk –l • lvmcfgbackup CIT 470: Advanced Network and System Administration

6. Filesystem / Data Types Operating System Standard OS image on server. Software Software + config files specific to server. Data Data files specific to server. CIT 470: Advanced Network and System Administration

7. Backing Up Selected FS Saves media space and network traffic • But OS is small compared to data today. Harder to administer • Must remember or document which fs to backup for each server. Easier to split up between volumes • Can easily distribute backup of a server across different backup volumes on a per fs basis. Worst case • Forget to backup an important filesystem. CIT 470: Advanced Network and System Administration

8. Backup Entire System Complete automation • Create script to parse /etc/fstab and LVM, then backup every disk filesystem. • Do this once, then it works on all servers. Worst case • Increase network traffic by a few percent. CIT 470: Advanced Network and System Administration

9. Backup Types Image level • Backup raw disk partition or entire disk. • Back up every byte on drive, used or not. • Use compression to eliminate GBs of zeros. • Cannot restore individual files. Filesystem level • Backup files within filesystem. • Backup tool must understand filesystem. • Can restore files, no backup of unused blks. CIT 470: Advanced Network and System Administration

10. Backup Types Full backup Complete copy of all files from a particular time. Backup: slow, requires high capacity. Restore: fast, simple. Incremental backup Storage of changed files since last backup. Backup: fast, may store multiple per tape. Restore: slow, complex (requires multiple tapes) CIT 470: Advanced Network and System Administration

11. Capacity Planning: Space Partition: 40GB Full backup every week. Daily incremental backups. 50% full now, grows 2GB per day Tape capacity needed Day 1: 40GB Day 2: 2GB … Day 7: 12GB Day 8: 40GB CIT 470: Advanced Network and System Administration

12. Capacity Planning: Time Fileserver: 4TB Full backup must finish overnight (8 hours) Tape drive: 40MB/s = 144 GB/hr = 1.15TB/night Need 4 tape drives running simultaneously. Additional concerns: Network performance btw file & backup servers. Does any capacity need to be reserved for restores? Actual performance vs. manufacturer specs. CIT 470: Advanced Network and System Administration

13. Capacity Planning: Media How much media do you need? Determined by policy and schedule. How long are full backups kept? How often are incrementals recycled? How often are tapes moved off-site? CIT 470: Advanced Network and System Administration

14. Choosing a Backup Drive • Reliability • Flexibility • Transfer speed • Time-to-data • Capacity • Compatibility • Cost CIT 470: Advanced Network and System Administration

15. Reliability MTBF • Remember that drives fail faster than MTBF claims early and late in lifespan. • Talk to people who have used system. Duty cycle • Expected usage per day. • 40% duty cycle = 10 hours per day • MTBF based on listed duty cycle. Hard drives are more reliable than tape. • Closed system protects from contaminants. CIT 470: Advanced Network and System Administration

16. Flexibility Flexibility means • Able to respond to different data rates. • Can be used in different ways. Tapes aren’t very flexible • Typically require a standard data rate. • Slower/faster rates result in I/O errors. • Can only read/write sequentially. Disks are very flexibile • Random access medium. • Can change data rates rapidly. • Combine with RAID or LVM for capacity or perf. • Virtual tape software allows disks to appear as tape. CIT 470: Advanced Network and System Administration

17. Transfer Speed Compare native sustained transfer rates • Transfer rates often assume compression • Burst or synchronous rates are temporary, best-case scenarios. Disks are much faster than tape. CIT 470: Advanced Network and System Administration

18. Time to Data Time to Data • How long to load a volume + • Seek to appropriate place on volume + • Start reading data. Load time can include • Time to manually find and load tape. • Time for tape robot to locate and load tape. Most restores are for a few files. • User deletions. • Time to Data matters more than Transfer Rate CIT 470: Advanced Network and System Administration

19. Capacity Want one backup to fit on single volume. • Easy to manage than backups across multiple volumes. • Tape capacity grows slower than disk capacity. Cost • Lots of small backups to a single volume. • Reduces number of volumes to purchase + store High capacity is faster • Fewer volume switches when backing up. CIT 470: Advanced Network and System Administration

20. Compatibility Want to be able to restore everything. • If new system incompatible with old, must transfer old backups to new format. Use a single format for easy of management. CIT 470: Advanced Network and System Administration

21. Backup Media Flash Memory Very expensive, small media, personal use only. Super floppies ZIP 750MB, small capacity, high $/GB media. Optical CD-R cheap drives, slow + small (650MB). DVD-R cheap drives, slow + small (4.7GB). Ultra Density Optical (UDO) expensive but larger (60GB). Hard disk Large capacity (1TB), bulky, low $/GB media. Tapes Large capacity (800GB), low $/GB media; expensive drives. CIT 470: Advanced Network and System Administration

22. D2D2T Backup data first to disk • Take advantage of fast disk speeds. • Complete backups within nightly window. Copy backup disks to tape • Copy backup data from disks to tape. • Disks aren’t in production, so this can be slow. Reuse backup disks each night. CIT 470: Advanced Network and System Administration

23. Tape Formats CIT 470: Advanced Network and System Administration

24. Tape Appearance LTO and SDLT cartridges Full and half height 5.25” SCSI LTO drives CIT 470: Advanced Network and System Administration

25. Common Tape Features Form Factor 5.25” FH SCSI drives Media are ~1/2” wide tape stored in cartridges. Hardware compression Usually cited as 2:1, some cite higher. Depends heavily on nature of data stored. Future Roadmaps Plans to double capacity in next few years. CIT 470: Advanced Network and System Administration

26. Hardware vs Software Compression Software Compression • Compress data via software before writing. • Can use high compression tools like 7zip, bzip2. • Lowers amount of data xfer across network. • Higher CPU usage. Hardware Compression • Compress using specialized hardware on tape. • Does not require additional CPU usage. • Increases throughput of drive. CIT 470: Advanced Network and System Administration

27. Tape Autochangers Stackers Works sequentially through a stack of tapes. Autoloader / Jukebox Provides random access to set of tapes. Library / Silo Multiple drives w/ random access to set of tapes. May incorporate bar code reader, ethernet, etc. CIT 470: Advanced Network and System Administration

28. Backup Software OS Provided (backup individual systems) cpio, dump, tar, ntbackup Open source (backup servers) AMANDA Bacula Commercial (backup servers) Tivoli Storage Manager (IBM) Veritas Storage Manager CIT 470: Advanced Network and System Administration

29. Provided Backup Software UNIX/Linux • tar • cpio • dump • dd Windows • ntbackup • Windows Restore MacOS • ditto CIT 470: Advanced Network and System Administration

30. Windows Restore Points Auto backup of registry + critical files • Must have System Restore enabled. • Done automatically + can create manual too. • Useful when software install has corrupted. Recovering Windows with a Restore Point • Boot into safe mode. • Select “Restore my computer to an earlier time” • Choose date from list of restore point dates. • If that doesn’t work, reboot, try an older one. CIT 470: Advanced Network and System Administration

31. Backup Software CIT 470: Advanced Network and System Administration

32. Open Source Backup Tools Amanda • Single master backup server. • Simple, fast, uses native backup tools. Bacula • Client/server backup system. • More advanced features but slower than Amanda. BackupPC • Web-based so works with any client OS. • Backs up PCs and laptops to disk on server. CIT 470: Advanced Network and System Administration

33. Snapshots Virtual read-only copy of filesystem • Snapshot has same contents that filesystem has when snapshot was made. • Snapshot uses pointers to data and copy-on-write to avoid making a copy of entire fs. • Snapshots require ~1% of fs size, depending on updates. CIT 470: Advanced Network and System Administration

34. Snapshot Applications Quick restore times • Snapshots take seconds to create, restore from. • Won’t help you if disk or other hardware fails. • Snapshot lifetime could be short as 1m or as long as a few days for this purpose. Staging for backups • Snapshot filesystem before starting backup. • Files on snapshot do not change during backup. • Snapshot lifetime is how long it takes to backup. CIT 470: Advanced Network and System Administration

35. LVM Snapshots LVM can create snapshots of logical volumes. lvcreate -L500M -s -n backup /dev/db/db1 • Creates snapshot (-s) volume named backup • Volume is snapshot of /dev/db/db1 LV • Can make 500M of changes. What happens if >500M of changes? • LV can’t receive copies of old data if changes are made to original logical volume. • Ensure 500M is more than enough space for changes made during lifetime of snapshot. CIT 470: Advanced Network and System Administration

36. Continuous Data Protection Copy every file change to backup server. • Stores changes in a log like RCS or a database. • Can restore to any point of time. Near-CDP • Snapshots + replication. • Does not provide a log, so can only restore to saved snapshots, not to any change like CDP. CIT 470: Advanced Network and System Administration

37. Backing up Virtual Machines Back up VMs as physical machines • Connect VM to your standard backup system. • Have to configure backups for each VM. Back up VM files • Can back up all VMs on host at once. • VM files are constantly changing, so either • Suspend VM • Snapshot filesystem with VM files CIT 470: Advanced Network and System Administration

38. Automation The key to efficiency and reliability. Use cron instead of manually backing up. Single tapes require manual media change. Tape libraries automate this process. Other automated tasks Monitoring (up/down, disk space, security) Logs (rotation, monitoring) File distribution CIT 470: Advanced Network and System Administration

39. Cron Performs tasks at scheduled times. Crontab files specify schedule of tasks root: /etc/crontab users: /var/spool/cron/crontabs/* Cron may log activities and errors. Timing limitations: Runs tasks (if any) every minute. Does not perform scheduled tasks if system down. May or may not perform tasks on DST transition. CIT 470: Advanced Network and System Administration

40. Crontab Format minute hour day month weekday user command Examples 30 4 * * 0 root yum –y update 3 * * * * root (cd /var/www; make) 20 1 * * * root /usr/local/rot-logs CIT 470: Advanced Network and System Administration

41. Managing Automated Tasks Divide by time: Hourly, daily, weekly, monthly tasks Crontab uses run-parts meta-script: 17 * * * * root run-parts --report /etc/cron.hourly 25 6 * * * root run-parts --report /etc/cron.daily 47 6 * * 7 root run-parts --report /etc/cron.weekly 52 6 1 * * root run-parts --report /etc/cron.monthly Add crons by placing script in time directory. Add random delay if all hosts share same crontab. CIT 470: Advanced Network and System Administration

42. Backup Security Tape security Tapes contain all of your important data. Data isn’t secure unless tapes are secure. Solutions: tape vault, encrypted tapes. Backup server security Has read access to all important data. If backup server isn’t secure, data isn’t secure. Solutions: integrity checking, least privilege Restore process Who can request files to be restored? Where will restored file be placed? What will its ACL be? CIT 470: Advanced Network and System Administration

43. References • AIT, AIT Forum, http://www.aittape.com/index.html, 2006. • Lynne Avery, “A Brief History of Tape,” Exabyte white paper, http://www.kontron.com/techlib/whitepapers/A_brief_history_of_tape.pdf, 2000. • Aeleen Frisch, Essential System Administration, 3rd edition, O’Reilly, 2002. • LTO, http://www.ltotechnology.com/newsite/index.html, 2006. • Peter McGowan (ed), Quantum DLTape Handbook, http://downloads.quantum.com/sdlt320/handbook.pdf, 2001. • Evi Nemeth et al, UNIX System Administration Handbook, 3rd edition, Prentice Hall, 2001. • Shelley Powers et. al., UNIX Power Tools, 3rd edition, O’Reilly, 2002. • W. Curtis Preston, Backup & Recovery, O’Reilly, 2007. • Quantum, “Tape Storage Automation,” http://www.dlt.com/storage/whitepapers/quantum/dlt/Tape%20Storage%20Automation.pdf • “The Tao of Backup,” http://www.taobackup.com/ • Wikipedia Contributors, “Magnetic Tape,” http://en.wikipedia.org/wiki/Magnetic_tape, 2005. • Elizabeth Zwicky, “Torture Testing Backup and Archive Programs,” ftp://ftp.berlios.de/pub/star/testscripts/zwicky/testdump.doc.html, 1991. CIT 470: Advanced Network and System Administration