SPO3292

1. VeeamBackup & Replication:Tips and Tricks SPO3292 Anton GostevVeeam Software@Gostev Doug HazelmanVeeam Software@VMDoug #vmworldsponsor

2. Quick Overview of v6 Architecture • Backup serversBackup proxy serversBackup repositories • “Automated everything” • Intelligent load balancing • Centralized management via Enterprise Manager

4. Scaling your backups 3 simple rules

5. Scaling your backups • Keep the management server happy • Disable default proxy (requires 6.1 or later) • Allocate enough RAM for job manager processes • Keep concurrent (running) jobs under 100 per management server • Backup proxy servers: the more, the better? • No! Too much load on storage and network • Use max concurrent tasks limit on proxies or repositories • Watch for “job timed out waiting for resources” messages • Backup repository considerations • Be careful with the reversed incremental backup mode! • Limit concurrent jobs on backup repositories to a reasonable amount • Use ingest rate throttling for cross-SAN backups

6. A word on backup repositories • Don’t underestimate the importance of performance! • By far, the most commonly reported bottleneck • What makes the best backup repositories? • Windows or Linux server (can be same as backup proxy server) • Local storage, DAS or SAN mounted for physical server • pRDM disk (vSphere 5+), or iSCSI LUN connected via in-guest iSCSI for virtual server • Sub-optimal backup storage • NAS or network share • VMDK on VMFS (size and recoverability considerations) • RAID level • If you can afford it, use RAID10 (again, performance)

7. Processing Modes All you need to know

8. Direct SAN Access: The good • Fastest processing mode • Least impact on production • Backup processing is fully offloaded to dedicated backup proxy servers • Backup traffic is isolated to the storage network (aka LAN-free) • Does not impact consolidation ratio, so cheapest too

9. Direct SAN Access: The bad • Supports block storage only • FC (fibre channel): physical backup proxy server only! • iSCSI: physical and virtual backup proxy servers both supported • Physical backup proxy server requirement for FC SAN • Might not go along well with your virtualization project • Consider repurposing older servers • Might be hard for beginners to setup • See Veeam Forums FAQ for step-by-step guide • Manual datastore mapping might be required • For certain SANs, B&R may not able to detect proxy connectivity

10. Direct SAN Access: The ugly • What’s the worst that could happen? • Windows re-signaturing your VMFS LUNs! • vSphere will no longer recognize datastores • Don’t panic, VMware Support should be able to fix • Three easy ways to get into trouble • Windows Explorer automounting new volumes (but not with Veeam) • Clicking Disk Management snap-in popup without reading • Giving Local Administrator rights to random people

11. Direct SAN Access: The safe way • Present VMFS LUNs to backup proxy server as read-only • Most SANs support it these days—chase your vendor if yours does not • Disable automount on your backup proxy servers • Do it the right way: use SANPolicy Windows setting! • Veeam backup proxy server setup does this automatically for you • Disable Disk Management snap-in with Group Policy User Configuration > Administrative TemplatesWindow Components > Microsoft Management Console >Restricted/Permitted snap-ins > Disk Management • Keep Local Administrator rights on backup proxy servers to yourself • Cannot really do this for default proxy due to FLR requirement • Another reason to use dedicated backup proxy server!

12. Direct SAN Access: Tips & tricks • Got a fast SAN? Get a modern backup proxy server! • Multi-core CPU (compression) and fast RAM (inline deduplication) • Update firmware and drivers across the board • Disabling MPIO may increase performance • iSCSI SAN? Tweak TCP/IP on backup proxy netshinterface tcp set global autotuning level = disable • Increase read-ahead buffer • Default is 4MB (optimal setting for most SANs) • To change, create the new value in bytes: VddkPreReadBufferSize (DWORD)

13. Hot Add: The good • Easy to setup—very little planning involved • Any Windows VM can be made a Hot Add backup proxy • Fast data transfers with any storage • Direct storage access (albeit through ESXi storage stack) • Supports all types of storage (including NFS) • Shared storage: at least 1 backup proxy server per vSphere cluster • Local storage or DAS: at least 1 backup proxy server per host • Use your existing Windows VMs (save on licensing) • Data processing engine process runs with lower priority (6.1) • Further CPU usage reduction in 6.5 • Allows for 100% virtual deployment

14. Hot Add: The bad • Not as mature as other modes • Affects your consolidation ratio • Backup proxy servers take host resources • Ultimately means more ESXi hosts, and more VMware licenses • Hot Add process itself is slooow • Can take up to 1–2 minutes to complete for each VM—adds up quickly! • Hot Add as a vSphere feature has a number of limitations • Good news—many are being removed as VDDK matures • See FAQ on Veeamforums FAQ for the complete list

15. Hot Add: The ugly • Snapshot removal problems due to locks • Veeam B&R: multiple hooks in place to work around • CBT must be disabled on backup proxy VM • Prevents stun on Hot Add due to CBT initialization • NFS-specific issue • Extended VM stun on hot remove in some scenarios

16. Hot Add: Tips & tricks • Add extra virtual SCSI controller to backup proxy server • A single SCSI controller can have a maximum of 16 disks attached • Concurrent jobs on the same backup proxy server can result in more! • Keep vSphere and Veeam up to date • Single block size in VMFS5 removes the most common hot add issue • Latest Veeam Backup & Replication will have latest and greatest VDDK version • Try increasing read-ahead buffer • Seems to really help with certain NFS storage • Avoid cloning backup proxy VM • For example, to provision additional backup proxies

17. Network (NBD): The good • Easy to setup—in fact, no setup is required • Any existing server (physical or virtual) would do • Supports all types of storage, including NFS • Server placement does not matter (unlike with Hot Add) • Very quick to initialize data transfer • Can be quite fast—with 10Gb Ethernet

18. Network (NBD): The ugly • Painfully slow performance on 1Gb Ethernet • Average speed reported is 10-20 MB/s • Leverages ESXi management interface

19. Network (NBD): Tips & tricks 1 Gb Ethernet • Use for sites with low change rate • Works faster than other processing modes in such conditions • Keep at least one Hot Add backup proxy server around • Full VM and virtual disk restores take forever over NBD • Keep in mind intelligent load balancing algorithms • Network backup proxy servers have lowest priority! • Upgrade to 6.1 or later • Improved network proxy location awareness

20. One last thing • This hack significantly reduces supportability! • Cut up to 5 minutes of processing time per VM by disabling VDDK logging • Apply in stable environments only! • Create the new value and set to 1: DisableVDDKNetworkOutput(DWORD)

21. Deduplicating Storage Yes, you can afford it!

22. Deduplicating storage: The good • What gives? Global dedupe! • Deduplication across backup files from different jobs • Perfect for long-term backup archival • Top hardware appliances among Veeam users • EMC DataDomain • ExaGrid • HP StoreOnce • Top software appliances among Veeam users • ZFS-based appliances • StarWind • Windows Server 2012 dedupe is awesome

23. Deduplicating storage: The bad • Hardware appliances are expensive • Although they do provide excellent dedupe ratio • Software appliances are resource hogs • Both performance and dedupe ratio are sub-par, too • Windows Server 2012 dedupe is awesome • Included free of charge—start using it today! • Provides very decent dedupe ratio

24. Deduplicating storage: The ugly • Random access performance is lacking • A problem for all solutions featuring inline deduplication • Typically insufficient out-of-the-box for large-scale vPower usage • Exception: post-process deduplication • ExaGrid • Raw disk landing zone (full-speed vPower from recent backups) • Veeam-specific logic further optimizes performance • Windows Server 2012 • Backups “land” on raw storage at full speed • Only old backup files are deduplicated—great for vPower • Decent speed even off already deduped backups

25. Deduplicating storage: Tips & tricks • Already own storage with inline deduplication? • Inline data “rehydration” process is what makes vPower slow • Reduce the block size in Advanced job settings • WAN (256KB) and LAN (512KB) • Reduced block size might impact backup performance • Use Linux-based backup repository • Large client cache, or even caching client file system (FS-cache), can significantly improve vPower performance

26. Deduplicating storage: Tips & tricks (continued) • Keep Veeamdedupe on • Use incremental backup mode • Choice of synthetic or active fulls depending on actual storage • For best dedupe ratio on device side… • Disable compression (significantly increases amount of data transferred from backup proxy server to backup repository over network) • For best backup performance and smallest window • Keep compression at default • If you like to avoid extremes… • Set compression to Low (dedupe-friendly)

27. Deduplicating storage: Tips & tricks (continued) Got more than one deduplicating storage device? • Use internal replication to sync backups offsite! • Extremely traffic-efficient approach • Many customers use and report great success! • Keep the backups imported for easy DR • To automate repository refresh in DR site, use: Get-VBRBackupRepository -Name "DR_Repository" | Sync-VBRBackupRepository

28. WAN Accelerators Your WAN on steroids

29. WAN accelerators: The good • Two types of WAN accelerators • Caching WAN accelerators provide significant bandwidth savings with Veeam replication, but are typically quite expensive • Transport layer WAN optimizers are unlikely to offer significant bandwidth savings with Veeam replication, and are usually cheap • Both improve reliability of TCP • Long distance wireless or satellite links • IPsec rekey operations on a VPN tunnel • WAN links with high jitter, packet loss or occasional drops • Both allow long-running jobs to finish more consistently • For example, initial replication over network • Both improve WAN utilization for most workloads

30. WAN accelerators: What’s hot? • Top caching WAN accelerators among Veeam users • Cisco WAAS • Riverbed • SilverPeak • Top WAN optimizers among Veeam users • Hyper-IP

31. WAN optimizers: Tips & tricks • Veeam Backup & Replication leaves little room for bandwidth reduction by WAN optimizers • Built-in WAN optimizations in v6: • Multiple TCP streams to maximize throughput • Network traffic compression • Consider using on unreliable networks, but don’t expect them to add you extra bandwidth • Update to Backup & Replication 6.1 Patch 1 before evaluating • Veeam Backup & Replication not using all available bandwidth? • Increase the amount of TCP streams (default is 5) DownloadStreamsNumber (DWORD)

32. Caching WAN accelerators: Tips & tricks • Multiple TCP streams can cause issues! • Disable multiple streams in B&R traffic throttling settings • Disable network traffic compression in Veeam • Low (dedupe-friendly) compression level might be a better option • If required, have network admins configure bypass on Veeam backup proxy servers to avoid polluting caches

33. Veeam Backup & Replication 6.5 We never stand still

34. What’s coming in 6.5 • Veeam Explorer for Microsoft Exchange • Veeam Explorer for SAN Snapshots • VMware vSphere 5.1 support • Windows Server 2012 support

35. Veeam Explorer for Microsoft Exchange • Visibility into Exchange VM backups • Immediate: No need to provision storage, restore the VM or restore the mailbox store • Agentless • Requires no special backups or metadata collection—even works with existing Veeam backups (and SAN snapshots) • Free! • Included in all versions of Veeam Backup & Replication 6.5,including Free Edition • Eliminates need for expensive standalone tools licensed per-mailbox • Currently available in “exclusive beta” Just restored a 145 GB #MSExchange Public Folder database in 2 min. using @veeam. Then restored a single item all under 10 min. SWEET! The new #veeam explorer for exchange looks veeamy. That's right, I just made up a new word (it means awesomesauce) Even the *beta* of @veeamExchange Explorer works a treat. Saved literally, hours of work.. and saved my bacon. #recommend Really excited with the new @veeam Explorer beta for #Microsoft Exchange VM backups - we have lots of interested customers ready for this!

36. Veeam Explorer for Microsoft Exchange (continued) • Capabilities • Browse: familiar Explorer-type interface • Search: familiar Outlook-like Find, including Advanced Find • Export: export to PST file, MSG file or attachment • Uses cases • E-discovery • Item-level restore: export and send to affected user • Mailbox archive • Supports Exchange Server 2010

37. Veeam Explorer for SAN Snapshots • Veeam restores from SAN snapshots • Supports tiered data protection strategy • Perform all restores through familiar, easy-to-use Veeam interface • Supports HP StoreVirtual VSA and HP LeftHand • SAN snapshots + Veeam restore = Best RPOs and RTOs for operational recovery

38. Veeam Explorer for SAN Snapshots (continued) • Fast: recover entire VM or individual items in < 2 minutes • Fully automated: clone & promote snapshot, present to vSphere, clean up • Restores directly from VM files on the SAN snapshot: no staging or intermediate restores required • Flexible • Specific VM • Individual guest files: Windows, Linux, et al • Individual Microsoft Exchange items • Free • Worry-free: automated process eliminates human errors and protects integrity of SAN snapshots and production LUNS • Agent-free: no agents to deploy on hosts or VMs • Literally free: included in all editions of Veeam Backup & Replication 6.5, including Free Edition

39. Questions? Comments?

40. Thanks for attending!

41. FILL OUTA SURVEY AT WWW.VMWORLD.COM/MOBILE COMPLETE THE SURVEY WITHIN ONE HOUR AFTER EACH SESSION AND YOU WILL BE ENTERED INTO A DRAW FOR A GIFT FROM THE VMWARE COMPANY STORE

42. VeeamBackup & Replication:Tips and Tricks SPO3292 Anton GostevVeeam Software@Gostev Doug HazelmanVeeam Software@VMDoug #vmworldsponsor