1 / 68

ATO Overview

ATO Overview. Adam Kirkpatrick Created: July 2011 Last Update: July 11, 2011 kirkpatrick_adam@emc.com. ATO is a utility providing tape-out capabilities from an Avamar system. It relies on a re-hydrated approach recovering data to a disk staging area.

dysis
Download Presentation

ATO Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ATO Overview Adam Kirkpatrick Created: July 2011 Last Update: July 11, 2011 kirkpatrick_adam@emc.com

  2. ATO is a utility providing tape-out capabilities from an Avamar system. It relies on a re-hydrated approach recovering data to a disk staging area. The staging area is subsequently backed up to tape using a standard tape Backup application. The tape out process involves three steps; client/backup selection confirmation view to see which backups were selected staging selected backup data and perform a tape backup ATO Batch Manager fully automates these three phases. ATOAvamar Tape Out

  3. ATO – Data Flow Source or Target Single or Multi Node Systems Networker De-Dupe Nodes ATO Environment File Configuration Criteria ATO Client Configuration File Criteria

  4. ATO - Features • Single Point of Management & Control • MENU Driven User Interface • Multiple Concurrent Tape Out Sessions (1/Staging-Server) • Failure Checkpoint Recovery • Non-Inc & Incremental Tape Out • Staging Recoveries Visible in Activity Monitor • File Level Recovery from Tape to Original or Alternate Client • Comprehensive Backup Selection Filters • Structured Event and Batch Logs • Audit Mechanism Tracks Critical Path TO Components • Automated Generation of Required Tape Script • Email Notifcations

  5. ATO - Support Matrix • Staging Platforms • Windows, Red Hat, SCO, FreeBSD, Solaris, AIX, HPUX • Avamar Plug-In • Lotus Notes, SQL, Exchange 2003, NDMP, VMimage, (Oracle) • Tape Applications • Networker, NetBackup, Backup Exec, Arcserv, HP Data Protector, TSM, Commvault • Avamar Node Types • Single & GRID, Source or Target, AVE, Networker De-Dup

  6. User Interface • Fully Interactive • All Functionality Available • Color Coded Displays • Access to Logs • Online Help • Automatic Upgrade Process ATO Rocks!

  7. 3-Step Operational Cycle

  8. Selection Paramaters • Client Selection • By client group–gid group-id • By individual client within a group–gid group-id –client client-id • Client must be in a group • Plug-In Types • FS (default) • SQL–sql • Exchange–exchdb or -exchvss • NDMP –ndmp • Oracle –oracle • Sharepoint –shpt • Lotus Notes –lotus • Backup Reduction Filters • First–firstlast–lastor–first_For–last_F • Day of the week–week sunor day instance of the month –week sun_1 • Avamarpolicy group name –gname policy-group-name • Retention tag type –rtype daily | weekly | monthly | yearly • Backup-ID number–buid “#’s • Backup Type –but mod | cod | nah ... “

  9. Selection Date Range • Default Search Range • 1st day of current month to your current date • Expand default range: -nday +# expand start date going back in time the specified # of days • Relative Search Range • By Month: -rdate #where # is a relative month count going back in time from your current month. I.E. –rdate 1 will go back one month in time • By Day: -rday # where# is a relative date of the month • By Day Range: -nday # where # is the number of days back in time from your current date • Fixed Date Range • Start date: -sdate yyyy-mm-dd • End date:-edate yyyy-mm-dd • Refer to Flds-2 & 3 in client configuration file, used for a more permanent override • Day Count: -nday # adjust # to number of days from your current date

  10. Sample Select Process • –lastfilter used limiting selected backups to most recent only • clients scanned are those defined in group linux only • final selected backup count is 1 with 3 being excluded • used default date range 2011-07-01 to 2011-07-05 • used ATO environment file 1 • Note: By default the selection process acts on scheduled backups only • refer to –butype option to select non-scheduled backups

  11. Sample View Process • Selected Backup View • backup metrics per client • total combined backup size • combined size allocated per staging FS or folder location • To View: ato-vorato–view

  12. Tape-Out Process • A Tape-Out Session Initiated with –tapeout Includes 2 Steps • Staging or redirected recovery of the selected backups to a staging location • Tape-Backup refers to initiating the tape backup automaticaly of the staged data • Step-1 - Staging • destination determined by Environment & Client configuration files • concurrent staging supported only with multiple staging servers available • perform stage only using –s option (no tape backup will be initiated) • Incremental using –inc provides significant performance gains • Inclusion and exclusion filters for folders or files • –data “C:/Program Files | My Documents” • –xdata “C:/Program Files | My Documents” • Step-2 - Tape Backup • Perfromed by default or optionally using –t (tapeonly no stgaing will be performed) • appropriate tape script generated and initiated automatically • user defined scripts can be specified • tape backup policy referenced is assumed to exist within tape application • tape incremental backups can be leveraged only when incremental staging is used

  13. Sample Tape-Out Process

  14. ATO Environment File

  15. Environment Files • Used to define physical infastructure of Tape Out environment • x1 Windows & x1 UNIX staging server and x1 tape backup server name per environment file • Tape backup application • Path to binaries of Avamar & Tape backup agents • Event log, client configuration & temp file locations, pseudo client • Various control variables - key variables established automatically • Max of 20 environment files supported • each environment functions independently • ATO sessions can use any defined environment file • lock mechanism to prevent concurrent use of a given environment

  16. Environment File Contents • Site specific parameters will require modification • tape backup server parameters • Staging server paramaters • email notification address list • Operational parameters • adjust as required • refer to documentation for details • ATO Control Files • Caution: modification of control file names not recommended • TMP_PATH, CFG & LOG files, Pseudo Client name etc.

  17. Establishing & Accessing Environment Files • Establishing Environment Files • ato [#] –env where # represents the desired environment number, default of 1 is assumed • creates a environment config file /usr/local/avamar/etc/atoenv.cfg[#] • establishes temp file location /tmp/atocfg[#] • establishes event log file /usr/local/avamar/var/atoevent.log[#] • references client configuration file /usr/local/avamar/etc/atoclient.cfg • creates the Avamar pseudo client /clients/tapeoutato[#] • establish various control variables with default values • file names remain consistent with the exception of an appended numeric value • I.E. /usr/local/avamar/etc/atoenv,cfg2, /tmp/atocfg2, /usr/local/avamar/var/atoevent.log2 etc.. • Accessing Environment Files • Displayato [#] -env - displays environment file contents • Update: ato [#] -env update -opens a vi session using appropriate file name • Verify: ato [#] –env parse - verifies file contents and report on any anomalies • Note: environment # when required must be the first argument specified to ATO

  18. ATO Client Configuration File

  19. Client Configuration File • CSV file with 18 fields per client entry • client names and how they are grouped • destination staging server type UNIX or Windows • destination staging FS or folder, can vary between clients within a group • References the tape policy name to use • Determines whether to use an auto generated or user defined tape script • various backup filters applied on a per client basis • Enables Simplified Tape-Out Operations • client groups enable more precise selection and allows segmented work flow • better to define several smaller groups as opposed to one large one • clients from different domains are permitted

  20. Client Configuration File View • Establishing Client Configuration • CSV formatted file • base file established automatically • a single client configuration file is shared between all environment #’s • update interactively or manually • automatic Avamar domain and client discovery process ensures accuracy • Primarily a one time task at install time with infrequent updates afterwards

  21. Client Configuration File Contents • Viewing Configuration File Contents • To Display: ato –cfg (all groups) or ato –cfg <grp-id> (specific group only) • Content Verification: ato [#] –cfg parse – perform checks of file contents • Color Codes: Blue=predefined sample lines Green=user comments Cyan=important field highlight Red=field separator Configuration File Display

  22. Client Configuration File Usage • Key ATO Process’s Rely on This File • All tape out clients must first be defined in this file • Client Groups • group names cannot contain spaces or special characters • a client can be defined once per group but may exist in any number of groups • all clients in a group must share a common tape policy name & staging server type • An Organized Group Naming Convention is Beneficial • I.E. A marketing dept has 100 clients where 25 clients is the desired workload per session. Group names might be mkt1,mkt2, mkt3, mkt4 maintaining a logical connection between them while facilitating easy work load segmentation and automation from CRON or 3rd party Scheduler using ATO’s Batch Manager described later • Several Options Available on Both Command Line & in Client File • If specified on command line will pertain to all clients in the group • If client requirments vary within a given group, define these in the client file • date ranges, incdel include/exclude folders, Avamar policy group name, retention tag and destination path

  23. Client Configuration File Update-1 Selection Phase • Manual Update:ato –cfg update • Update Method-1 (bulk – useful at install time only) • Automated Update: ato –cfg add or ato –cfg add_v (initiate discovery & update process) • 1st blue highlighted line contains a generic potential line entry • DOMAINX & CLIENTX are keywords replaced automatically, do not modify these • green highlighted line shows a user modified line with fields Fld-1=training, Fld-10=Drive T being changed • first non configured client displayed was optionally rejected • second non configured client displayed and optionally accepted & added to configuration file • subsequent non configured clients are displayed and added or rejected as required • existing client entries requiring modification can be edited manually or using interactive update (Method-2) ► ► ► ► Method-1 Update Session ►

  24. Configuration Manager Domain & Group View Selection Phase • Update Method-2 (interactive & preferred method) • Via Interactive Update: ato –cfg manager • Clients viewed by Avamar Domain or ATO Client Group • Drill down into a Avamar domain or ATO group provides options to • add, delete, disable, enable, modify&view individual client entries Method-1 Update Session

  25. Configuration Manager Cont... • Interactive Method-2 cont.... • Client status view realtive to ATO showing all available clients • Select client by number to add, disable, enable, modify, remove & view individual client entries

  26. Client Enable/Disable • Interactive Method-2 cont.. • Sample process to disable/enable a configured client to ATO • Client status display, Blue=Not configured, Green=Enabled, Red=Disabled to ATO enter “d” to disable a client enter the client-# to disable confirm the displayed line is okay to disable client#2 disabled confirmed by its color change

  27. Client Entry Update-2 • Method-2 cont.. • Sample process to modify an existing client entry • Process to add a new client (not-shown) is similar enter “m” to modify a client entry enter the client-# to modify enter the group name for client entry to be modified verification screen can be looped through as often as necessary modifying field contents as required. When ready to accept the change enter “c” to commit the change

  28. Locate & Track Data on Tape System • All Staged Data Belongs to the Staging Server(s) Clients Used • tape backup effectively owned by staging server client • Data Staged to Structured Directory • <user-defined>/<ATO-defined>/Orig-Client-Name/<client-data> • Tape Solution Reports Relative to Staging Client Name Only • Combine Tape System Reports with ATO Audit Report • tape media involved • relative folder location where to find TO client in staging client backup • staging and tape backup dates • staging server name used • tape backup server used

  29. Audit Log • Used to Track Critical Path Component Name Changes Impacting Recovery Procedure From Tape • Critical Path Components Tracked • Tape Backup Server name • Tape Policy Name • Staging Server Name • Staging Server Destination Folder • Audit Process Is Automatic Requires No User Intervention • Critical path component changes generates a environment audit record • Audit records readily available interactively basedd on client name from Configuration Manager option T=Audit • Used to determine where a given clients data resides on tape system

  30. Audit Record Updates • Environment Change Detected in Critical Path • will occur relatively infrequently over time • displayed in audit summary or detail report • Non Critical Path records • audit record established every time a client is staged • displayed in audit detail report • Audit Log File • any filters or script can be run against audit log contents to format your own custom or alternative report formats • available through ATO interface for display purposes

  31. Sample Audit Summary Report • Summary Report Displays Critical Path Change Records Only initiate audit report initiate audit report initiate audit report initiate audit report select client for report select client group selected report type defaults to summary or enter D = detail

  32. Sample Audit Detail Report • Detail Report Displays Critical and Non Critical Audit Records selected report d=detail

  33. ATOTape Backup ScriptStaging Server SpecsDestination Path

  34. ATO - Tape Scripts • Automated Tape Script Generation • criteria taken from environment and client configuration files to generate a suitable script autotapeout.bat or autotapeout.sh located in the environments temp directory • transferred to staging server <AvamarHomePath>/etc/scriptsandexecuted as a pre script • supports incremental and non-incremental • non-incremental script will remove staged data after a confirmed successful tape backup • Supported tape applications • Arcserv – Brightstore, HPDP,Networker, Backupexec NetBackup, TSM & Commvault • Networker support for client alias name, I.E. tape out data owned by original client name (disabled) • any tape solution with a suitable CLI could be accommodated • User Defined Scripts • must contain necessary logic to enable ATO to capture its completion status • refer to ATO doc for example or use an auto generated script as a reference • must be kept on the Utility node, suggested location /atoadmin/userscripts • Tape Backup Status • tape status is logged on staging server in <Avamar-Home-Path>/etc/scripts/autotapeout.stat • The contents of this log are recovered back to ATO which parses and displays results • detailed failure information logged to event and batch logs • any non-zero tape RC is considered a failure

  35. Staging Server & Destination Path Specs • Staging Server Specs • can be physical or virtual • Must support required Avamar and Tape App file system agents • Decent CPU capacity with the equivelant of at leat 6-8GHz • Structured Non-Incremental Destination Path: • <user-defined-path>/BYDATE/<client-name>/<date-time-buid#>/<backup-data> • Sturctured Incremental Destination Path: • <user-defined-path>/INCREMENTAL/<client-name>/<backup-data> 1. Path definition shown in RED is not user modifiable 2. Optional path suffix inserted prior to </backup-data> using the -path option or from Fld-16 in client configuration file

  36. Sample Networker Save Set Definition • Sample Networker client resource • BYDATE • INCREMENTAL • Define multiple client resources as required • All staged clients to a given location will be included in the Tape Backup • User portion of staging path defined in client cfg. file • Each client resource may have its own media pool, group assignment and schedule etc.

  37. Networker Recover View Of Tape-Out Data • Recovery View on Staging Server of Incremental TO • Recovery View on Staging Server of non-Incremental TO referred to as BYDATE

  38. ATO Incremental Advantage

  39. Incremental Tape-Out • Incremental Concept • incremental staging is relative to previously staged data, 1st stage is always a full • requires staged data remain on the staging server between ATO sessions • refer to –incdel option for a method to resynchronize staged data with backup client • beneficial for FS & NDMP not for SQL, Exchange, Oracle or Lotus Notes • total disk space required equivalent to size of one copy of source data involved • staging disk can be any disk type, Arrays, JBOD’s, USB stand alone etc. • disk I/O performance less critical when using incremental • makes it feasible to leverage an incremental tape-backup • Performance Gains • typical effective throughput gains versus non-incremental ~5 to 8 times • expect ~600-900 GB’s/Hr forstagingdependent on % incremental change • If using incremental tape policy • expect equivalent tape backup throughput gains • expect reductions in tape media usage of ~70-80% • use of concurrent staging servers will improve overall aggregate throughput • How to Enable Incremental • Add to the -tapeout action, –inc option

  40. Incremental AdvantageExample-1 Environment: single node Initial Stage 4.7 GB’s 2.45 min’s Initial Tape Backup 4.7GB’s 4.5 min’s

  41. Incremental AdvantageExample-1 Environment: single node Post Initial Stage 4.7 GB’s 15 sec’s Post Initial Tape Backup 4.7 GB’s 1 min

  42. Incremental Advantage Example-2 Environment: single node Initial Stage 10.5 GB’s 11.45 min’s Session continued... Initial Tape Backup 10.5 GB’s 8 min’s

  43. Incremental AdvantageExample-2 Environment: single node Post Initial Stage 10.5 GB’s 30 sec’s Post Initial Tape Backup 10.5 GB’s 1.5 min’s

  44. Incremental Advantage Single Node Incremental Comparison Results • Factors Influencing Test Results • Networker Server is using Windows • Networker configured to use a DL3D VTL • no changed files were involved

  45. Incremental Advantage Example-3 Environment: multi node(4) Initial Stage 44.5 GB’s 12 min’s Initial Tape Backup 44.5 GB’s 23 min’s

  46. Incremental AdvantageExample-3 Environment: multi node(4) Post Initial Stage 44.5 GB’s 45 sec’s Post Initial Tape Backup 44.5 GB’s 30 sec’s

  47. ATO – Incremental Advantage Multi Node Incremental Comparison Results • Factors Impacting Test Results • Networker Server & Staging server were the same server, an Avamar spare node • Networker configured using adv_file type device B2D • no changed files were involved • Staging configured to use all-nodes, provided ~10-15% increase in throughput

  48. ATO Event Log

  49. ATO - Event Log • All –select and –tapeout actions logged • Consolidates informational & Error Messages • Readily accessible to user for success / failure confirmation or problem diagnosis • To access Event Log: ato –l or ato -log • most recent event displayed initially • specify default browse direction by entering P=Previous or N=Next then press enter to browse all events in the direction specified • to jump to a specific event#, enter desired event number & press enter • to search event log enter “s” then “h” for search editor syntax, similar to vi commands • Event messages are color coded • informational messages displayed in your default terminal color • errors messages displayed in red • info returned from tape application displayed in white • Each Environment# maintains its own Event Log • Email alerts contain corresponding Event Log details

  50. optional debug flags used to provide additional command trace information Avamar recovery log name located on SS and available within Activity Monitor staging process completed okay tape backup command syntax issued by auto generated tape script details returned from tape backup command issued by tape script ATO - Event Log Sample – Successful Stage & Tape Status

More Related