Deep Security 7 TOI – Performance

1. Deep Security 7 TOI – Performance Justin Foster/Kevin Boyce/Richard Abbott

2. Agenda for Deep Security 7.0 Performance • Manager Performance Factors • Agent Performance • ESX/Appliance Performance

3. Manager Performance Factors • # of managed Agents/Appliances • # of Events per Agent/Appliance • # of Manager nodes • Database Type/Location • Manager/Database Hardware • Configuration • # of concurrent Manager-Initiated Jobs • # of concurrent Agent-Initiated Jobs • Size of Security Configuration sent in Update • Heartbeat Frequency

4. # of Managed Agents/Appliances • Goal: 5,000 – 10,000 per Manager node. • Up to 5 Manager nodes connected to a single Database • Assumptions: • Depending on protection configuration, servers can generate many events • Server heartbeat typically set to 10 minute intervals

5. # of Events per Agent/Appliance • Manager has been tested with 100GB of events • Customer environments have surpassed 1TB of events, with challenges (when purging) • Scalability • Events are typically retained for 2 weeks • Events are summarized in Counters • Counters are aggregated into daily totals after 3 days • Counters are retained longer than Events for historical data • Events use clustered indexes on MSSQL

6. # of Events per Agent/Appliance (part 2) • Configuration has large impact on # of Events • Firewall: Turn off UDP Stateful Logging to eliminate broadcast Events (default is ON) • DPI: Large # events typically only from Web App Protection (tuning) or App Control, not IDS/IPS. • Integrity Monitoring: Set exclusions on common Integrity Monitoring changes (Example: self-modifying EXE files) • Log Inspection: Set clipping level for Log Inspection to medium-high (default = 7 High)

7. # of Manager nodes • Managers distribute all autonomous work among Managers. Agents round-robin Managers • Same model provides availability • Testing has confirmedefficient scaling to 3 nodes • Performance plateau as DBbecomes bottleneck (> 5 nodes)

8. Database Type/Location • Embedded database only recommended for Demos & Lab Eval • MSSQL 2008 or Oracle 10g recommended for production deployments • Dedicated database server recommended for performance • Ping time in system information shows DB latency. Typically should be <1ms, Problem if > 10ms

9. Manager/Database Hardware • Depending on the job type (Heartbeat, Update, Recommend, etc.) the bottleneck changes • Some job types are highly CPU intensive (like Recommendation) • Other job types are highly network or Agent/Appliance intensive • 1GB maximum heap used, however 64-bit build can be user tuned much higher • configuring a .vmoptions file next to the service (in linux's case, 'dsm_s' - on windows, it would be 'Deep Security Manager.exe') with the same name (ie 'dsm_s.vmoptions') • -Xmx2g • Recommend server-grade dual core (or higher) with 4GB of RAM

10. Configuration - # of concurrent Manager-Initiated Jobs • Default set to 5 * Number of Cores • Tuning to the HW/DB configuration provides benefit:  • dsm_c -action changesetting -name "configuration.schedulerPoolSizePerCPU" -value "5" • dsm_c -action changesetting -name "configuration.heartbeatPoolSizePerCPU" -value "20" • Balance required, tests show increasing the number evetually causes perf degradation

11. Configuration - # of concurrent Agent-Initiated Jobs • Default set to 20 * Number of Cores • Agents round-robin managers • Random entropy built in to Agent-initiated heartbeats to spread out load • Agents are rejected if # of concurrent jobs is surpassed on all nodes (they will wait until the next heartbeat)

12. Configuration - Size of Security Configuration sent in Update • Number of assigned rules has an impact on performance of updates • Use the Recommendation feature to tune the content needed • Testing shows a linear cost

13. Configuration - Size of Security Configuration sent in Update (part 2) • Firewall Rules are low impact (usually low #) • DPI Rules are low-medium impact (linear scale based on # of rules). High impact on Agent • Log Inspection Rules and Integrity Monitoring Rules have medium-high impact (Content is compiled for the Agent & requires roundtrip) • Update times differ based on: • Security Update version • Agent/Appliance version • Platform • Installed Software

14. Configuration Heartbeat Frequency • More timely Events retrieval = more overhead • Events are sent in batches, compressed, expanded, parsed then batch inserted in DB • Recommend: • 10 Minutes for Servers • 60 Minutes for Desktops • Impacts • # Events • Speed at which Security Update is distributed to all protected Computers • Ongoing Recommendation Scanning (If Enabled)

15. Agent Performance • Software Footprint Summary • Memory Impact • Network Impact • System Scanning Impact

16. Agent Software Footprint Summary • Agent-Windows-X.msi: ~ 3.5 MB • Typical disk footprint: • Software (~10 MB) • Data (Events) (~2 MB Desktop and ~12 MB Server)

17. Agent Software Footprint Details 17 Classification 12/3/2009

18. Agent Memory Impact • The memory impact by Deep Security Agent varies depending on the state of the system • Typically 15 – 30 MB User mode (DPI only, to DPI + Integrity Monitoring, Log Inspection • Typically 5 – 20 MB Kernel mode (Driver software, Config.bin, Connection memory) • Config.bin is read in to memory and varies depending on the number of rules assigned to a host • The number of active connections to the host also impacts memory

19. Agent CPU Impact (Traffic Processing) • There are a number of factors that affect how the processing network traffic affects performance • The number of rules assigned • The more rules that are assigned to a specific port the more processing is required • The traffic patterns within the rules • Depending on how complex the traffic pattern and matching rules are the rule itself can impact performance • The amount of traffic being filtered • The more connections to the box that is being filtered the more the performance impact

20. Agent CPU Impact (Traffic Processing) • Measure impact on host CPU for firewall & deep packet inspection for web server. • The number of assigned rules should typically be around ~ 100 or less when only recommended rules are assigned. • Bypass rules can be added for high throughput applications that don’t require scanning.

21. Agent CPU Impact (system scanning) • Scanning time for recommendation scan (used in IDS/IPS rules), integrity monitoring and log inspection heavily depends on the scope of the system being scanned. • This is defined by the rules that are assigned, and the number and sizes of files being monitored. • Recommendation scans and Integrity Monitoring scans can be scheduled using Scheduled Tasks to occur during off-peak times. By default they are unscheduled and occur only when requested by the user.

22. Agent Scanning Impact • The following are results for integrity monitoring using default Windows rules on a Windows 2003 server.

23. Agent Performance Summary • Overall the performance of the Deep Security Agent is dependent on many factors • Each component (DPI, Integrity Monitoring, Log Inspection) should be considered separately (and can be enabled/disabled individually) when diagnosing performance issues as each component interacts with the system in a different way • Field deployment issues we have seen are: • Too many rules deployed (run recommendation) • Filtering web traffic in reverse direction (new rule being issued) • Filtering backup traffic (use bypass rules)

24. Vmware/Appliance Performance • Functions are distributed across ESX hypervisor (FilterDriver), Security Appliance (DSVA) and In-guest agent. • Performance characteristics are similar to normal agent but magnified by number of VMs protected as these all share ESX hardware. • VMs with heavy traffic requirements should be deployed with an agent

25. Hypervisor Performance • Hypervisor memory usage increases with the number of protected guests, but only uses a small amount of memory, approx 200kB per protected VM (powered on). • Memory usage is not sensitive to the number of connections, number or type of DPI rules etc. • Tap mode, Standby mode and Bypass mode functions are implemented in hypervisor. Performance impact of hypervisor functions is negligible. • Standby mode performance is very close to normal agent performance. Bypass is slightly faster than normal agent bypass. • Tap-mode: guest network performance is not impacted as much but appliance uses CPU.

26. Appliance Performance • Initial Appliance Memory allocated 512MB. Each protected VM uses 10-20MB memory, plus leave ~100MB of temporary working memory. • Config and connection memory usage similar to normal agent for DPI rules. • Approx 50-100MB of disk depending on log configuration • Total appliance network throughput is approx 1Gbit shared across all VMs. For high throughput apps, an In-guest agent should be used. Bypass should be used for high traffic ports. • Appliance is not involved for VMs in standby mode or bypass traffic. • Powered off/suspended VMs do not use memory/cpu just disk

27. In-Guest vs Appliance Throughput • Shown below for a single VM, aggregate for multiple VMs is higher • No impact for VMs it is not protecting and virtually no impact for standby and bypass

28. Deep Security Virtual ApplianceAppliance Capacity • Max throughput The appliance is capable of up to 1Gbit/s aggregate throughput for all VMs it is protecting in one ESX node. The total VM throughput can be greater for VMs with in guest agents. Bypass rules can be defined for protected guests to bypass traffic at rates greater than 6Gbit/s. • Latency Baseline appliance latency is ~0.1ms per packet, ~0.2ms round trip. • CPU tuning By default 1 CPU is assigned to the appliance. For aggregate throughput above 500Mbits/s and DPI intensive, the appliance can be configured with 2 CPUs. Adding further CPUs is not currently expected to increase overall throughput.

29. Deep Security Virtual ApplianceAppliance Capacity • Number of Protected VMs The out of box maximum number of VMs protected by the appliance is 25 untuned (powered on). Some hypervisor tuning is required to support more VMs (technote to follow). For more than 10 powered on VMs VMware's 4.0 update 1 release is needed. • Disk Requirements The appliance requires 20GB disk, this includes 16GB for virtual agent logs and is sufficient for more than 200 protected VMs. • Memory Requirements The default memory allocation of the appliance is 512MB. This is enough for approx 25 VMs. For each additional 50 VMs, configure a total of 1GB for 100 VMs configure a total of 1.5GB (tune in virtual center vm settings)

30. THANK YOU!