Stephen Schwab September 28, 2005

1. DETER/EMISTDDoS Defense Experimental MethodologyImpact of Traffic Generation Selection on Precision of Detection and Response Characterization Stephen Schwab September 28, 2005

2. SPARTA Team Participants • DETER • Steve Schwab, Ron Ostrenga, Brad Harris, David Balenson, Dan Sterne • EMIST DDoS • Steve Schwab, Brett Wilson, Ron Ostrenga, Roshan Thomas, Alefiya Hussain, Brad Harris, Dan Sterne

3. Overview • EMIST DDoS Experimental Methodology • FloodWatch and CloudShield Experiment Design • Testbed Methodology Automation • Process for Incorporating Hardware and Appliances • Traffic Generation • Visualization and Metrics • Future Plans

4. Objectives for EMIST • Create reusable library of test technology for conducting realistic, rigorous, reproducible, impartial tests • For assessing attack impact and defense effectiveness • Test data, test configurations, analysis software, and experiment automation tools • Provide usage examples and methodological guidance • Recommendations for selecting (or developing) tests and interpreting results • Test cases and results, possibly including benchmarks • Facilitate testing of prototypes during development and commercial products during evaluation

5. Canonical DDoS Experiment • DDoS experimentation involves a vast multidimensional space of experimental variables. • Canonical experiment form intended to organize the experimental space and facilitate navigation through it. • Canonical experiment consists of: • Attack Mechanisms • Background Traffic • Network Topology • Defense Mechanisms • Measurements and Metrics • Network Services Infrastructure • Risk

6. Typical DDoS Toolkit Architecture

7. Overview • EMIST DDoS Experimental Methodology • FloodWatch and CloudShield Experiment Design • Testbed Methodology Automation • Process for Incorporating Hardware and Appliances • Traffic Generation • Visualization and Metrics • Future Plans

8. SPARTA DDoS ExperimentSeptember 2005 Background Traffic: REPLAY | NTCG | HARPOON HIGH FIDELITY TRAFFIC Topology: BUILDING-BLOCKS | JUNIPER ROUTER CORE REALISTIC CONNECTIVITY AND SCALE-DOWN Attack Traffic: DETER-INTEGRATED ATTACK SCRIPTING AUTOMATION OF VARIETY OF SCENARIOS UNDER STUDY Instrumentation: PACKET AND HOST STATISTICS CAPTURE | SPECTRAL ANALYSIS | METRICS CALCULATION | INTEGRATED VISUALIZATION TOOLBOX FOR RIGOROUS INVESTIGATION OF RESULTS CORE AS-11357 ATTACK TRAFFIC BACKGROUND TRAFFIC

9. SPARTA DDoS ExperimentSeptember 2005 Defenses: FloodWatch STATISTICAL DDOS DETECTION McAfee IntruShield COMMERCIAL IDP APPLIANCE CloudShield NETWORK PROCESSOR APPLIANCE PLATFORM Juniper IDP-200 COMMERCIAL IDP APPLIANCE STUDY AND QUANTIFY HOW PLACEMENT WITHIN THE NETWORK IMPACTS EFFECTIVENESS OF DEFENSE DEFENSE DEPLOYMENT POINT

10. Example traffic flow From one edge network FloodWatch Experiment – Example Topology • Root and leaf networks transmit/receive replayed packets • Packet source and destination addresses randomly remapped to experiment topology • Each end network (leaf and root) is both a source and sink for background traffic

11. Host 2nd Tier Router Core Router Appliance Core Router 2nd Tier Router Host Schematic of Network Connectivity Juniper M7i Target or Attack Victim FloodWatch DDoS Statistical Detector CloudShield 2200 DDoS Entropy Detector Attack Source Replay Traffic Source TCP Traffic Source

12. Experiment Goal • Validate fidelity of attack and background traffic in reproducing characteristics of real DDoS experimental scenarios. • Case #1: Attack Traffic • Recreate an attack captured in a real-world network • Compare spectral analysis of real-world and testbed network to assess fidelity of phenomena reproduction. • Case #2: Background Traffic • Use TCP analytical model of throughput as a reference • Compare theoretical throughput with observed throughput for 100Mb/s and 1Gb/s networks • Compare model-vs-testbed discrepancy between100Mb/s and 1Gb/s to gauge ability to preserve phenomena while scaling-up traffic.

13. Overview • EMIST DDoS Experimental Methodology • FloodWatch and CloudShield Experiment Design • Testbed Methodology Automation • Process for Incorporating Hardware and Appliances • Traffic Generation • Visualization and Metrics • Future Plans

14. Automation • DDoS Requirements generalize to broader Security Experiment Requirements: • Automate whenever possible for: • Repeatability, efficiency, ease-of-use • Experiments described in Emulab ns2 format may include primitive events • Under base Emulab system, provides control for a very limited number of operations

15. Event System Integration • Testbed Automation Agents are controlled by events • Events can be created in several ways: • Specified in the NS file and played at runtime • Played from an external list of events generated by hand or script • Dynamically generated by a GUI or script • Current Agents • Injector: traffic replay agent • Flooder: attack agent • Collector: packet traces (TCPDUMP) control and per-node filter counters • Harpoon: traffic generation agent for the Harpoon TG • FloodWatch: controls FloodWatch DDoS defense

16. Automation Dashboard

17. Overview • EMIST DDoS Experimental Methodology • FloodWatch and CloudShield Experiment Design • Testbed Methodology Automation • Process for Incorporating Hardware and Appliances • Traffic Generation • Visualization and Metrics • Future Plans

18. Appliances • DETER requirement: support the experimental test and evaluation of appliances • Commercial products often packaged as appliances, critical future user segment • EMIST requirement: high-speed appliances stress the testbed and the tools supporting our methodology • Requirements: • Provide the ability to seamlessly integrate appliances as nodes in testbed experiments • Stress all aspects of our methodology at line-rate • Topology – Gigabit forwarding routers (Juniper) • Traffic – Greater aggregation • Data Capture – vanilla TCPDUMP inadequate

19. DDoS Network Appliance Defense Evaluation Scenarios • Introduce scenarios in which technology developers evaluate network appliances • CloudShield 2200 • IXP2850 network processor + high-level application development environment • Evaluate prototype statistical DDoS defenses • McAfee IntruShield 2600 • Commercial product, focus on DDoS capabilities • Juniper IDP-200 • Commercial product with 8 Gigabit ports enabling study of placement and connectivity • Push envelope of DDoS defense evaluation methodology to test Gigabit Ethernet rates

20. DETER Hardware and AppliancesProcess • Develop a systematic process for integrating hardware devices and appliances in the DETER testbed and within EMIST experiments: • Hardware connection • Control plane ns topology • Control plane manual configuration • Data plane manual configuration • Control and Data plane semi-automatic configuration (scripting) • Control and Data plane automation • Integrate generalized scripts behind the scenes into DETER and EMIST tools

21. Juniper Routers • Deter has 5 Juniper M7i routers • 4 Gigabit Ports/Router • The Juniper routers in DETER are almost first-class DETER experimental devices. • Can be allocated into an experiment by Assign • Can be assigned IP addresses within ns topology • Assign cannot YET configure router to use the IP addresses that were allocated • Must manually map the MAC and IP addresses into a router configuration • Plan to use the Juniper supported XML API to automatically configure the router

22. CloudShield Appliance • A CloudShield Appliance with 4 Gigabit interfaces has been added to DETER as an experimental device. • Can be allocated into an experiment by Assign • Must be configured manually • Mapping of interfaces into an experiment is difficult since there are no exposed MAC or IP addresses • Usage is complicated by the transparent bridging function that causes the DETER switches to go into layer 2 loops. • Spanning Tree Protocol (STP) is disabled on DETER

23. CloudShield CA2200 • 2 Intel IXP2800 network processors running CPOS • 16 microengines each • 1 StrongARM processor each • Dual Pentium-III management processor running Red Hat Linux • 4 GigE or 4 Copper 10/100/1000 network interfaces

24. Entropy Based Detection • Runs on CloudShield CA2200 network appliance • Data plane • Detectors: source IP, destination IP, packet size, source port, destination port • Create a histogram for each detector on packets in a fixed window size • Calculate entropy for each detector • Optionally filter packets • Control plane • Every N seconds, read entropy values • Compare with high/low thresholds for normal operation • When threshold is crossed, create a packet filter using max item from each detectors histogram

25. Entropy Detection Issues • Unable to run completely in the data plane • Packet processing runs in parallel • Entropy algorithm requires synchronization to avoid race conditions • CPOS (CloudShield Packet OS) provides no support for “manager threads” or mutexes

26. Chi-Square Algorithm • The top bin contains the item with the highest count • Next bin contains the four next highest counts, etc. • Items are swapped to different bins as their relative counts change • Chi-square statistic is the “shape” of the height of the buckets other 16 8 4 1

27. Problems with Chi-Square • When a packet is received on the data plane, the item counter is incremented • If the new count causes the item to be moved to another bin (because it has a higher count than one of the other items), the bins must be rebalanced • Since packets are received in parallel, it is necessary to lock the data structure so that only one thread can modify it during rebalancing • Without rebalancing, the algorithm can’t do detection • Without synchronization primitives, data plane can only collect data and detection must be done at the control plane

28. Experiment process using Juniper routers and CloudShield appliance • Allocated via assign into an experiment that put the CloudShield sensor on a GB link between two Juniper routers • Router interfaces had to be configured manually • Routes had to be configured manually • Configuration scripts were created to help speed up the experiment setup. • Routes needed to be verified after setup completed • There were issues in determining which physical CloudShield interfaces were actually being used

29. IntruShield Sensors • Two McAfee IntruShield Sensors, each with 2 unidirectional Gigabit interfaces and 6 unidirectional 100 MB IPS interfaces have been added to DETER as an experimental device. • Can be allocated into an experiment by Assign • Usage is complicated by the transparent bridging function that causes the DETER switches to go into layer 2 loops. (STP issue revisited) • May require physical attachment to a permanent anchor PC node. • Requires a dedicated Windows2000 Manager node

30. Overview • EMIST DDoS Experimental Methodology • FloodWatch and CloudShield Experiment Design • Testbed Methodology Automation • Process for Incorporating Hardware and Appliances • Traffic Generation • Visualization and Metrics • Future Plans

31. Background Traffic Generation • DDoS Specific Requirements: • Detection and mitigation of attacks need to be evaluated against a statistically valid mix of background traffic. • Interactive TCP traffic exhibits very different characteristics due to congestion control and recovery • Two traffic generation systems under active investigation and development: • NTCG (UC Davis) • Harpoon (U. Wisconsin)

32. Traffic Specifications Packet Trace analyzer TG, etc. commands Topology Description spec compiler NTCG: Network Traffic Compiler Generator • Interactive Traffic: Paradigm is to create controlled mix of traffic produced by an array of traffic generators (TG, etc.). Designed and developed by Allen Ting and colleagues/U.C. Davis

33. Harpoon Traffic Generator • Provides extreme configurability • specify precisely ordered temporal distributions of connection duration or use Netflow traces • accepts multiple subnets as destination, allowing a single Harpoon client to generate traffic across all networks • client and server accept either 0.0.0.0 (default routes) or the IP of a specific interface • Can connect to and listen on all TCP and UDP ports • Multithreaded implementation • Allows one agent to handle fifty or more connections (governed by the OS threads per process limit and CPU) • Each thread can connect to any destination, rather than using a single source / single sink paradigm • Plug-in architecture allows for extensibility • Well documented, clear and precise Developed by Joel Sommers and Paul Barford, U. Wisconsin

34. IP Address Diversity • Background traffic should involve a wide variety of IP addresses • There are usually not enough nodes to obtain enough IP diversity • Solution: Network Address Translation (NAT) • Using Linux IPTables, one machine can serve large blocks of IP addresses • Real machine 10.1.1.2 has a single IP address • Use IPTables to add a pre-routing entry to the NAT table • iptables -t nat -A PREROUTING -d 3.0.0.0/8 -j DNAT --to 10.1.1.2 • Packets arriving on any 3.0.0.0/8 are translated into packets destined for 10.1.1.2 • NAT is opaque to 10.1.1.2, allowing full TCP stacks to reply, including proper TCP backoff, etc.

35. Demo Video Clip • An Attack run against the FloodWatch DDoS defense, as performed and recorded by Brett Wilson on DETER.

36. Requirements for NAT (Routed) • Traditional network with routes in and out • Start NATs on each of the nodes • iptables -t nat -d 2.0.0.0/8 -j DNAT -A PREROUTING --to 10.1.2.3 • Configure routers so that there are valid routes for the NAT networks. • At the final hop, include a route to the NAT network (10.1.2.3) through the next hop • This prevents the router from ARPing for a NAT address, as it believes 10.1.2.3 is another route • On 10.1.2.3, IPTables catches the packet first (PREROUTING), translates it, and the connection completes

37. Requirements for NAT (LAN) • Isolated LAN configuration • More Complex Configuration • This scenario has no routers; all nodes are on a single broadcast domain • Configure each node so that the physical interface has a /32 netmask • ifconfig eth0 netmask 255.255.255.255 • eth0 will no longer answer ARP requests itself • Add explicit route for the experiment network via eth0 • Use the broadest net possible • route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0 • Add routes for all the NAT subnets on the other machines • route add -net 3.0.0.0 netmask 255.0.0.0 dev eth0 • Proxy ARP for all subnets this node is NATting. ./tarpd eth0 2.0.0.0 255.0.0.0 • Add the NAT via iptables • iptables -t nat -d 2.0.0.0/8 -j DNAT -A PREROUTING --to 10.1.2.3

38. Overview • EMIST DDoS Experimental Methodology • FloodWatch and CloudShield Experiment Design • Testbed Methodology Automation • Process for Incorporating Hardware and Appliances • Traffic Generation • Visualization and Metrics • Future Plans

39. Data Capture and Analysis • DDoS Specific Requirements: • Automate collection of both packet traces and host-based statistics • to calculate metrics • to analyze traffic behavior • Support visualization and interaction between experimenter and potentially large, unwieldy data harvested from the testbed

40. DDoS:Instrumentation and Visualization • Tracefiles produced, collected and extracted from DETER for post-mortem analysis. • ESVT analysis and experimental data graphical browser: • Navigate within experiment topology. • Drill down on link or node traffic/metrics: visualize and compare one or more statistics between different links or time intervals. • Key Milestone: Integrated support for visualization and analysis of DDoS experiments across multiple links, topologies, attack patterns, and defense configurations. • ESVT integration in collaboration with Jason Hart / Penn State

41. ESVT Experiment Topology

42. ESVT Visualization

43. ESVT Visualization: Connectivity

44. Spectral Analysis Rationale • Can we create high fidelity testbed experiments? • Gross parameters such as rate, packet type, etc. • Small time scale behavior captured by spectral analysis • High fidelity experiments help capture accurate interaction with cross traffic.

45. Spectral Analysis • Periodic behaviors encoded into the attack stream • Host, Network, aggregation effect, …. • Leverage experience from signal processing • FFTs, wavelets, detection theory, etc. 08:14:33.495046 2005host1.starwoodbroadband.com.domain > 2005host74.starwoodbroadband.com.32894: 40672 2/1/1 CNAME[|domain] 08:14:33.495924 2005host74.starwoodbroadband.com.33350 > camembear.fogcreek.net.http: S 1688588990:1688588990(0) win 5840 <mss 1460,sackOK,timestamp 27125396 0,nop,wscale 0> (DF) 08:14:33.496502 2005host74.starwoodbroadband.com.32894 > 2005host1.starwoodbroadband.com.domain: 19420+ PTR? 219.120.243.64.in-addr.arpa. (45) (DF) 08:14:33.496956 london-bar3.ja.net > 12.160.3.255: icmp: echo request 08:14:33.497347 2005host74.starwoodbroadband.com > london-bar3.ja.net: icmp: echo reply Characterize attack behavior

46. packet trace time-series FFT frequency domain Fourier Analysis 08:14:33.495046 2005host1.starwoodbroadband.com.domain > 2005host74.starwoodbroadband.com.32894: 40672 2/1/1 CNAME[|domain] 08:14:33.495924 2005host74.starwoodbroadband.com.33350 > camembear.fogcreek.net.http: S 1688588990:1688588990(0) win 5840 <mss 1460,sackOK,timestamp 27125396 0,nop,wscale 0> (DF) 08:14:33.496502 2005host74.starwoodbroadband.com.32894 > 2005host1.starwoodbroadband.com.domain: 19420+ PTR? 219.120.243.64.in-addr.arpa. (45) (DF) 08:14:33.496956 london-bar3.ja.net > 12.160.3.255: icmp: echo request 08:14:33.497347 2005host74.starwoodbroadband.com > london-bar3.ja.net: icmp: echo reply 08:14:33.499372 i.am.an.animal.and.i.irc.from.zoo-gate.fi > 12.160.3.255: icmp: echo request (DF) 08:14:33.499441 2005host74.starwoodbroadband.com > i.am.an.animal.and.i.irc.from.zoo-gate.fi: icmp: echo reply … • FFT summarizes the frequency content • Used to characterize spatial effects

47. Wavelet Analysis 08:14:33.495046 2005host1.starwoodbroadband.com.domain > 2005host74.starwoodbroadband.com.32894: 40672 2/1/1 CNAME[|domain] 08:14:33.495924 2005host74.starwoodbroadband.com.33350 > camembear.fogcreek.net.http: S 1688588990:1688588990(0) win 5840 <mss 1460,sackOK,timestamp 27125396 0,nop,wscale 0> (DF) 08:14:33.496502 2005host74.starwoodbroadband.com.32894 > 2005host1.starwoodbroadband.com.domain: 19420+ PTR? 219.120.243.64.in-addr.arpa. (45) (DF) 08:14:33.496956 london-bar3.ja.net > 12.160.3.255: icmp: echo request 08:14:33.497347 2005host74.starwoodbroadband.com > london-bar3.ja.net: icmp: echo reply 08:14:33.499372 i.am.an.animal.and.i.irc.from.zoo-gate.fi > 12.160.3.255: icmp: echo request (DF) 08:14:33.499441 2005host74.starwoodbroadband.com > i.am.an.animal.and.i.irc.from.zoo-gate.fi: icmp: echo reply … long packet trace time-series wavelet frequency and time domain • Wavelets summarize both time and frequency information • Used to characterize temporal effects

48. Case #1: Analysis technique • Validate fidelity of experiments by comparing spectral behavior of real-world traces to testbed experiments.

49. Experiment 1: Lander Attack TCP NULL attack from a single source sending 40B packets at 1100 packets/s Packet rate Trace Spectrum Bit rate Testbed Spectrum Number of Flows

50. Experiment 1: Lander Attack TCP NULL attack from a single source sending 40B packets at 1100 packets/s Testbed Spectrum