An Architecture for Virtual Internets

1. An Architecture for Virtual Internets Joe Touch Director, Postel Center for Experimental Networking Computer Networks Division USC/ISI

2. Outline • Background • Architecture • Projects • X-Bone – FreeBSD/Linux tool to deploy VIs for experiments, testbeds, and lab classes • DynaBone – applying layered Vis for fault tolerance and DDOS resistance • NetFS – OS extension of network control and access API to support concurrent VIs

3. A network using encapsulation-based links A way to test new protocols A way to share infrastructure A way to virtualize a network topology as VM is to memory – What is a VI? –

4. Concurrent VIs Star-ovl Ring-ovl IP Base Network

5. IP Base B A D C ring-ovl star-ovl B B A A D D C C User’s view of Vis

6. Uses of VIs • Increase sharing • Concurrent use • Partition resources • Deploy peer services, test protocols • Simplify views of a complex structure • Hierarchies: layering (recursion), divide-and-conquer, embedding • Increase portability • Indirection allows remapping • Remapping for fault tolerance, mobility

7. Virtual Systems • Logical version of a real, physical resource • Virtual memory • Larger space • Via map memory onto hard disk • Virtual machine • Emulated PCs (VMware), portable code (P-mch, JVM) • Via emulation of PC • Virtual circuit • Multiple connections over a single path • Via packet swithing and end-to-end state

8. Network Virtualizations • Wire -> virtual wire (packets) • Share links, provide fault tolerance • NIC -> VIF • Emulate multiple end systems • LAN -> VLAN • Share switching / bridging resources • VPNs and overlays • Emulate and share the entire network

9. Challenges of VIs • Extension of the Internet architecture • Compatible, incrementally-deployable • Scalable deployment and management • Divide-and-conquer, merge, split • Inter-VI access • Access to services across VIboundaries • ‘the Graph Embedding problem’ • Optimization, fault tolerance can be hard

10. – VI Architecture – • Multihoming • Multiple Internets, not just AS’s • Use VIFs and iterative forwarding • Tunneling • Weak network layer for endpoint addressing • Strong link layer for routing, forwarding control • Integrate with dynamic routing, Ipsec • Addressing • In the end system, e.g., OS API • Naming over the wide area

11. Apps can’t select source IP, no IP w/o NIC NIC RFC 1122/1123 Host NIC IP address binds to one NIC Virtual Router VNIC NIC NIC NIC Multihomed Host IP address binds to each NIC Multihoming

12. Host Implications • Need for an internal router • Must participate in routing protocols • Input interface groups • Inaddr-any on subsets of interfaces • Output interface selection • VIF as source of all traffic • DNS • context sensitive replies

13. Router Implications • VI-sensitive forwarding • Solve via separate IP spaces(merge VI-ID with endpoint ID) • Intra-VI routing protocols • Solve via admit/exclude rulesamong subsets of interfaces(preprocess gated/mrtd config files)

14. Problem: lost context • Incoming tunnel is context • input (de-tunnel, de-IPSEC, demux) • forwarding • route exchanges • Keep this context • retain on decaps. • use as context for processing • currently via separate IP space • later via Overlay ID

15. Dynamic routing • Double encapsulation: • Overlay endpoints • Overlay link • Supports • Multihop overlay (routing within the overlay) • Multiple visits to a single router Ovl Link Base Inet Data Ovl Ends

16. DATA DATA DATA A A A à à à D D D Q à R X à Y DATA A à D S à T Y à Z Ovl-A Ovl-D OLink-Q OLink-T Base-X Base-Z Ovl-B Ovl-C HOST HOST OLink-R OLink-S Base-Y ROUTER DE Networking

17. 1 2 3 4 5 6 DE In Action 1. App emits (D)[-,E4] 2. *E routes to VIF1 3. VIF1 adds: source IP (D)[E1,E4] ‘link’ (D)[E1,E2]+[L1,L2] 4. L2 routes to VIF2 5. VIF2 adds ‘phys’ (D)[E1,E4][L1,L2]+[P1,P2] 6. Internet routes (D)[E1,E4][L1,L2][P1,P2]

18. Parallel tunnels • Multiple paths between two endpoints • allows a single node to play more than once in a single overlay • Multiple tunnels • ‘Strong’ host model (IP per NIC) • Peek-ahead during decapsulation • Provides per-tunnel statistics and control • Aliases • Susceptible to interface contention • Harder to control source address • Requires less OS support

19. Ovl-A Ovl-B Base-X Base-Y Olink-Q Olink-R Ovl-C Ovl-D Olink-S Olink-T Ovl-A Ovl-B Aliases: One VIF decapsulates both Ovl-C Ovl-D Base-X Base-Y Olink-R-Q Olink-T-S Olink-Q-R Olink-S-T DATA A à B Q à R X à Y DATA C à D S à T X à Y Multi-tunnel vs. aliases Multi-tunnel: Which VIF decapsulates? Packets on the wire (same)

20. HBH IPSEC • Use where E2E not available • Secures HBH protocols – routing, ICMP

21. IPSEC for the overlay DATA Ovl-Src, Ovl-Dst OLink-Src, OLink-Dst Base-Src, Base-Dst Application IPSEC (overlay endpoints) Virtual network IPSEC (overlay links) Base network IPSEC (base endpoints)

22. K1 B IPSEC Tun src à Tun dst Tunnel Mode IPSEC A Z DATA IP src à IP dst C K2 K1 V1 B IIPtran 2 A Z DATA IP src à IP dst IPSEC Tun src à Tun dst 1 C V2 K2 Dyn. Routing + IPSEC • Key-per-link interferes with routing • Solve with VIF using IPIP then IPSEC

23. Gated / mrtd via gated.conf / mrtd.conf script processing isolate RIP announcements within each overlay, separate from base network Mrouted via mrouted.conf pre-processing isolate overlay multicast routing via boundary on virtual IP interfaces Integrating Routing

24. Packet MTU limits Layers eat packet space May stress impls. Bandwidth costs 20% (10% IPSEC’d) Latency costs 0.02-0.06 msec per hop Costs of Encapsulation

25. Layered double tunnels DATA Base-Src, Base-Dst Base-Src, Base-Dst DATA Ovl-Src, Ovl-Dst OLink-Src, OLink-Dst Base-Src, Base-Dst DATA Ovl-Src, Ovl-Dst OLink-Src, OLink-Dst OvlSrc2-OvlDst2 OLinkS2-OLinkD2

26. (User Input) Overlay-Specific Parameters: TCL/ACL, JDK RD Node Action File RD http ring-ovl Node Action File B XB-OM A RD D C RD Node Action File Node Action File Problem: Service Deployment (XBone-Auto) Node-Specific Parameters: Ovl Name, IPs, Topology Generic ABone Generator Script Action File Generator Script

27. Network Reentrancy • Need VI context sensitive: • View of interface list • View of ports • Logins • File systems (for logs)

28. policy policy Problem: Recursion • Easy if deterministic • One inner layer • Harder if policy-based layering • Layer N determines Layer N-1 A X Y B C

29. Recursion solutions: • ARP • Treats lower layer like link • Needs broadcast • BGP • Treat inner network like a transit AS • Needs to determine encapsulation

30. ––– Projects ––– • X-Bone • DWIM VI Deployment • DynaBone • Multilayer spread-spectrum VIs • NetFS • Context-sensitive views

31. – DWIM VIs (X-Bone) – • DWIM concept • API • Useful defaults (esp. to get around complexities) • “COTS” distributed management • Expanding ring search • Soft state with hard backup • Heartbeats • ACLs and resource management

32. X-Bone Objectives • Dynamically deploy overlay networks • user/application setup, monitor, teardown • Via existing stacks in new ways • integrate IPsec, dynamic routing • With enhanced capability • hierarchical, stackable • nodes in multiple overlays, in a single overlay multiple times

33. IP Base B A D C ring-ovl star-ovl B B A A D D C C xd GUI Resource Daemon Overlay Manager Resource Daemon Resource Daemon link host router X-Bone System View Web GUI Multiple views Star Overlay Ring Overlay Base IPv4 Network Automated monitoring X-Bone system

34. Result sin eql cos div udel sec isipc2 bbn Creating the Ring Request OM Ring Ovl. Internet

35. X-Bone Components VI Architecture Impl. Cartwheels SNMP/RSVPDistributedControl

36. Reduce deployment effort Deploy tools easily, overlays effortlessly Safe configuration, management, monitoring Existing OSs, apps., network infrastructure Extend network architecture Dynamic, concurrent overlays Recursive / stackable overlays Share in multiple overlays, multiply in one Impact Goals

37. The X-Bone is… • A system for automated overlay deployment • among a closed set of trusted hosts and routers • provide coordination, configuration, management • many details are plug-replaceable • New tricks for overlays (use of overlays) • overlays on overlays on overlays on … • fault tolerance, service deployment • member in multiple overlays, in single multiple times • New tricks for old dogs (extend network arch.) • use existing stacks and applications

38. What We Don’t Do… • Optimize the overlay topology • we use a plug-in module (AI folk can provide) • it requires network status (emerging now) • fault tolerance only via ground truth (admin. issue) • X-Bone is capability more than performance (now) • Non-IP overlays • IP is the interoperability layer • IP recurses / stacks nicely

39. Before/after

40. Darwin/VNS (CMU) deploy a reserved core overlay (QoS) Netscript/VAN (Columbia) deploy a set of virtual NICs in EEs (Anets) Detour (U. Wash.) / RONs (MIT) patch routing with tunnels VPNs fence-out, incremental, exclusive, host-focus Multi-level – MorphNet, Supranet ATM – GUILN, Switchlets Manual overlays – Mbone, 6bone, A-Bone Related Work

41. Integrated end-to-end overlays overlays as more than an interim solution extend architecture (IPsec, multihoming) Recursive Internet architecture runs on IP; provides IP to upper level Deploying an alpha-grade tool increase sharing, ease setup (CAIRN, AN) simplify applications, user use safe, secure, coordinated Production use for classes, testbeds X-Bone Differences

42. X-Bone Users • ISI Network lab – 17 Fbsd/Linux • USC CS net lab – 24 Linux, 48 students • UCL - 6 Fbsd nodes • CAIRN – 10 Fbsd nodes • LUT / 3G - Finnish dynamic mobile svcs Canadian Gov’t (CRC) Project • A-Bone – deploying the backbone

43. –AntiDDOS (DynaBone)– • Spread-spectrum parallel defenses • RAID for packets • Adaptive configuration • Proactive and reactive management • Using existing OS/App/protocols • Like X-Bone

44. Performance tradeoffs Bandwidth CPU load Latency Security Service

45. Bandwidth variations

46. Latency variations

47. Innerlays Outerlay P R M P R M DynaBone architecture Spread-Spectrum Multilayer Internet Overlays 3DES encrypt / Linkstate RC5 encrypt / RIP MD5 auth / static Base network

48. Innerlays Outerlay 3DES encrypt / Linkstate P R M P R M RC5 encrypt / RIP MD5 auth / static Base network Reacting to attack X

49. DDOS Attack Detection Performance Metrics (pathchar) PRM Detail Mux per packet? per TCP? P R M Monitor inject measure M Demux reorder? drop dups?

50. Why use overlays? • PRMs can coordinate • FEC-style replicate on each Innerlay, filter copies at receiver • TCP SYN send on high-security Innerlay, data on high-speed Innerlay; receiver accepts SYNs only from high-security Innerlay • Algorithmic diversity • IPsec, routing, DNS, etc.