DREN IPv6 Implementation Update

DREN IPv6 Implementation Update Techs in Paradise, 2008 Jan 21, 2008 Honolulu, HI Ron Broersma DREN Chief Engineer High Performance Computing Modernization Program ron@spawar.navy.mil DREN IPv6 Update

Background • WAN provider for the DoD R&D community • Also serving as DoD’s IPv6 pilot implementation of the DoD CIO June ’08 mandate • Deploying IPv6 where possible in a production environment • See what works and what’s broken • See what’s missing • Share lessons learned DREN IPv6 Update

Previously • Reported to you previously: • Most serious problem is lack of IPv6 support in many security products (firewall, IDS, IPS, VPNs, web proxy, etc). • Major inhibitor to deployment of IPv6 in protected enclaves • Numerous bugs • hurts deployment and require workarounds • Increased complexities with running dual-stack • Vista missing some v6 support • Some things getting better • Important fixes in key software components • Some incentives from Asian demands • The scare over RH0, now deprecated • Router meltdown • Proponents and providers aren’t eating their own dogfood DREN IPv6 Update

What’s new • Still finding bugs, but they are getting harder to diagnose • Increased campus deployment efforts • Expanded peering • Testbed downsizing • Tunnel brokers updated • www.v6.dren.net web site restored and updated • Moved to new server (virtual) • Fedora 7 • Some cleanup and restructuring of content • DHPCv6 interoperability testing • Vendors still aren’t eating their own dogfood DREN IPv6 Update

DREN IPv6 Testbed • Starting to shut down DRENv6 testbed • 3 (of 9) Core nodes shut down • ERDC • NAVO • AFRL Kirtland • Moving Peering from testbed to production network • SPRINT • UUNET • NTT/Verio • … and others in the works DREN IPv6 Update

DRENv6 “testbed”Logical Topology Cisco AIX-v6 C&W Global Crossing 6TAP Abilene FIX-West Hurricane Electric Abilene LAVAnet TIC WPAFB Dayton NTTCom Verio ARL JITC HP Aberdeen Tunnel broker WCISD San Diego SD-NAP SDSC AOL SSC San Diego Wash D.C. SPRINT HICv6 (Hawaii) NRL Vicksburg Albuquerque SSC Charleston SSAPAC ERDC AFRL Kirtland AFB Stennis vBNS+ ATM PVC (OC-3) NAVO IXP Core Router tunnel DREN IPv6 Update ISP or BGP Neighbor “site”

Peering • Upgraded all NIDS at peering locations to insure visibility of IPv6 traffic (security). • Renewed effort to improve peering • Production network only had 8 operational as of 2 months ago. • Move peers from testbed to production • Production network used testbed for transit to most peers. • Add new peers • UUNET (WAE and HAY) • TWAREN (Seattle, working on Starlight) • SPRINT (MAEwest) • NTT/Verio (vastly improved performance to Japan sites0 • NLR (Seattle, MAEwest) • Hurricane Electric (WAE tunnel 1/22/08, MAE west cross connect being worked) • UH (last week) • HIX and LavaNet in the works • USGS (this week) • Qwest (working final draft of agreement) • Had to adjust policy • Waive requirement to peer at 3 locations • Waive requirement for high bandwidth native peering DREN IPv6 Update

Path from UH to DREN Before… dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.net traceroute6 to narf.v6.dren.net (2001:480:10:1050::5) from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets 1 2607:f278:5:3::1 2.873 ms 3.163 ms 5.251 ms 2 2607:f278:0:5::2 19.21 ms 6.222 ms 12.63 ms 3 so-2-1-0.bb1.a.syd.aarnet.net.au 95.954 ms 103.687 ms 102.948 ms 4 ge-0-0-0.bb1.b.syd.aarnet.net.au 107.493 ms 95.859 ms 100.499 ms 5 so-2-1-0.bb1.b.sea.aarnet.net.au 264.709 ms 259.346 ms 259.667 ms 6 dren-1-lo-jmb-706.sttlwa.pacificwave.net 157.101 ms 160.606 ms 170.337 ms 7 so12-0-0-0.sandiego.dren.net 189.137 ms 189.147 ms 186.446 ms 8 narf.v6.dren.net 186.873 ms 190.175 ms 207.715 ms dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.net traceroute6 to narf.v6.dren.net (2001:480:10:1050::5) from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets 1 2607:f278:5:3::1 2.927 ms 2.413 ms 4.273 ms 2 2607:f278:0:1::1 9.538 ms 18.885 ms 2.41 ms 3 2001:480:0:c81::1 17.07 ms 2.736 ms 3.02 ms 4 2001:480:0:c2f::1 54.571 ms 57.151 ms 54.373 ms 5 so12-0-0-0.sandiego.dren.net 88.173 ms 90.586 ms 84.086 ms 6 narf.v6.dren.net 84.489 ms 84.001 ms 84.304 ms After… DREN IPv6 Update

Recent Frustrations • Products still not doing IPv6 • Juniper NSM - slipped 18 months • Tipping Point - slipped 18 months • Bluecoat web/proxy - still no beta code, although promised a year ago • Bugs, bugs, bugs • Netscreen ND is broken • BIND sometimes stops responding when using IPv6 transport • Losing first packet breaks NTP • ping6 annoyance • … and more DREN IPv6 Update

Netscreen ND bug No. Time Source Destination Protocol Info 1 0.000000 fe80::211:92ff:fe10:ca90 ff02::1:ff00:1 ICMPv6 Neighbor solicitation 2 0.001721 fe80::212:1eff:feaf:9906 fe80::211:92ff:fe10:ca90 ICMPv6 Neighbor advertisement Internet Control Message Protocol v6 Type: 136 (Neighbor advertisement) Code: 0 Checksum: 0x5e5b [correct] Flags: 0x00000060 0... .... .... .... .... .... .... .... = Not Router .0.. .... .... .... .... .... .... .... = Not Solicited ..0. .... .... .... .... .... .... .... = Not Override Target: 2001:480:822:1f01::1 ICMPv6 Option (Target link-layer address) Type: Target link-layer address (2) Length: 8 Link-layer address: 00:12:1e:af:99:06 DREN IPv6 Update

Neighbor Advertisement 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Code | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |R|S|O| Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + Target Address + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options ... +-+-+-+-+-+-+-+-+-+-+-+- S Solicited flag. When set, the S-bit indicates that the advertisement was sent in response to a Neighbor Solicitation from the Destination address. The S-bit is used as a reachability confirmation for Neighbor Unreachability Detection. It MUST NOT be set in multicast advertisements or in unsolicited unicast advertisements. DREN IPv6 Update

Switch loses first packet • Foundry BigIron will lose first packet after period of no activity When the mac address of ins1.sd is not in the mac table of the BigIron....[ron@ins2.sd ~]$ ping6 ins1.sdPING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=0.778 ms64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=2 ttl=64 time=1.87 msWhen the mac address is in the table....[ron@ins2.sd ~]$ ping6 ins1.sdPING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=0 ttl=64 time=0.789 ms64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=4.85 msNotice the first time that it loses sequence 0. DREN IPv6 Update

Losing first packet First there is the IPv6 neighbor discovery, which makes it through and is answered because it is sent to a ethernet multicast address..17:57:26.913312 00:0b:db:93:b5:73 > 33:33:ff:00:00:02, 2001:480:10:4::3 > ff02::1:ff00:2: icmp6: neighbor sol: who has 2001:480:10:4::217:57:26.915433 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3: icmp6: neighbor adv: tgt is 2001:480:10:4::2By now the mac address (00:0b:db:93:b5:85) should be in the mac table (but I think it isn't yet, due to some internal delay). The next packet is the one that never makes it through the BigIron...17:57:26.915457 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 0That echo request is never replied to. But, the next one makes it OK...17:57:27.912144 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 117:57:27.912901 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3: icmp6: echo reply seq 1Is it possible that 24 microseconds (between the NA and the echo request packet) isn't enough time for the BigIron to get the mac table updated? DREN IPv6 Update

Losing first packet: impact • NTP fails • Normally backs off to 1024 second updates • Single UDP packet sent on update • Client never sees packet, declares server down, and restart at 64 seconds, backs off, then fails again DREN IPv6 Update

ping6 does a DNS lookup on EVERY packet Bug in addrconf patch The value was never copied to the cache ping6 annoyance --- iputils/ping.c.addrcache 2002-09-20 17:08:11.000000000 +0200 +++ iputils/ping.c 2003-05-15 16:41:19.000000000 +0200 @@ -1124,6 +1124,12 @@ { struct hostent *hp; static char buf[4096]; + static __u32 addr_cache = 0; + + if ( addr == addr_cache ) + return buf; + + addr_cache = addr; if ((options & F_NUMERIC) || !(hp = gethostbyaddr((char *)&addr, 4, AF_INET))) --- iputils/ping6.c.addrcache 2002-09-20 17:08:11.000000000 +0200 +++ iputils/ping6.c 2003-05-15 16:41:19.000000000 +0200 @@ -893,7 +893,14 @@ */ char * pr_addr(struct in6_addr *addr) { - struct hostent *hp = NULL; + static struct hostent *hp = NULL; + static struct in6_addr addr_cache = {{{0,0,0,0}}}; + + if ( addr->s6_addr32[0] == addr_cache.s6_addr32[0] && + addr->s6_addr32[1] == addr_cache.s6_addr32[1] && + addr->s6_addr32[2] == addr_cache.s6_addr32[2] && + addr->s6_addr32[3] == addr_cache.s6_addr32[3] ) + return hp ? hp->h_name : pr_addr_n(addr); if (!(options&F_NUMERIC)) hp = gethostbyaddr((__u8*)addr, sizeof(struct in6_addr), AF_INET6); DREN IPv6 Update

Expanded deployment • Many campus LANs now 100% IPv6 enabled • One site has achieved dual-stacking ALL desktops and servers (except 2) • Others have active programs to do the same, along with getting help desk trained and tooled for supporting it. • Remove “A” record from DNS for servers • Possible when all clients for that server are dual-stack • Servers running v6-only (no IPv4 address on any interface, nor on network it is connected to) • Fedora 7 - some manual fiddling of the config is required, but works • DNS issues (BIND sometimes stops responding when doing v6 transport queries) DREN IPv6 Update

Fedora 7 IPv6-only • Connect to IPv6-only LAN, so no cheating. • Configure from GUI, like any point-and-click sysadmin would do. • Getting it to actually work… • Manual hacking of ifcfg-eth0 • Delete IPV6ADDR and IPV6PREFIX • Set ONBOOT = yes • Fix broken /etc/hosts • Configure sendmail to listen on ::1 • Fix /etc/yum.repos.d/* files to point to an IPv6-capable mirror DREN IPv6 Update

IPv6-enablement milestonesand scoring (proposed) • IPv6 address allocation and address/routing plan developed • LAN (wired and wireless) is fully v6-enabled (all routers do v6 on all interfaces) and is connected to the IPv6 Internet (WAN). • The implication is that any v6-enabled device can connect anywhere in the LAN and get IPv6 Internet connectivity. • (worry about routing implementation, make sure address plan is right, think about security and performance, play with multicast, make sure network staff is trained to support it, etc) • Internal infrastructure services (DNS, NTP, DHCP, SMTP, etc) are IPv6-enabled • Public facing services (public DNS, MXs, public web site) are IPv6-enabled • Network management infrastructure (management LAN, SNMP, SYSLOG, MIBs, management access to switches/routers/etc) is IPv6-enabled • Most workstations and servers are v6-enabled • (behind this is the support infrastructure, i.e. help desk and other IT support, and a message to sys admins to turn on IPv6 where possible, and servers have AAAA records in DNS, and your help desk tools/scripts for things like host registration and update are upgraded to handle IPv6 addresses • Once critical mass is reached on the client side, remove "A" records for servers (resulting in final incentive and some pain for those that didn't dual-stack their workstations). • You don't need to do them all at once, just one at a time as their clients all become dual-stack • Migrate some network segments to IPv6-only, with no IPv4 addresses anywhere • (this forces one to figure out how to make hosts operate in a v6-only environment, helps the organization figure out what's missing, forces the implementation of some kind of transition mechanism to bridge to the IPv4-only world, plus adds continued incentive to get more stragglers upgraded since they can't even get there by typing the IP address • Deploy advanced features (mobile-ip, v6 multicast, etc.), provide remote IPv6 access (travellers, telecommuters, home, etc) from v4-only environments, cleanup and reduce complexity (readdressing network if necessary), .... • Declare victory • you claim a perfect score in public, and are willing to stand up to scrutiny Scoring: Scale of 0 to 10. 1 point for any milestone that is 95% complete. DREN IPv6 Update

Commitment to IPv6? • Are network product vendors really committed to IPv6 support? • Are they using it in their production networks? • Do they have an IPv6 presence on the Internet? • Do they follow the “eat your own dogfood” principle? • A survey… DREN IPv6 Update

Vendor scorecard June 2007 Jan 2008 • Looked in DNS to see if there were AAAA records for www, MX, and DNS. • Quick sampling of major computer and network companies showed no public facing IPv6. • 6 months ago, and then today. DREN IPv6 Update

Scorecard – IPv6 Summit Sponsors (March ’07) • Grand and Gold Sponsors of 2007 IPv6 Summit. • Only one has an IPv6 presence at their corporate “front door” DREN IPv6 Update

Summary: Situation Today • DREN has been successfully using IPv6 in a production environment, with many dual-stack systems and services, for years • Modern operating systems just work, out of the box (MacOSX, Linux, Vista, Solaris 10, etc) • Most urgent needs from our perspective: • Need parity with IPv4 in all implementations • Enabling IPv6 must NOT break things • Need to make security stacks fully IPv6 capable • Firewalls, IDS, proxies, IDP/IPS, ACLs • Eat your own dogfood • The rest of DoD will not make the June ’08 deadline. • The US is falling far behind Asia and other regions. • Prevalent attitude: IPv6 just another GOSIP, or ADA, or …. • A call to action DREN IPv6 Update

END DREN IPv6 Update

DREN IPv6 Implementation Update