Presented by Louis Marais (Deputy Director ICTS, UFS) and Andrew Alston (Owner, Alston Networks)

1. University of the Free StateRedesigning the campus network for innovation, VoIP, multicast and IPv6 Presented by Louis Marais (Deputy Director ICTS, UFS) and Andrew Alston (Owner, Alston Networks) AfriNIC Meeting – November 2012 Remote Presentation

2. The Original NetworkWhat we had: • Approximately 19000 Wired Ports • 450+ CCTV Cameras • 500+ Switches • 400+ Wireless Access Points • /16 Worth of Legacy IP Space

3. The Original NetworkWhat we didn’t have: • No Routing Protocols • No VLAN segmentation • No working multicast • No IPv6 • No true IP Plan • No QoS Implementation (or way to implement it!)

4. The Original NetworkWhat we REALLY had: • ONE Broadcast Domain • Overloaded ARP Tables • Switch CPU utilizations in excess of 90% • Regular broadcast storms • Spanning Tree / Topology Loop issues • A disaster waiting to happen….

5. Making a plan…. • Step 1 – Figure out what the objectives are • Step 2 – Figure out how to get there • Step 3 – Obtain the required prerequisites • Step 4 – Plan the deployment • Step 5 – Implement (carefully) • Step 6 – Resolve any unforeseen issues • Step 7 – Skills Transfer

6. The Objectives…. • Create a design that is scalable • Cater for the deployment of IPv6 • Cater for the deployment of Multicast Technologies • Cater for IP Telephony (VoIP) • Allow for sufficient flexibility to support innovation • Get rid of the single broadcast domain • Don’t over complicate – it has to be manageable • Avoid a design that will require huge amounts of downtime during implementation • Ensure that NAT doesn’t exist on campus – it breaks too many things

7. The design we chose… • Build on service provider principles • Segment the network – Core / Distribution / Edge • OSPF for loopback distribution purposes – We wanted ISIS but it wasn’t supported • BGP for the majority of the table (IPv4) • OSPFv3 for all IPv6 – The vendor didn’t support v6 under BGP • No VLAN spanning between distributions • PIM for multicast routing – The vendor didn’t support MSDP

8. The problems and prerequisites • Rollout of routing protocols, multicast and IPv6 could not begin before the network was properly segmented • Segmenting the network would normally result in large amounts of downtime due to subnet mask issues. • The switches required additional licenses that had to be purchased to support routing protocols

9. Why the segmentation problem? • Devices in the various domains would have the same subnet mask • Everything in each broadcast domain would still believe it was “directly connected” to everything in other broadcast domains • Either insert more specific routes on every device (impractical), or change the subnet masks on each device as we went (impractical)

10. The Solution? • Get more IP Space – Needed anyway for expansion • Renumber the newly structured segments into the new IP space as we went • Leave the original broadcast domain intact, simply reducing its size as we went

11. The planning exercise • Performed a comprehensive IP planning exercise, and figure out just how much IP space we needed both today and going forward • Applied for and were granted an additional /15 worth of IP space from AfriNIC • Put the new IP space through our IP plan and created a segmentation plan for the network • Network divided into Core, 13 distributions, and the edge • Work out the size of the supernets required per distribution • Devise a VLAN plan that could be duplicated across the distributions in a uniform fashion • Further divide the supernets routed to the distribution into VLAN based “connected” subnets.

12. Further planning details • Each building on campus (100+ of them) received 6 VLAN’s • Wired, Wireless, CCTV, Access Control, Building Management, VOIP • The distributions carried all VLAN’s for a number of buildings (In some cases up to 200 VLAN’s) • A decision was taken NOT to route the subnets, ONLY the supernets per distribution, let the distribution connected routes take care of the subnets. • Only place the core -> distribution point to points and the core/distribution loopbacks into OSPF, let BGP carry the rest. • Connected route redistribution into the dynamic routing protocols was avoided.

13. IP Planning – Supernet’s

14. IP Planning – Subnet’s

15. Management Tools

16. Management Tools (2)

17. Management Tools (3)

18. Management Tools (4)

19. Management Tools (5)

20. The Implementation… • Since VLAN 1 was already going to each distribution, leave it alone at the start… we didn’t want downtime • Create a second VLAN for each distribution, and trunk it between the core and the distribution. • Apply a /30 to each of these new point to point VLANs. • Create the required building VLAN’s on the distribution and assign the new gateway IPs to each VLAN. All of this in the newly allocated AfriNIC space. • Assign the IPv6 addressing to the new VLAN’s at the same time. • Setup the OSPF / OSPFv3 / BGP between the core and the distribution and ensure routing is working back to the core. • Trunk the building VLAN’s down to the edge switches from the distributions, again, leaving VLAN 1 still trunked to the edge • NOTE: At this point there was still ZERO effect on production traffic – we had NO downtime doing this.

21. Implementation Part 2… • Enable PIM Sparse Mode between the core and distributions on the point to point VLAN’s. • Ensure the DHCP relays were setup correctly on the distribution switches • Create the DHCP Scopes for the new IPv4 space. • We decided to use RA(EUI-64) for IPv6, since DHCPv6 is not as supported on all edge devices. • NOTE: We still had not touched production traffic, all of this was done as an “overlay” network.

22. The moment of truth… • Come back in the dead of night… consume lots of red bull… take a deep breath… and then…. • Re-Tag the edge ports into the new correct VLAN’s for their respective buildings / uses – LEAVING PRINTERS AND STATIC IP DEVICES IN VLAN 1 • Force a switch port flap (shut down the port, wait 30 seconds, un shut the port). • Edge devices see a disconnect and reconnect and request a new DHCP lease as a result. • On bringing the port back the edge device requests a new DHCP lease, and lands in the correct subnet. It also gets an IPv6 address through RA automatically. • Total downtime on any network segment: 30 seconds.

23. The secondary moment of truth… • Once we knew that edge devices were getting IP and IPv6 addresses in the correct subnets • Send some techs around to renumber static assigned devices (printers and such things) into the new space and move them into the correct VLAN’s as we went • Enable IPv6 on the proxy servers… • Enable IPv6 on the DNS servers… • Watch the traffic swing instantly to approx. 60% IPv6

24. The end result…. • Smaller Broadcast domains • Less broadcast traffic – an increase in network performance of *70%* • Switch CPU loads down from 90% to 3% as a result of smaller ARP tables • A more manageable network – we know where IP addresses are in use, so tracing things is easy • A more flexible network – work on one segment doesn’t affect the whole network • Far less spanning tree / looping / L2 topology issues • IPv6 to every edge device – yes, it can be done

25. The end result (IPv6 Stats)

26. The costs / time involved • Planning took 3 weeks – the large majority of the time • Implementation was done largely at night over a 2 week period • No one on the network saw more than 30 seconds of downtime • The total cost to achieve a 70% performance increase and ensure the network was future proof was less than $50,000. • The money spent included: • Routing licenses for the core and distribution switches (the majority of the cost) • Consultancy fees • Vast quantities of redbull and pizza over the 2 week implementation period.

27. Next Steps… Where to from here • We want a more robust multicast network – this means changing the way we do multicast. • In order to cater for multi-homing, better traffic analysis using netflow/ipfix, and to allow for more flexibility we want a full BGP table in the core • We want to reduce the over-subscription between the edge and distribution and distribution and core • MPLS Implementation across the border/core/distribution to allow for EoMPLS cross connects between network segments and external entities. • To this end… a core and distribution upgrade is planned for early 2013.

28. The new core / distribution design • Replace the core switches with proper routers • Replace the distribution layer with 10G capable L3 switches that support the required protocols • Implement a proper border router that is full table BGP capable • Implement MSDP from border to core to distribution. • Uplink each distribution with redundant links to separate core routers in a virtual chassis configuration • Ensure that each distribution has at least 20G back to the core (and potentially more) • Link the cores with redundant 100G technology. • Rollout MPLS across the border/core/distribution. • Support QoS where appropriate for VoIP implementation

29. The new core / distribution design

30. Lessons Learnt • Re-structuring a network for IPv6 / Multicast / The Future does not: • Require loads of downtime • Require vast amounts of money • Re-structuring a network for IPv6 / Multicast / The Future does require: • Some vision as to where you want to go • VERY careful planning • Careful documentation

31. THANK YOU! Any questions? Our Contact Details: Louis Marais: maraisl@ufs.ac.za Andrew Alston: aa@alstonnetworks.net