280 likes | 410 Views
London Internet Exchange Point Update. Keith Mitchell, Executive Chairman NANOG15 Meeting Denver, Jan 1999. LINX Update. LINX now has 63 members Second site now in use New Gigabit backbone in place Renumbered IXP LAN Some things we have learned! Statistics What’s coming in 1999.
E N D
London Internet ExchangePoint Update Keith Mitchell, Executive Chairman NANOG15 Meeting Denver, Jan 1999
LINX Update • LINX now has 63 members • Second site now in use • New Gigabit backbone in place • Renumbered IXP LAN • Some things we have learned! • Statistics • What’s coming in 1999
What is the LINX ? • UK National IXP • Not-for-profit co-operative of ISPs • Main aim to keep UK domestic Internet traffic in UK • Increasingly keeping EU traffic in EU
LINX Status • Established Oct 94 by 5 member ISPs • Now has 7 FTE dedicated staff • Sub-contracts co-location to 2 neutral sites in London Docklands: • Telehouse • TeleCity • Traffic doubling every 4-6 months !
Now totals 63 +10 since Oct 98 Recent UK members: RedNet XTML Mistral ICLnet Dialnet Recent non-UK members: Carrier 1 GTE Above Net Telecom Eireann Level 3 LINX membership
Second Site • Existing Telehouse site full until 99Q3 extension ready • TeleCity is new dedicated co-lo facility, 3 miles from Telehouse • Awarded LINX contract by open tender (8 submissions) • LINX has 16-rack suite • Space for 800 racks
Second Site • LINX has diverse dark fibre between sites (5km) • Same switch configuration as Telehouse site • Will have machines to act as hot backups for the servers in Telehouse • Will have a K.root server behind a transit router soon
LINX Traffic Issues • Bottleneck was inter-switch link between Catalyst 5000s • Cisco FDDI could no longer cope • 100baseT nearly full • Needed to upgrade to Gigabit backbone within existing site 98Q3
Gigabit Switch Options • Looked at 6 vendors: • Cabletron/Digital, Cisco, Extreme, Foundry, Packet Engines, Plaintree • Some highly cost-effective options available • But needed non-blocking, modular, future-proof equipment, not workgroup boxes
Old LINX Infrastructure • 5 Cisco Switches: • 2 x Catalyst 5000, 3 x Catalyst 1200 • 2 Plaintree switches • 2 x WaveSwitch 4800 • FDDI backbone • Switched FDDI ports • 10baseT & 100baseT ports • Media convertors for fibre ether (>100m)
New Infrastructure • Catalyst and Plaintree switches no longer in use • Catalyst 5000s appeared to have broadcast scaling issues regardless of Supervisor Engine • Plaintree switches had proven too unstable and unmanageable • Catalyst 1200s at end of useful life
New Infrastructure • Packet Engines PR-5200 • Chassis based 16 slot switch • Non-blocking 52Gbps backplane • Used for our core, primary switches • One in Telehouse, one in TeleCity • Will need a second one in Telehouse within this quarter • Supports 1000LX, 1000SX, FDDI and 10/100 ethernet
New Infrastructure • Packet Engines PR-1000: • Small version of PR-5200 • 1U switch; 2x SX and 20x 10/100 • Same chipset as 5200 • Extreme Summit 48: • Used for second connections • Gives vendor resiliency • Excellent edge switch - low cost per port and • 2x Gigabit, 48x 10/100 ethernet
New Infrastructure • Topology changes: • Aim to be able to have major failure in one switch without affecting member connectivity • Aim to have major failures on inter-switch links with out affecting connectivity • Ensure that inter-switch connections are not bottlenecks
New backbone • All primary inter-switch links are now gigabit • New kit on order to ensure that all inter-switch links are gigabit • Inter-switch traffic minimised by keeping all primary and all backup traffic on their own switches
IXP Switch Futures • Vendor claims of 1000baseProprietary 50km+ range are interesting • Need abuse prevention tools: • port filtering, RMON • Need traffic control tools: • member/member bandwidth limiting and measurement
Address Transition • Old IXP LAN was 194.68.130/24 • New allocation 195.66.224/19 • New IXP LAN 195.66.224/23 • “Striped” allocation on new LAN • 2 addresses per member, same last octet • About 100 routers involved
Address Migration Plan • Configured new address(es) as secondaries • Brought up peerings with their new addresses • When all peers are peering on new addresses, stopped old peerings • Swap over the secondary to the primary IP address
Address Migration Plan • Collector dropped peerings with old 194.68.130.0/24 addresses • Anyone not migrated at this stage lost direct peering with AS5459 • Eventually, old addresses no longer in use
What we have learned • ... the hard way! • Problems after renumbering • Some routers still using /24 netmask • Some members treating the /23 network as two /24s • Big problem if proxy ARP is involved! • Broadcast traffic bad for health • We have seen >50 ARP requests per second at worst times.
ARP Scaling Issues • Renumbering led to lots of ARP requests for unused IP addresses • ARP no-reply retransmit timer fixed time-out • Maintenance work led to groups of routers going down/up together • Synchronised “waves” of ARP requests
New MoU Prohibitions • Proxy ARP • ICMP redirects • Directed broadcasts • Spanning Tree • IGP broadcasts • All non-ARP MAC layer broadcasts
Statistics • LINX total traffic • 300 M/sec avg, 405 M/sec peak • Routing table • 9,200 out of 55,000 routes • k.root-servers • 2.2 Mbit/sec out, 640 Kbit/sec in • nic.uk • 150 Kbit/sec out, 60 Kbit/sec in
Things planned for ‘99 • Infrastructure spanning tree implementation • Completion of Stratum-1 NTP server • Work on an ARP server • Implementation of route server • Implementation of RIPE NCC test traffic box