OmniPoP: GigaPoP-of-GigaPoPs Design and Future

1. OmniPoP:GigaPoP-of-GigaPoPsDesign and Future Patrick Christian, OmniPoP TAC (chair) University of Wisconsin - Madison April 25, 2007

2. Outline • Formation drivers • Administrative organization • Design goals • Architecture • Questions

3. Formation Drivers OmniPoP:GigaPoP-of-GigaPoPsDesign and Future

4. Omni- A prefix meaning “all” PoP Three letter acronym popular in network engineering meaning “Point of Presence” A place where an organization’s (or several) networking equipment is present Formation Drivers • OmniPoP • A place where all types of networking equipment is present • A place where all members are welcome to place equipment • A place to develop a rich mesh of all types of services between members • A facility to provide common and small group services

5. Formation Drivers

6. Formation Drivers

7. Administrative Organization OmniPoP:GigaPoP-of-GigaPoPsDesign and Future

8. Administrative Organization • CIC OmniPoP participants (4/07 connection status) • University of Chicago (soon) • University of Illinois (UIUC & UIC) (soon) • Indiana University (active) • University of Iowa (active) • University of Michigan (active) • Michigan State University (active) • University of Minnesota (active) • Northwestern University (soon) • Ohio State University (soon) • Purdue University (active) • University of Wisconsin – Madison (active)

9. Administrative Organization

10. Administrative Governance • Governing Board • One representative from each Partner (initially the participating CIC schools) • Responsible for • Policies • Finances, including major expenditures • Approving new Partners (not limited to CIC schools) • Selecting the OmniPoP operator • Requires two-thirds vote to approve all actions • Meets at least annually

11. Administrative Governance • Technical Advisory Council (TAC) • Up to two representatives from each Partner plus representation from the OmniPoP operator & CIC • Responsible for • Preparing an annual strategic technical plan • Recommending an annual budget • Recommending changes to services offered • Approving routine purchases and service contracts • Tactical Technical Team • Not more than six members • To work with the OmniPoP operator and CIC on day-to-day technical coordination of operational issues

12. Design Goals OmniPoP:GigaPoP-of-GigaPoPsDesign and Future

13. Design Goals • Platform capabilities • Robust, flexible and highly available infrastructure • OmniPoP facility capable/supports leading edge interfaces • High-speed interface density (needed 20x10GEs day 1) • Line rate AND/OR oversubscription architecture capabilities • Provision high-performance, full-featured switching & routing capability to reduce CIC duplication of infrastructure in Chicago • Transmit data at low latency and low jitter rates as well as minimize bottlenecks between devices • Separation of control/data plane

14. Design Goals • (select) Technical features • Nx10GE (802.3ad) link aggregation support (e.g. 2x10GE to I2) • Significant pluggable optics support • VLAN tagging (802.1q) & tag swapping • Queue in Queue capabilities • RADIUS support • Strong QoS capabilities • Multicast (PIM, MLD, and IGMP snooping) • Broadcast/multicast suppression • (rapid) Per VLAN spanning tree • Spanning tree controls (root guard, etc.) • BFD (bidirectional forwarding detection) • L2 filtering • Broadcast/multicast suppression

15. Design Goals • Misc. non-technical • Give up a little control to enable member-institutions to save $$$ & time • # & location(s) of facilities • How to buy issues (RFP v. existing contract) • Reduced operating system software complexity (single train to reduce “feature fights” within CIC) • Operator has experience managing platform (familiar CLI & tool extensions) – reduces OpEx

16. Architecture OmniPoP:GigaPoP-of-GigaPoPsDesign and Future

17. Architecture • Selected Force10 E1200 switch • Equipment collocated at “StarLight” (710 NLSD) • Each institution receives a 1 & 10GE “line rate port” • Required 20x10GE ports • Purchased 3 line rate & 2 (4:1) 16 port cards • Shared links use line rate ports • Best effort across shared links (for now) with no shaping/policing • Aggregation of CIC NLR access • Aggregation of connectivity to national/international R&E networks • Interconnection of CIC institutions to each other • (pending) Aggregation of Internet2 access

18. Architecture (future) • Commercial peering (non-transit/transit) • Collocation space in other Chicago locations • Layer 3 infrastructure • Layer 1 infrastructure??? • DRAGON/OSCARs RD&D???

19. Architecture – Phase I

20. OmniPoP: Phase II

21. Architecture - Current

22. Questions Patrick.christian@doit.wisc.edu

23. OmniPoP CollaborationLessons learned • Need collaboration idea & drivers to maintain group cohesion • Recognize and balance constant tension between collaboration and self-control exists & requires management • CIC an important group to facilitate “people networking” to identify other collaborators and common problems • Good ideas grow – Chicago fiber and OmniPoP collaboration groups started small (2 members) and grew quickly • Needs assessment to determine sharing opportunities difficult and time consuming but vital to success • Successful collaborations yield substantial rewards after accounting for additional time, communication and effort required to sustain collaboration