IRNC Measurement

1. IRNC Measurement John Hicks HPCC Engineer Indiana University 18th APAN Meeting – Cairns 4-July-2004

2. Overview • Global Federated Measurement Infrastructure • Similarities with the Abilene Observatory • Tools • NLANR MNA • Work with APAN JAPAN

3. Global Federated Measurement Infrastructure • The Consortium of International Research and Education Networks (CIREN) is name of Indiana University’s response to the NSF’s International Research Network Connections (IRNC) program solicitation. • CIREN’s goal is to connect the Asian Pacific Advanced Network (APAN) in Asia, the GEANT network in Europe and the CLARA network in Latin America to the US high performance infrastructure (Abilene and the various Federal networks). • CIREN US node locations include the East and West coast as well as the Midwest.

4. Measurement Infrastructure • An important part of CIREN is to develop a Global Federated Measurement Infrastructure. • In order to effectively monitor the performance and utilization of a global high performance networks, we need to bring together existing and emerging measurement and analysis tools and frameworks. • This environment needs to provide measurement data to network engineers and researchers. • The data needs to be stored and published in standard data formats agreed upon by the measurement community. • We will start with the schemas developed by the GGF NMWG.

5. Measurement Infrastructure (Cont.) • Data will be collected and published through the Global NOC at Indiana University and databases will be replicated at strategic location to provide timely access worldwide. • Through a publicly available web interface, we would like to provide a “one stop shop” for all your measurement needs. • The idea is to provide a facility that will let network engineers and researchers visit one location to decompose a network path, segment by segment, in order to determine performance characteristics and help debug network problems.

6. Measurement Infrastructure (Cont.) • We will deploy hardware (similar to the Abilene Observatory hardware) to strategic location on the network and work closely with Internet2, APAN, GEANT, and CLARA to coordinate measurement efforts. • We are already working with Internet2 and APAN to accomplish this goal. • Internet2 is also working with GEANT.

7. Similarities with the Abilene Observatory • Like the Abilene Observatory, this program supports the collection and dissemination of network data. It provides Network Engineers with an operational view of the network. • It also provides researchers a platform to conduct experiments and collect data from a High-Performance network. • Rack space, power, and connectivity will be provided to network researchers wanted access to high performance international networks. • Data from these projects must be made publicly available in some form.

8. Co-locating a measurement project • This project will reserve space for researchers to deploy equipment in the CIREN Rack. • Formal procedures will be announced when appropriate. • Requirements are similar to the Abilene Observatory: • Description of the Project including participants and duration • Space, Network Power Requirements • System Information • Security

9. Co-locating Caveats • Commodity Routes will not available • 48V DC Power (most places) • 23” Racks • Address Space • Security • “Lights Out” Remote Operation

10. Measurement Racks • In each CIREN node there is a one rack dedicated to the measurement project. The rack is dedicated for: • NOC Administrated measurement Machines • Co-located Machines • DC Power Controllers for Observatory machines No Routing equipment is in the Rack

11. Typical configuration (front)

12. Typical configuration (back)

13. Advanced services • Since the Machines are directly connected to the Backbone there are many advantages to test “Advanced” Services • Native V6 • Native V4 and V6 Multicast • 1Gb ports directly connected to the backbone • 9000 MTU • Alternative TCP project testing, such as FAST TCP and Highspeed TCP, will also be supported.

14. Monitoring • The state of the machines will be closely monitored. • Nagios/AlertMon monitor and display alerts if any machine or service is down • Ganglia Cluster Toolkit is used for system (load,mem,disk usage, etc..) monitoring

15. Tools • Netflow • Owamp (One-Way Latency) • BWCTL • SNMP Interface Statistics • Multicast Beacon • Ping/Traceroute V6 Destination • NLANR AMP • NLANR PMA

16. Netflow • Sampled (100:1) Netflow will be sent from all CIREN Routers to one of the local measurement machines. The flows are sent to researchers and also cashed locally and retrieved to a central storage using rsync. • The Netflow records will be anatomized by masking the low-order 11 bits of the IP address. Unanatomized data is not stored.

17. Netflow Data • Netflow Data will be available ether as a direct feed from the measurement Machines or as a download from the centralized storage area at the Global NOC • More information will be available about retrieving netflow data when appropriate

18. Owamp • One-Way Latency Measurements using a mesh of nodes in each CIREN Node. • Owamp tests one-way latency between machines using both IPv4 and IOv6. The measurement machines will be synchronized with Praecis Ct CDMA Clocks connected to each machine. • Data is collected from each machine and processed in a centralized database.

19. BWCTL • BWCTL is a command line client application and a scheduling and policy daemon that wraps Iperf. • Supports (and requires) versions 1.7.0 or 2.0.b of Iperf. • Full IPv6 support. • Supports dynamic TCP window size determination.

20. SNMP Interface Statistics • SNMP Collection will be done in a distributed way on the measurement machines. Data for local routers and switches are captured and copied back to a central repository. • The data collected is a Hi-Resolution (10 sec) capture of interface and environmental statistics • In addition a SNMP router proxy similar to the Abilene router proxy is in the works to allow access to query SNMP variables on the router

21. Multicast Beacon • Mesh of all CIREN Router Nodes, running on measurement machines • Modified version of NLANR Multicast Beacon • Saves data into RRD Database • Graphs of Delay, Loss and Jitter statistics • Multicast Group 233.1.2.3

22. NLANR AMP • The Active Measurement Project (AMP) has a joint research/engineering focus with site-to-site active measurements and analyses conducted between locations connected by high performance networks. • The AMP monitors measure the "heartbeat" of the high performance connection (HPC) networks, dissecting the networks in over 15,000 ways every minute. • The performance of AMP monitors are well known and we would like to work with our international partners to extend the deployment of AMP machines.

23. NLANR PMA • The goal of the PMA project is to deliver new insights into the operation, behavior, and health of the Internet, for the benefit of network users and operations. • Passive header trace data provides the means to study workload profiles for a number of strategically located measurement points in high speed environments. • Passive monitors are available up to OC192 but they are prohibitively expensive. • We will support the PMA project by provide an environment to deploy passive monitors.

24. Work with APAN JAPAN • We are currently working with the APAN-JP NOC to establish a measurement environment across TransPAC. • The measurement effort will be consistent with that of the Abilene Observatory (i.e. PiPEs, including bwctl and owamp and other codes). • Other measurement projects, discussed in a later talk, will also be deployed. • This will provide valuable measurement data for network engineers and researcher using APAN, TransPAC, and Abilene.

25. Work with APAN JAPAN (Cont.) • Strong relationships are required to make this global measurement environment successful. • A special thanks goes to the APAN-JP NOC for their support and eagerness to move the development and deployment forward.

26. Thank you John Hicks Indiana University jhicks@iu.edu