TransPAC QBSS “Scavenger Service”

1. TransPAC QBSS“Scavenger Service” Chris Robb Indiana University TransPAC Network Engineer chrobb@iu.edu

2. Topics • QBSS on TransPAC • Scavenger Availability • Juniper Implementation • Juniper Testing • Learnings • Future Directions • Questions and Comments TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

3. Scavenger on TransPAC • The need to implement QBSS on the TransPAC circuit came about after discussions with a group of researchers from Indiana University and the University of Tokyo. • A GRAPE-6 machine at Tokyo will be generating terabytes of data • Indiana University is supplying an HPSS archive to store the data for analysis by other US researches. • This data will need to be sent across the TransPAC OC-3 in a fashion that wouldn’t cause any significant network impact • Researchers agreed to tag their IP datagrams as QBSS so they wouldn’t interrupt the other Tokyo-Chicago traffic. • This project is slated to begin transfers very soon. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

4. Where is Scavenger Available? • TransPAC circuit from Tokyo to Chicago • We will focus on this link because it focuses on a Juniper implementation TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

5. Juniper Implementation • Both sides of the TransPAC link were serviced by Juniper routers, so a Juniper implementation of Scavenger Service needed to be created and tested. • Juniper proposed an implementation and the Global NOC volunteered to test its functionality. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

6. Juniper Test Setup • Juniper donated an M5 for general Global NOC testing in June of 2001 • The test setup was as follows: • Both workstations were Linux-based. JunOS 4.3 was loaded on the M5 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

7. Juniper Test Setup (cont.) • Juniper doesn’t yet have DSCP support, so TOS bits were used instead. The I2 QBSS implementation accounts for this and Juniper has said that DSCP support is forthcoming. • A precedence map was created to place the tagged packets into their appropriate queues. This remained fairly constant throughout the testing: precedence-map QBSS-test { bits 000 output-queue 0; <- Best Effort Traffic bits 001 output-queue 1; <- QBSS tagged traffic bits 101 output-queue 0; bits 010 output-queue 0; bits 011 output-queue 2; <- Premium traffic bits 100 output-queue 0; bits 110 output-queue 3; <- Network Management Traffic. Juniper advises bits 111 output-queue 3; <- not to change the queue assignment for these } TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

8. Juniper Test Setup (cont.) • After the traffic is placed in the queue, a WRR profile is created for the individual interfaces. • The following WRR profile was created to service the outgoing ATM interface: weighted-round-robin { output-queue 0 weight 98; <- best-effort traffic output-queue 1 weight 1; <- QBSS gets a minimum of 1% of the circuit output-queue 2 weight 0; <- Premium service receives no servicing output-queue 3 weight 1; <- Network Management traffic must have some servicing } • This configuration would change, depending on what we wanted to test. In the above configuration, we weren’t testing premium service TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

9. Juniper Test Setup (cont.) • A patched version of iPerf that allowed for TOS tagging was used to generate multiple streams from the GigE NIC towards the ATM NIC. • Most of the tests involved running 200Mbit UDP streams of one type, and then introducing another stream of the other type • The results were logged by iPerf at one second intervals, providing a picture of overall traffic behavior during the particular test. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

10. Test results • The overall results were positive! • The following graph shows the results of one particular test: • At +0s, a 200Mb QBSS stream was run. At +10s, a best-effort stream was introduced, pushing down the QBSS stream to about 1 percent of the available bandwidth. At + 40s, the BE stream was removed and the QBSS stream reclaimed the pipe. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

11. Test Results (cont.) • Here’s another graph showing a 200Mb QBSS stream before and after the introduction of a 10Mbit best-effort stream: • The full results of these tests can be viewed at: http://www.transpac.org/qbss-html TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

12. Learnings • Juniper will only use the WRR queuing profile when a circuit is saturated. • Because you cannot configure WRR queuing on a per-VC basis, this has wide implications for small VCs. • For example, a policed 10Mbit VC will never fill an entire OC-3 ATM PIC. • QBSS traffic on that VC will not be serviced any differently than best-effort traffic unless the circuit is filled up with traffic from other configured VCs • One way to get around this is to generate traffic on the other VCs to artificially fill up the circuit, although this is highly undesirable. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

13. Learnings (cont.) • Fortunately, Juniper does have a way around this! • There is a “hidden” command in the JunOS that will force all traffic on an interface, regardless of the link saturation, to be subject to the WRR profile: set chassis fpc <x> pic <y> transmit-buffers <n> • It is also important to set queue lengths for the individual VCs on the circuit to make sure that a small VC doesn’t eat up all the buffer space when delaying QBSS traffic. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

14. Learnings (cont.) • Expedited traffic must be placed in queue 0 • When swapping the best effort and QBSS queues around (i.e. QBSS in queue 0 and best-effort in queue 1), QBSS traffic did not yield to best-effort traffic • Instead, they evenly split the circuit, with both flows getting half of the bandwidth • We spoke at length with Juniper about this problem. Unfortunately, we had to return the M5 before this was resolved. • It isn’t crippling, but something to keep in mind TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

15. TransPAC Implementation • Thanks to the APAN NOC, TransPAC is now enabled for QBSS transmissions, using the WRR profile shown earlier. • Initial tests have proven positive, but more testing is needed before the GRAPE transfers are initiated. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

16. Future Directions • Results must be shared with QoS working group QBSS mailing list • More complete testing of the “queue 0” problem is needed • We also need to test TCP performance • IU has recently reacquired the hardware to complete such tests. Results will be shared. TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu

17. Questions? Comments? TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu