QBone Service Specification and TE Early DiffServ Deployment Experiences and Reduced Expectations

1. QBone Service Specification and TEEarly DiffServ Deployment Experiences and Reduced Expectations Ben Teitelbaum <ben@internet2.edu> January 25th, 2001 TEQUILA Workshop on Internet Design for SLS Delivery

2. “The Holy Grail of computer networking is to design a network that has the flexibility and low cost of the Internet, yet offers the end-to-end quality of service guarantees of the telephone network.” • - S. Keshav Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

3. The Paradox of Internet QoS • Internet’s success largely due to its lack of QoS! • Best-Efforts: scalability on steroids • Routers: complexity of forwarding very low • L2: minimal requirements • Providers: “magic”-free provisioning and low user expectations • Developers: E2E transport and simplicity of Berkeley socket interface • QoS complicates everything (costs), but opens many opportunities (benefits) • Developers: new apps exploit the ubiquity of the Internet • Providers: may make more efficient use of the network • Big Question: How can we build upon DiffServ without compromising scalability? Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

4. How We Got Here (short version) • Began chanting: “enable advanced applications,…” • Assessed requirements “What do you want?” “What can you give us?” • Recommended DiffServ • Selected “Premium” service to meet demands of loss/jitter sensitive apps • Charted QBone initiative • Specified QBone architecture • Now proceeding to implement it and tweak the architecture architecture deployment Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

5. QBone Architecture • A Service: QBone Premium Service • Built on Expedited Forwarding (EF) (RFC 2598) • Assurance: near-zero loss & low, bounded jitter for marked traffic conforming to a specified peak rate • a.k.a. “virtual leased line”, “virtual wire” • Reservation Setup Protocol • Now: long-lived, manual setup • Proposed: SIBBS protocol between QBone domains; RSVP end-to-end between hosts • QBone Measurement Architecture • Uniform collection of QoS metrics • Uniform dissemination interface Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

6. Collection metrics,EFandBE... Active metrics (paths) One-way delay-variation One-way loss Traceroutes e.g. IPPM Surveyors Passive metrics (interfaces) Load EF reservation load Discards (suggested) Link bandwidths (suggested) e.g. OCxMon, RTFM, MIBs QBone Measurement Architecture1/2 ActiveMeasurements MIB-basedstatistics Boundary Router AM node Intra-Domain Premium Path Inter-Domain Premium Path PM node PM node PassiveMeasurements PassiveMeasurements QBone Domain2 QBone Domain1 QBone Domain3 Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

7. QBone Measurement Architecture2/2 • Dissemination • Standard URL query syntax: • whois server to learn canonical names for QBone domains, routers, sniffers, etc • label ::= <alphanum> { <alphanum> } • router ::= label"-ROUTER” • probe ::= label"-PROBE” • sniffer ::= label "-SNIFFER” • host ::= router | probe | sniffer • path ::= host "/" host • PHB ::= "BE" | "EF” • metric ::= "LOSS" | "ONEWAY" | "PING" | "IPDV" | "LOAD" | "TRACEROUTE" | "COMMITMENT" | "RESERVATION" • year ::= digitdigitdigitdigit • month ::= "01" | "02" | ... | "12” • day ::= "01" | "02" | ... | "31” • YYYYMMDD ::= yearmonthday • aggregation ::= <unsigned_integer> • prefix ::= “http” | “ftp” | <other> • query ::= prefix ”://" path "/" PHB "/" metric "/” YYYYMMDD "/" metric "/" aggregation Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

8. Simple Interdomain Bandwidth Broker Signaling (SIBBS) • Basic Idea • Simple domain-to-domain reservation request/response protocol to signal changes in the aggregate reservations of globally well-known services • Design Goals • Simple & extensible • Bootstrap wide-area experimentation with QoS-needy apps, while providing hooks for clouds to experiment with various QoS TE approaches • Integrate with end-to-end signaling capabilities of hosts • Status • Protocol draft nearing completion (see: http://qbone.internet2.edu/bb/) • New protocol for now; looking at mapping to COPS Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

9. SIBBS: Basic Reservation Setup Web interface RSVP DIAMETER COPS RYO CLI SNMP COPS Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

10. Virtual Peerings & Core Tunnels SIBBS SIBBS “Virtual” SIBBS peering BB BB Src Dest “Core Tunnel” RTR RTR BB BB BB Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

11. SIBBS Approach to SLSs • Basic approach: • Define services and reservations, but punt on SLSs • Services • Globally specified, standardized, and named • Include many parameters from draft-tequila-sls (scope, traffic envelope, performance guarantees, etc) • Reservations • Identify a service ID • Specify service parameters • SLSs • Each SLS along a reservation path should be regarded as a black box that answers admissions questions and extends reservations across the peerings Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

12. Simple Interdomain Bandwidth Broker Signalling (SIBBS) BB BB BB Control S LS1-2 S LS2-3 H H Data R R R R DS Domain1 DS Domain2 DS Domain3 Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

13. QBone E2E Picture X Kbps of QPS from hither to………..yon CampusA CampusB Campus C CampusD GigaPoPA GigaPoPB Backbone Key Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

14. Abilene TopologyJanuary, 2001 • 47 connectors • 183 participants • 34 connections to 20 peer networks Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

15. Developing International Peerings STARTAP AbilenevBNSESnetDREN NREN NISN CA*net3SURFnetMIRnetNORDUnetREUNA IUCCAPAN CERNSINETSingarenTanet GEMNETRenater NYC (Telehouse 25 Broadway) DFNINFN DANTE CA*net3 SEATTLE AbileneESnetCA*net3AARnetCERNET NYC (60 Hudson) CERN SINET AbileneESnetvBNS JAnetSURFnetNORDUnet LA Abilene CUDIHARNETSINET Singaren Miami ArgentinaBrazilChileColumbia Brazil Courtesy: Linda Winkler, STAR TAP Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

16. Abilene Load Snapshot Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

17. APS Participation • Goal: • Make APS a reference implementation of the QBone architecture • Current Participants • MAGPI (U. Penn) • iCAIR • PSC (Penn State) • OARNet (Ohio State) • NASA EOS • Others in the wings • TF-NGN (through DANTE) • MIRnet • Various other international • ANL • UIUC • DOE Science GRID (peering transit network) Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

18. Initial Engineering Plan (obsolete) Sweetwater Midland Odessa Pecos Measurement Edge Policing Manual Setup EF Core Forwarding EF Edge Forwarding (MDRR) (BB) Shaping (Surveyor + SNMP + HTTP) (“Firehose” CAR) (Whiteboard + CLI) (MDRR) Automated Setup (GTS) Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

19. APS Measurements • Status: • Collecting EF/BE loads and CAR conform/exceed stats • Not currently monitoring IPDV, but Abilene Surveyor nodes now OC-3 connected and operational • Ohio-ITEC hosting APS measurements and QBone-wide whois server • Near Future: • IPDV along edge-to-edge QBone paths • Abilene Surveyor timing improvements • Better NTP • New CDMA timing sources (can't get GPS in Qwest POPs) • Collection of AS-level traffic matrices Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

20. Committed Access Rate (CAR) • Classifies traffic based on certain matching criteria (interface, DSCP, or ACL) and meters it to a leaky bucket traffic profile • Depending on metering result, different actions applied (drop, transmit, set DSCP,…) • Syntax: • rate-limit {input | output} [access-group [rate-limit] acl-index] bps burst-normal burst-maxconform-actionactionexceed-actionaction Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

21. CAR Experience • For the most part, CAR is exactly what the DiffServ doctor ordered • However, there are some limitations… • PPS performance cost • Quirky constraints on token bucket depths • Not easy to do "virtual trunk" style classification • Problem: we want... • But what we get is… "virtual trunk" "firehose" Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

22. Abilene Architecture Limitation: “Porous” Edge Problem • DoQoS problem with current architecture • MDRR (EF forwarding) on interior interfaces easily subverted by unpoliced connectors Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

23. How to “Crisp” the Edge? • Problem: EF requires that all connectors be policed • Solution: • Short term: Stochastically detect illegal EF traffic with NetFlow and/or OCxMon passive monitoring; gigaPoP would be asked to police • Longer term: Wait for E3 edge cards, deploy them aggressively, and police everywhere • Road not taken:Re-write all non-participant traffic with DSCP 000000 (using PIRC hack); need to pass some DSCP values (reason why coming up…) Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

24. Looking Forward: Resource Accounting and Admissions • Problem: how to account for link EF capacity and commitment in routed network? • Potential solution: DiffServ-Aware MPLS-TE • Basic idea: • Edge-to-edge MPLS tunnels • LSP setup constrained by available EF bandwidths • OSPF augmented to carry QoS link state attributes • See draft-lefaucheur-diff-te-reqts • “Solves” DiffServ admissions problem • Does not solve general DiffServ provisioning problem • Status: completed extensive lab test of Cisco’s implementation of this idea; considering field trial Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

25. Current Engineering Plan Sweetwater Midland Odessa Pecos Manual Setup EF Core Forwarding EF Edge Forwarding (MDRR) Automated Setup (SIBBS? + DSTE?) Shaping (Surveyor + SNMP + HTTP + whois + traffic matrices) Measurement Edge Policing (CAR + QPPB) (Whiteboard + CLI) (MDRR) (GTS) Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

26. QPS Status (Good News) • The Good News: • With enough elbow grease, E2E services can be built • DiffServ functionality in most modern routers • Many hosts now support QoS signaling (RSVP) • Numerous testbed trials • Partial implementations in ESnet and Abilene will form nucleus of QBone Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

27. SC2000 Interdomain QoS DemoNovember 6-9, 2000 • Premium service over two wide-area paths • LBNL-ESnet-Abilene-SCinet-Internet2 booth • Stanford-CalREN2-Abilene-SCinet-Internet2 booth • Congestion induced at multiple points • CD-quality interactive audio application shown with/without QoS • SC2000 Network Challenge Winner: Award for "Most Captivating and Best Tuned" Demo Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

28. Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

29. QPS Status (Bad News) • The Bad News: • Router vendor hype/reality mismatch • Deploying QPS “requires upgrading the world” • Low demand for QPS (app/net chicken/egg dynamic) • Elbow grease in short supply in most campus IT shops Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

30. Bottom Line • Progress on QPS will continue to be slow • Internet2 will continue with a QPS work program in 2001… • Deploy incrementally where it counts • Revise/complete architecture (joint work with TF-NGN) • Work to build QoS-sensitive application user communities who understand their needs • .However, QPS will no longer be the only Internet2 QoS activity • Need to reap lower hanging fruit from DS (I2SS) • Much stronger push on measurements/monitoring • Big Internet2 E2E performance initiative ramping up Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

31. E2E Performance and Measurements • Internet2 E2E Performance Initiative • Typical BE TCP throughputs often much less than one would expect • Common problems • Broken TCP stacks • Ethernet auto-negotiation failures • Evolving attributes of initiative • Performance Emergency Response Teams (PERTs) • Sharper tools for measurement, monitoring, and analysis • Measurement arsenal • WEB100 (Mathis et al.) • Open Internet2 measurement architecture (à la QBone) • Open source and protocols for one-way delay measurement • Application-level performance fault-isolation (reflector infrastructure) Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

32. Internet2 Scavenger Service • Basic idea • A lower priority class of best-effort • Voluntary marking hints to network that degraded service is OK (think of a "nice" for the network) • Intended uses • Non-time-critical traffic (e.g. server-to-server NNTP, anonymous FTP, network backups) • Bulk data transfers using TCP • Non-mission applications (e.g. Napster, games, etc) • New kinds of distributed applications that attempt to use idle network capacity Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

33. I2SS Service Specification • Rigorously defining the E2E I2SS is difficult! • Hoping to define relative to best effort • I2SS traffic indicated by DSCP 001000 • Modification of class selector PHB • Note that the I2SS codepoint has global significance • I2SS domain requirements • Traffic leaving must be marked I2SS, if it entered so • Router requirements (all SHOULDs) • Forward I2SS independently giving it a lower probability of timely forwarding OR forward in same manner as BE • Offer I2SS a very small minimum departure rate (unlike BH PDB) • Offer I2SS all un-used bandwidth Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

34. For more information... • Internet2 Home: • http://www.internet2.edu/ • Internet2 QoS Working Group Home: • http://www.internet2.edu/wg/qos/ • QBone Home: • http://qbone.internet2.edu/ • Abilene Premium Service Home: • http://www.internet2.edu/abilene/qos/ Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)

35. Service Specification and TE for the QBone—Amsterdam (January 25th, 2001)