1 / 13

Advancing Metrics on the Standards Track

Advancing Metrics on the Standards Track. Lab Testing Results draft-morton-ippm-advance-metrics-01 Al Morton July 2010 ACK: Len Ciavattone for NetProbe. Outline. Definition-centric metric advancement Possible Report Template Supports the Protocol Action Request

aiko-perez
Download Presentation

Advancing Metrics on the Standards Track

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Advancing Metrics on the Standards Track Lab Testing Results draft-morton-ippm-advance-metrics-01 Al Morton July 2010 ACK: Len Ciavattone for NetProbe

  2. Outline • Definition-centric metric advancement • Possible Report Template • Supports the Protocol Action Request • Examples of the Definition-centric approach: • Revised Lab test methods • Results for one implementation • Open call for participation in Lab tests

  3. Definition-Centric Process (revised) ,---. / \ ( Start ) \ / Implementations `-+-' +-------+ | /| 1 `. +---+----+ / +-------+ `.-----------+ ,-------. | RFC | / |Check for | ,' was RFC `. YES | | / |Equivalence..... clause x -------+ | |/ +-------+ |under | `. clear? ,' | | Metric \.....| 2 ....relevant | `---+---' +----+---+ | Metric |\ +-------+ |identical | No | |Report | | Metric | \ |network | +--+----+ |results+| | ... | \ |conditions | |Modify | |Advance | | | \ +-------+ | | |Spec +----+RFC | +--------+ \| n |.'+-----------+ +-------+ |request | +-------+ +--------+

  4. Key Points (the sub-points) • Start with an RFC • Focus on a specific clause • Run test(s) with Implementations • Test plan is customized to a specific clause • Evaluate Measurements & Compare • Expected measurement results are Clear • Obvious place to take action if text is found to be ambiguous • Final state is Report Dev. for Protocol Action Req.

  5. Test Set-up (more details in I-D) negotiated 100baseTx-FD NetProbe MS-1 10.201.0.1 Network Emulator eth2 10.201.0.254 eth3 10.202.0.254 NetProbe MS-2 10.202.0.1 10.203.0.1 NetProbe MS-3 eth4 10.203.0.254 eth5 10.204.0.254 10.204.0.1 NetProbe MS-4 Test LANs Management LAN

  6. NetProbe 5.8.5 • Runs on Solaris (and Linux, occasionally) • Pre-dates *WAMP, functionally similar • Software-based packet generator • Provides performance measurements including Loss, Delay, PDV, Reordering, Duplication, burst loss, etc. in post-processing on stored packet records

  7. Report Template – Loss Threshold • 3.1. One-way Delay, Loss threshold, RFC 2679 (procedure) • 3.1.1. NetProbe Lab results for Loss Threshold • 3.1.2. XXX Lab Results for Loss Threshold • <your compliant implementation here> • 3.1.3. Conclusions on Lab Results for Loss Threshold

  8. Section 3.1 – Loss Threshold • See Section 3.5 of [RFC2679], 3rd bullet point and also Section 3.8.2 of [RFC2679]. • 1. configure a path with 1 sec one-way constant delay • 2. measure (average) one-way delay with 2 or more implementations, using identical waiting time thresholds for loss set at 2 seconds • 3. configure the path with 3 sec one-way delay (or change the delay while test is in progress, measurements in step 2) • 4. repeat measurements • 5. observe that the increase measured in step 4 caused all packets to be declared lost, and that all packets that arrive successfully in step 2 are assigned a valid one-way delay. Results: 22 of 38 packets were declared lost (post-process).

  9. Section 3.2: First-bit to Last-bit See Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330]. • 1. configure a path with 1000 ms one-way constant delay, and ideally including a low-speed link (10-baseT, FD) • 2. measure (average) one-way delay with 2 or more implementations, using identical options and equal size small packets (e.g., 32 octet IP payload) • 3. maintain the same path with 1000 ms one-way delay • 4. measure (average) one-way delay with 2 or more implementations, using identical options and equal size large packets (e.g., 1400 octet IP payload) • 5. observe that the increase measured in steps 2 and 4 is equivalent to the increase in ms expected due to the larger serialization time for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.

  10. 3.2: First-bit to Last-bit, NetProbe 1400 byte payload 32 byte payload Delay for each mode (one mode) Delay Diff Expected Diff microseconds microseconds microseconds microseconds 1001621 1000356 1265 1094.4 1002735 1000356 2379 1094.4

  11. 3.2: First-bit to Last-bit, NetProbe • Bi-modal delay behavior in the Network Emulator for UDP payload 96 octets & up • Same for Poisson, Periodic, 1 sec, 10ms spacing • Not evident in tests on the Management LAN • Delay increased with larger payload, but more than expected (170 us for low mode) • Suggestions to investigate further are welcome Tests on Management LAN, 100baseTx-FD 1400 byte payload 32 byte payload Ave Delay Ave Delay Diff Expected Diff microseconds microseconds microseconds microseconds 443.37 143.57 299.8 109.44 (190.36)

  12. Other Examples • One-way Delay, RFC 2679 • This test is intended to evaluate measurements in sections 3 and 4 of [RFC2679]. Average delays before/after 2 second increase Average pre-increase delay, microseconds 1000276.6 Average post 2s additional, microseconds 3000282.6 Difference (should be ~= Y = 2s) 2000006 • Error Calibration, RFC 2679 • This is a simple check to determine if an implementation reports the error calibration as required in Section 4.8 of [RFC2679].

  13. Summary • Key clauses of RFC 2679 evaluated in Lab • One Implementation: NetProbe • Other volunteers? • Comments on approach to reporting? • draft-morton-ippm-advance-metrics-01 • What other RFCs should we target first?

More Related