Advancing Metrics on the Standards Track

1. Advancing Metrics on the Standards Track Equivalence Procedure & Lab Results draft-morton-ippm-advance-metrics-02 Al Morton Nov 2010 ACK: Len Ciavattone for NetProbe

2. Outline • Continue Definition-centric metric advancement • Procedure to determine Threshold of Equivalence • Key substitution for Interoperability • Possible Report Template with Procedures • Supports the Protocol Action Request • Open call for participation in Lab tests

3. Definition-Centric Process (revised) ,---. / \ ( Start ) \ / Implementations `-+-' +-------+ | /| 1 `. +---+----+ / +-------+ `.-----------+ ,-------. | RFC | / |Check for | ,' was RFC `. YES | | / |Equivalence..... clause x -------+ | |/ +-------+ |under | `. clear? ,' | | Metric \.....| 2 ....relevant | `---+---' +----+---+ | Metric |\ +-------+ |identical | No | |Report | | Metric | \ |network | +--+----+ |results+| | ... | \ |conditions | |Modify | |Advance | | | \ +-------+ | | |Spec +----+RFC | +--------+ \| n |.'+-----------+ +-------+ |request?| +-------+ +--------+

4. Key Points (the sub-points) • Start with an RFC • Focus on a specific clause • Run test(s) with Implementations • Test plan is customized to a specific clause • Evaluate Measurements & Compare • Expected measurement results are Clear • Obvious place to take action if text is found to be ambiguous • Final state is Report Dev. for Protocol Action Req.

5. Setting the Equivalence Threshold • Proposal – part of the allowable error (or variability) is bothtest configuration and implementation dependent • Maximum_Error = Same_Implementation_Error + Systematic_Error • Only Systematic_Error need be agreed a priori • Systematic_Error should be set such that unexpected implementation differences are detectable <<= KEY • At least, the variability in results from the Same Implementation should be allowable.

6. Section 3.4 One-way Delay, ADK Sample Metric (Lab) • 1. Configure a path with X ms one-way constant delay. • 2. Measure a sample of one-way delay singletons with 2 or more implementations, using identical options. • 3. Measure a sample of one-way delay singletons with additional instances of the *same* implementations, using identical options, • noting that connectivity differences MUST be the same as for the *cross* implementation testing.

7. Test Set-up (6 mcast streams from MS-1 to all others, OW meas at MS-2) negotiated 100baseTx-FD NetProbe MS-1 10.201.0.1 Network Emulator eth2 10.201.0.254 eth3 10.202.0.254 NetProbe MS-2 10.202.0.1 10.203.0.1 NetProbe MS-3 eth4 10.203.0.254 eth5 10.204.0.254 10.204.0.1 NetProbe MS-4 Test LANs Management LAN

8. Section 3.4 One-way Delay, ADK Sample Metric (Lab) --- Results Agate Statistical Analysis Program

9. Section 3.4 One-way Delay, ADK Sample Metric (Lab) --- Results

10. One-way Delay, ADK Sample Metric (Lab) --- Normalized Results

11. Notes on Network Emulator Correlated Loss Generation • Many emulators use this relationship: Corr_value = Last_value * corr_coeff + New_value * (1-corr_coeff) • Works adequately for Delay • Tests with NIST Net indicate some shortcomings • Possible that this function was lost in revisions • Also Possible that our installation has this shortcoming • Investigating these possibilities and alternative, yet simple correlation functions

12. 3.2: First-bit to Last-bit, NetProbe 1400 byte payload 32 byte payload Delay for each mode (one mode) Delay Diff Expected Diff microseconds microseconds microseconds microseconds 1001621 1000356 1265 1094.4 1002735 1000356 2379 1094.4

13. Update on the Bi-modal Delay • Additional investigation appears to conclude that the modal behavior is related to interrupt-to-frame arrival settings of the specific interface board. • Various options appear to be configurable, but only when the interface driver is compiled as a module. • Also, the board/driver does not support the "coalesce" options of ethtool. • Until we can rebuild the Linux machine with this and other planned modifications, confirmation will have to wait.

14. Summary • Key clauses of RFC 2679 evaluated in Lab • Proposal for Equivalence Threshold and Procedure • Network Emulator Testing – Loss Correlation • One Implementation: NetProbe • Other volunteers? • P/O This draft could become basis for the Protocol Action Request for RFC 2679 • draft-morton-ippm-advance-metrics-02 • What other RFCs should we target first?

15. NetProbe 5.8.5 • Runs on Solaris (and Linux, occasionally) • Pre-dates *WAMP, functionally similar • Software-based packet generator • Provides performance measurements including Loss, Delay, PDV, Reordering, Duplication, burst loss, etc. in post-processing on stored packet records

16. Report Template – Loss Threshold • 3.1. One-way Delay, Loss threshold, RFC 2679 (procedure) • 3.1.1. NetProbe Lab results for Loss Threshold • 3.1.2. XXX Lab Results for Loss Threshold • <your compliant implementation here> • 3.1.3. Conclusions on Lab Results for Loss Threshold

17. Section 3.1 – Loss Threshold • See Section 3.5 of [RFC2679], 3rd bullet point and also Section 3.8.2 of [RFC2679]. • 1. configure a path with 1 sec one-way constant delay • 2. measure (average) one-way delay with 2 or more implementations, using identical waiting time thresholds for loss set at 2 seconds • 3. configure the path with 3 sec one-way delay (or change the delay while test is in progress, measurements in step 2) • 4. repeat measurements • 5. observe that the increase measured in step 4 caused all packets to be declared lost, and that all packets that arrive successfully in step 2 are assigned a valid one-way delay. Results: 22 of 38 packets were declared lost (post-process).

18. Section 3.2: First-bit to Last-bit See Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330]. • 1. configure a path with 1000 ms one-way constant delay, and ideally including a low-speed link (10-baseT, FD) • 2. measure (average) one-way delay with 2 or more implementations, using identical options and equal size small packets (e.g., 32 octet IP payload) • 3. maintain the same path with 1000 ms one-way delay • 4. measure (average) one-way delay with 2 or more implementations, using identical options and equal size large packets (e.g., 1400 octet IP payload) • 5. observe that the increase measured in steps 2 and 4 is equivalent to the increase in ms expected due to the larger serialization time for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.

19. 3.2: First-bit to Last-bit, NetProbe • Bi-modal delay behavior in the Network Emulator for UDP payload 96 octets & up • Same for Poisson, Periodic, 1 sec, 10ms spacing • Not evident in tests on the Management LAN • Delay increased with larger payload, but more than expected (170 us for low mode) • Suggestions to investigate further are welcome Tests on Management LAN, 100baseTx-FD 1400 byte payload 32 byte payload Ave Delay Ave Delay Diff Expected Diff microseconds microseconds microseconds microseconds 443.37 143.57 299.8 109.44 (190.36)

20. Other Examples • One-way Delay, RFC 2679 • This test is intended to evaluate measurements in sections 3 and 4 of [RFC2679]. Average delays before/after 2 second increase Average pre-increase delay, microseconds 1000276.6 Average post 2s additional, microseconds 3000282.6 Difference (should be ~= Y = 2s) 2000006 • Error Calibration, RFC 2679 • This is a simple check to determine if an implementation reports the error calibration as required in Section 4.8 of [RFC2679].