PMD Auto Load Balancing

1. PMD Auto Load Balancing Nitin Katiyar Pradeep Venkatesan Jan Scheurich

2. Agenda • Background • Auto Load Balancing – Proposal • Example • Summary • Questions

3. Background • Port Rx queues assignment to PMDs is realized • statically by configuration • dynamically - on demand or when certain events occur • Static port queue assignment: • Simple • Works best if traffic patterns are well understood and unchanging

4. Background(contd…) • Dynamic port queue assignment: • triggered by port addition, deletion etc. • triggered by CLI pmd-rxq-rebalance command • configurable assignment algorithm i.e. cycles or roundrobin • can better handle unpredictable traffic patterns but still not adaptive

5. Background(contd…) • Round Robin Assignment • Simple algorithm • Load distribution not uniform • Reassignment may or may not result in better load distribution • Cycles based Assignment • Assignment aims to keep load distribution uniform across PMDs • Each PMD keeps track of the processing cycles spent for each Rx Queue • Six snapshots of the queue processing cycles each spanning 10 seconds are maintained • When reassignment requested, the queue statistics are used to get a uniform distribution

6. Automatic Load Balancing - Proposal • Trigger cycles based assignment automatically (disabled by default) • Maintain additional per-queue statistics to help detect if any PMD is overloaded • Additional configuration parameters to fine tune the behavior • Reassignment will be triggered only if such a reassignment will result in better load distribution

7. Automatic Load Balancing - Proposal Details • Additional per-queue statistics to be maintained • Number of Rx drops for physical port queues • Number of times vhost ports exceeded the qfill threshold • New configuration parameters • Enable/Disable automatic load balancing • PMD Overload Threshold (utilization %) • Whether Q Drops should factor into PMD Overload determination • Minimum improvement in load distribution expected as a result of reassignment

8. Automatic Load Balancing -Proposal Details (contd…) • Modifications to Cycles based reassignment function • A new option termed "dry-run" • Dry-run option will return the poll list resulting from running the algorithm withoutactually performing the reassignment.

9. Automatic Load Balancing - Details (contd…) • A PMD will consider itself overloaded, if for six consecutive 10-second intervals: • the PMD utilization is more than the configured Overload threshold AND • if Q drops are to be factored, number of Rx queue drops or number of times qfill-threshold exceeded is non-zero

10. Automatic Load Balancing - Details (contd…) • OVS Main thread changes: • if any PMD considers itself overloaded, perform a dry-run of the reassignment • if dry-run poll list identical to current poll list, nothing more to do • if dry-run poll list different from current poll list, reassignment may offer benefits

11. Automatic Load Balancing - Details (contd…) • OVS Main thread changes: • Evaluate if the reassignment will meet the configured minimum improvement as follows • Calculate the total processing cycles for each PMD based on current poll list of RX queue • Calculate the average variance in processing cycles for each PMD • Repeat above calculations using dry-run poll list • If dry-run average variance is less, the reassignment will likely result in a more uniform load distribution • Calculate the percentage improvement of the new average variance over the old average variance • Trigger reassignment if the difference is more the configured improvement threshold

12. Automatic Load Balancing - Details (contd…) PMD Thread PMD Thread Record processing cycles and checks if PMD is overloaded Record processing cycles and checks if PMD is overloaded Monitor PMD load Monitor PMD load Main Thread Continue monitoring PMD load Is any of the PMD thread overloaded? No Yes Reload PMD with new “RX queue mapping” Reload PMD with new “RX queue mapping” Initiate PMD dry run Does dry run indicate improvement? No Yes Initiate data path reconfiguration

13. Automatic Load Balancing -Example Initial Configuration • Configured pmd-auto-lb-thresh = 80% • Processing load of PMD1 = 90% (Q0=25%, Q1=30%, Q2=35%) • Processing load of PMD2 = 35% (Q3=10%, Q4=20%, Q5=5%) • Mean load = 62% and Variance = 756

14. Automatic Load Balancing -Example(contd…) PMD Dry Run • Processing load of PMD1 = 60% (Q0=25%, Q1=30%, Q5=5%) • Processing load of PMD2 = 65% (Q2=35%, Q3=10%, Q4=20%) • Mean load = 62% and Variance = 6 • Improvement = ((756 – 6) * 100) / 756 = 99%

15. OVS CLI changes • We propose to have following knobs for Automatic PMD load balancing. • ovs-vsctl set open_vswitch . other_config:pmd-auto-lb="true" • ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-thresh=“<>" • ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-min-improvement=“<>" • ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-drop-check=“<>“

16. RFC Patch • RFC Patch v1 is available in ML: • https://patchwork.ozlabs.org/patch/982444/

17. Future Enhancements • Frequent change in RX queue to PMD assignment can be avoided by adding a dampening logic in main thread. • Time interval and frequency for PMD auto load balancing can be made configurable via CLI. • The threshold for RX Drops for physical ports and RX qfill for vhost user ports can be made configurable via CLI.

18. Summary • PMD auto load balancing feature provides a mechanism to efficiently utilize the PMDs if traffic pattern changes after RX queue assignment to PMDs. • This feature can be enabled/disabled dynamically via OVS CLI and no additional cost in processing when this feature is disabled. • Provides various tunable parameters via OVS CLI to setup automatic load balancing according to the needs.

19. Questions