TM8106 Optical Networking

1. TM8106Optical Networking Energy Consumption in Optical Networks By Ameen Chilwan Syllabus: [1] Energy Efficiency For Network Equipment: Two Steps Beyond Greenwashing, Juniper Networks, Whitepaper, 2009. [2] Network and Telecom Equipment – Energy and Performance Assessment, Test Procedure and Measurement Methodology, ECR Initiative, Draft 2.1.1, October 04, 2009.

2. Paper # 1 Energy Efficiency For Network Equipment: Two Steps Beyond Greenwashing. TM8106 Optical Networking - Green Networking

3. Introduction • Attention drawn to energy footprint of data networking • Due to rising energy costs and environmental standards • Missing verifiable data to support “green” claim • In this paper • Discuss practical aspects of energy efficiency measurement • Define efficiency metric related to equipment energy consumption • Reducing energy-related office & transport expenses • Fairly straight-forward • Reduce the operation cost but has a high initial expenditure • Determines the right point to invest in energy-saving technologies • Current state-of-the-art in network industry • No firm numbers or commitments, but an assorted collection of power management technologies • Not so simple, depends upon many parameters TM8106 Optical Networking - Green Networking

4. Basics of Moving Data • “An arbitrary message can be represented via units of informational entropy” • Modern computing and communication use “bits” for this • Energy consumption in networking equipment • Due to loss during transfer of electric charges • Caused by imperfect conductors and isolators • Depends upon technology, frequency of transitions, no. of gates • These details are not very interesting to end-users • Instead, a list of features required is maintained, containing: • Minimum packet processing requirements • Scaling parameters (bandwidth, queues, other revenue-generating) • Packaging format • Relative cost TM8106 Optical Networking - Green Networking

5. Step 1: Efficiency Criteria • In theory, to compare energy consumption: • Sufficient to put two or more devices under the same load and measure their respective power draw • In reality, this is rarely possible • Requires non-trivial investment in test gear • Different devices have different capabilities => not comparable • To define efficiency metric, we normalize energy consumption (E) by effective full-duplex throughput (T) • , ECR is Energy Consumption Rate [watts/Gbps] • E & T come from tests or vendor data, has to be verifiable • ECR is amount of energy (in Joules) required to move an array of data (in bits) across the device. • ECR is a peak metric • Highest performance capacity of the device TM8106 Optical Networking - Green Networking

6. ECR Metric • Valid differentiator within product class • Equipment with lower ECR use less energy to transfer same amount of data • Unlike light-bulbs, system capacity must be a factor into efficiency estimates • Growing capacity requirement may or may not fit energy budget • Provides unified way of testing • Can be also be used for defining long-term R&D goals • Also mean, specific setting for each customer may not be met • Customers needs 600 Gbps, ECR reported for 1.6 Tbps • Relative ECR standings remain same across wide config range • Assumption that provides compromise between custom and standard settings • Expectation: standard ECR rating will be adequate energy performance estimate TM8106 Optical Networking - Green Networking

7. Decoding Vendor Datasheets – “T” • Parameters E & T to be collected by same methodology • So, even if ECR not reported by vendor should be possible to be approximated from publicly available data (datasheets, tests) • Platform Throughput • Amount of data platform can process per second • Vendors often report platform capacity, not throughput • Capacity is half-duplex bandwidth in Gbps • Has to be divided by two to get “T” • Not always the case • Some vendors report capacity as theoretical peak utilization based on single isolated metric • This neglects limitations like: line cards that can’t use their whole BW, inter- slot restrictions and other limits • For ECR, “T” should include only effective capabilities • Calculated as sum of capabilities of all the line cards in mesh TM8106 Optical Networking - Green Networking

8. Decoding Vendor Datasheets – “E” • Several energy metrics in datasheets • Power System Rating [watts or amps] (aka Agent Label) • Site preparation requirements recommended by vendor • Actual consumption maybe fraction of what power supplies deliver • Maximum Energy Consumption [watts] • Vendor’s estimate for highest configuration footprint • Can serve as an upper boundary estimate for power draw • Max energy consumption estimate changes with replacing modules • Component-based Consumption Estimate [watts] • Adding power draws for configured components to make estimate • Arithmetic sum of components ratings never represents real system • Typical (Average) Power Draw [watts] • Vendors are free to report this metric with underpowered config, load profiles that yield best results and so on • Without disclosure of conditions, can’t be used for comparison TM8106 Optical Networking - Green Networking

9. Advanced Topics • ECR represents performance of fully-configured system running at maximum load • Platform Modularity (Partial configuration) • Energy consumption of modular router/switch is sum of: • Fixed parts “F” (chassis, host system, fabric, clocking) • Variable parts “V” (line cards, ports, physical line drivers) • E = F + V • More efficient in full config are more efficient in partial config too • If not, cross-over point can be found • Absolute Energy Footprint • Relative standings not changed in partial config, but interesting to know how much energy a platform will consume • Component-based energy calculators provided by vendors • ECR method can be adapted to collect and report realistic component footprints in addition to “ratings” TM8106 Optical Networking - Green Networking

10. Advanced Topics (continued…) • Energy Efficiency and Time Scales • When network not fully loaded, it has on- and off- peak periods • Dynamic power management require measuring energy consumption under variable load • Thus ECR is complemented with Energy Efficiency Rate (EER) • , Ef=full load, Eh=half load, Ei=idle • EER is synthetic and stimulates in area of Barroso’s principle of load proportionality • Any elasticity in energy consumption should not involve pkt drop • Any state change should happen automatically and in real-time • Extended Idle conditions may happen (weekends, day/night cycle) • Maybe useful a device state transition to standby with reduced capacity, i.e. small energy footprint • Transition between active and standby required not to be instant TM8106 Optical Networking - Green Networking

11. Advanced Topics (continued…) • Metric Accuracy and Practical Impact • Accuracy of ECR and EER reflects averaging and fluctuations • Averaging relates to sampling during test runs • Fluctuations relates to technological deviations • suggested accuracy within range ± 2.5% • Practical ECR/EER advantage can be within 10% or higher • Two ways to express ECR diversity • Monetary difference: related to cost of ownership • A datacenter consumes 100 kW for 5 years, ECR difference of 10% is 438,300 kWh (24*365.25*5) excluding cooling & power conversion • Consider fully burdened consumption, 1.5-3 times of input, 2x overhead, a cost of $ 0.10 and 10% yearly rate increase will give $107,000 for 10% ECR improvement, 200k for 20% and 500k for 50% • Environmental Impact: following Kyoto Protocol (CO2 reduction) • ECR forms basis for equipment selection criteria • Choosing equipment with 10% ECR advantage can mean 1 year advantage TM8106 Optical Networking - Green Networking

12. Step 2: Design Goals • Energy related improvements in equipment design are: • Organic (passive) • Inline with Dennard’s scaling law: every new generation of network silicon packs more performance in smaller energy budget • Engineered: active energy management, including: • Idle state logic, gate count optimization, memory access algorithm • Some enhancements of both are in step with building networks • E.g. better density, integration and heat management • On the other hand, dynamic power management proportionate to an instant load is a technology with no effect on e.g. capacity • But such technologies form a pool that may improve energy efficiency at pace even faster than Moore’s Law • Return on Investment is not always material • Being a pioneer is also expensive and challenging TM8106 Optical Networking - Green Networking

13. Step 2: Design Goals (continued…) • Network Design • Guidelines for improved results during network design • Select equipment with best ECR/EER rating • Plan for good system & network utilization • Include rack space, cooling and power conversion in assessment • Do not multiply entities beyond necessity • Consider local energy costs and trends • Conclusion • Modern networks growing faster than Moore’s Law • Offset by reduction in areas like commuting, offline shopping, offline banking and goods manufacturing • Net-centric world: better life quality with less material footprint • So, need more environmental friendly equipment to build networks • Two much-needed steps are: standardize network efficiency, and rise of sustainable network technologies TM8106 Optical Networking - Green Networking

14. Paper # 2 Network and Telecom Equipment – Energy and Performance Assessment, Test Procedure and Measurement Methodology. TM8106 Optical Networking - Green Networking

15. Beginning • Purpose • Framework for first-order energy efficiency approximation for • Packet-based network • Telecom equipment • Covers • Peak efficiency, Variable-load efficiency and Idle (statically configurable) energy efficiency • Theoretical Basis • Energy Efficiency = Energy Consumption / Effective Throughput • Basic energy efficiency test: measurement alongside EER • Amount of actual data equal or less than theoretical load • Additional tests for properties like: • Idle (static) energy management, energy management for connected (cascaded) devices and embedded energy monitoring TM8106 Optical Networking - Green Networking

16. Scope • Efficiency definition suited • For medium to large scale network systems • Less for small office, home, consumer grade devices • TP less relevant • Efficiency metrics can be based on allowances per unit functionality • Tests in this paper applicable to • Core & edge routers, L2/L3 switches, optical packet shelves etc • Anything with performance numbers less than face value of connected ports • Can also be adopted for • equipment with face value of ports same as effective bandwidth (TDM systems, optical converters) • Bandwidth is not a measure of load (control planes, AUTH servers) • Other metric to measure load, e.g. CPU Utilization TM8106 Optical Networking - Green Networking

17. Test Procedure 1 Energy consumption in relation to dynamically changing load TM8106 Optical Networking - Green Networking

18. Test Procedure 1 • Packet-based systems based on statistical MUXing • May or may not correspond to theoretical BW as on port face • Simultaneous performance and energy consumption measurements • Under load profile and conditions typical to environment where system under test (SUT) is intended operate • Class-specific requirements in Appendix B • SUT Preparation • Configured according to class requirements • Exposed to conditions as in App. A to settle potential temperature and humidity differences prior to test • Router testing equipment: to simulate load and collect data • AC or DC inline meters to calculate energy consumption TM8106 Optical Networking - Green Networking

19. Test 1 – Steps • Step 1 (Qualification) • Determine maximum Load (Lmax) sustain SUT at zero pkt loss • Any methodology can be used: binary search, heuristics etc • No time limit for this run • The following runs separated with idle time ≤ 300sec • Step 2 (full load) • Offer Lmax to SUT for 1200 sec, avg consumption E100 found • Step 3 (half load) (can be automated, reset without pkt loss) • Reduce load to Lhalf (=0.5 x Lmax)and run for 1200 sec • Calculate E50 which is avg energy consumption in this period • Load reduced by reducing pkt rate on all configured ports • Not by idling or disconnecting ports • Packet loss resets the test (step1), reset to Lmax should be possible TM8106 Optical Networking - Green Networking

20. Test 1 – Steps (continued…) • Step 4 (30% load) (can be automated, reset without pkt loss) • Reduce load to L30 (=0.3 x Lmax)and run for 1200 sec • Calculate E30 which is avg energy consumption in this period • Load reduced by reducing pkt rate on all configured ports • Not by idling or disconnecting ports • Packet loss resets the test, reset to any load should be possible • Step 5 (10% load) (can be automated, reset without pkt loss) • Reduce load to L10 (=0.1 x Lmax)and run for 1200 sec • Calculate E10 which is avg energy consumption in this period • Load reduced by reducing pkt rate on all configured ports • Not by idling or disconnecting ports • Packet loss resets the test, reset to any load should be possible TM8106 Optical Networking - Green Networking

21. Test 1 – Steps (continued…) • Step 6 (idle run) • Remove all the loadand run for 1200 sec • Calculate Ei which is avg energy consumption in this period • Load removal by idling pkt rate on all configured ports • Not by disconnecting or shutting down ports • Packet loss resets the test, reset to any load should be possible TM8106 Optical Networking - Green Networking

22. Test Procedure 2 (Optional) Energy consumption in relation to statically changing load TM8106 Optical Networking - Green Networking

23. Test Procedure 2 • Due to extended periods of low utilization • Demonstrate energy saving potential • Pre-requisite is to have Lmax from test 1 • Configuration, preparation, requirement same as test 1 • The following runs separated with idle time ≤ 300sec • Steps • Step 1 (half ports in use) • Reduce to Lhalf (= 0.5 x Lmax) and run for 1200 sec • Measure energy for entire period and find avg P50 • Load reduction by sending traffic at full rate to every second port • Active ports should be evenly mixed with inactive • Block designation (1 line card with all active, other none) not allowed • Packet loss resets to test 1, reset to any higher level not required TM8106 Optical Networking - Green Networking

24. Test 2 (continued…) • Steps • Step 2 (one quarter of all ports in use) • Reduce to L25 (= 0.25 x Lmax) and run for 1200 sec • Measure energy for entire period and find avg P25 • Load reduction by sending traffic at full rate to every fourth port • Active ports should be evenly mixed with inactive • Block designation (1 line card with all active, other none) not allowed • Packet loss resets to test 1, reset to any higher level not required • Step 3 (one-tenth ports in use) • Reduce to L10 (= 0.1 x Lmax) and run for 1200 sec • Measure energy for entire period and find avg P10 • Load reduction by sending traffic at full rate to every tenth port • Active ports should be evenly mixed with inactive • Block designation (1 line card with all active, other none) not allowed • Packet loss resets to test 1, reset to any higher level not required TM8106 Optical Networking - Green Networking

25. Test Procedure 3 (Optional) Component level energy footprint (Fc) TM8106 Optical Networking - Green Networking

26. Test Procedure 3 • A number of configurations for networks are possible • Impractical to test all of them • Perform detailed energy and performance analysis for some • Approximate for the rest of configurations by sum of component-level energy budgets • Less precise compared to actual measurements • Trade-off between operational cost and precise metric values • Vendor builds and publishes • Library of component-level energy consumption • For power budget estimations • Suitable for arbitrary configurations • Detailed measurements not applicable • Concentrates on peak energy performance only TM8106 Optical Networking - Green Networking

27. Test 3 – Steps • SUT Preparation • Configured according to class requirements including CUT • SUT neither required to be fully loaded nor to bear all modules • CUT that energy and traffic impact of CUT is minimum • Step 1 (Qualification) • Determine maximum Load (Lmax+) sustain SUT at zero pkt loss • Design Lmax+ to exercise all components to highest possible TP • Any methodology can be used: binary search, heuristics etc • No time limit for this run until max load is determined • The following runs separated with idle time ≤ 300sec • Step 2 (full load) • Offer Lmax+ to SUT for 1200 sec, measure energy for the period • average footprint F+ is calculated • Packet loss resets the test to step 1 TM8106 Optical Networking - Green Networking

28. Test 3 – Steps (continued…) • Step 3 (CUT Removal) • Remove CUT and readjust SUT to compensate • Determine new max load (Lmax-)sustain SUT at zero pkt loss • Design Lmax- to exercise all components to highest possible TP • Step 4 (final measurement) • Offer Lmax- to SUT for 1200 sec, measure energy for the period • average footprint F- is calculated • Packet loss resets the test to step 1 • Energy footprint of CUT is approximated • Fc = F+ - F- • Steps 1-4 repeated as many times as needed • Not applicable for non-redundant components • Only for some components as system chassis and backplane TM8106 Optical Networking - Green Networking

29. Test Procedure 4 (Optional Functional Test) Embedded energy monitoring capabilities TM8106 Optical Networking - Green Networking

30. Test Procedure 4 • Real-time energy consumption estimates • pivotal to design and monitor energy conversation policies • Device-level granularity energy utilization reports • not provided by power facilities • Networks to monitor their own energy consumption • by using sensors and embedded probes • Useful to report in parallel with system utilization • Allows for traffic affinity and energy pattern analysis • This test can be done in parallel with • Test 1 and test 2 TM8106 Optical Networking - Green Networking

31. Test 4 – Steps • Step 1 (Qualification) (can be combinedwith test1 step1) • Same as for test1 step1, Lmax is found • Step 2 (full load)(can be combined with test1 step2) • Offer Lmax to SUT for 1200 sec, avg consumption E100found • Average recorded energy consumption for the period R100 is read from the SUT • Optional: Avg system utilization is recorded U100 • Step 3 (half load)(can be combined with test2 step1) • Reduce to Lhalf(= 0.5 x Lmax) and run for 1200 sec • Measure energy for entire period and find avgP50, R50 and U50 • Load reduction by sending traffic at full rate to every 2ndport • Active ports should be evenly mixed with inactive • Block designation not allowed • Packet loss resets to test 1, reset to any higher level not required TM8106 Optical Networking - Green Networking

32. Test 4 – Steps (continued...) • Step 4 (25% load)(can be combined with test2 step2) • Reduce to L25 (= 0.25 x Lmax) and run for 1200 sec • Measure energy for entire period and find avgP25, R25 and U25 • Load reduction by sending traffic at full rate to every 4thport • Active ports should be evenly mixed with inactive • Block designation not allowed • Packet loss resets to test 1, reset to any higher level not required • Step 5 (1/10th ports in use)(can be combined with test2 step3) • Reduce to L10 (= 0.25 x Lmax) and run for 1200 sec • Measure energy for entire period and find avgP10, R10 and U10 • Load reduction by sending traffic at full rate to every 10thport • Active ports should be evenly mixed with inactive • Block designation not allowed • Packet loss resets to test 1, reset to any higher level not required TM8106 Optical Networking - Green Networking

33. Test Procedure 5 (Optional Functional Test) Collateral energy management TM8106 Optical Networking - Green Networking

34. Test Procedure 5 • Devices can have impact on energy consumption in connected (cascaded) devices • Examples include: • Ability to control power states in Power over Ethernet (PoE) • Ability to assist power states in LAN-connected devices • Ability to control IP and non-IP devices via remote control mechanisms TM8106 Optical Networking - Green Networking

35. Effective TP Calculation • Lmax converted into full-duplex TP Tf (in Gbps) • To normalize SUT energy consumption to performance • First method: • Lmax reported by traffic generator as a combination of egress packet-per-second rate and packet size compared to the load. • For variable pkt size: average proportion is computed • All applicable minimum L2 & L1 overhead added to compute effective wire rate • Example: • SUT is eth switch, 10 x 10 GbE ports @ 7,291,702 fps, 64 B/frame • Implies 7,291,702 pps • Tf = 10 x 7,291,702 x 8 x (64 + 1 + 7 + 12) = 49.000237440 Gbps • (1 + 7 + 12): accounts for eth start of frame, preamble, min interpkt gap TM8106 Optical Networking - Green Networking

36. Effective TP Calculation (continued…) • Second method: • Lmax reported tester equipment itself as highest achieved utlization on per-port basis (in percentage) • Line rates multiplied by port utilization to get final data rate • Example: • SUT is MPLS edge platform, 10x10 GbE ports each on core & access • Frames forwarded towards core with 100% utilization • Frames forwarded towards access with 99.22% utilization • Data rate for 10 GbE is 10,000 Mbps • Tf = 10 x 10,000 x 1.0 + 10 x 10,000 x 0.9922 = 199.22 Gbps • Example: • SUT is eth switch with 8 GbE ports opearting at • 100% line utilization when configured for VLAN • 90% line utilization when not configured for VLAN • Result from 2nd case is used as eth switches don’t need VLAN headers TM8106 Optical Networking - Green Networking

37. Results Representation • Array of results obtained from test 1-4 form energy “passport” of the SUT and can be used for evaluation and energy planning purposes • Base Metrics and Device Comparisons • Most straight-forward metric is to normalize energy consumption to the highest sustained throughput recorded • [W/Gbps] • Unlikely to obtain efficiency numbers over sustained intervals • Due to off- (low utilization) and on- (burst) peak periods that cause provider to size on the higher end of traffic profile and lose energy efficiency during off-peak times • Instead, providers can optimize efficiency using middle of their load band • [W/Gbps] • ECR-VL is the measure of dynamic energy management • ECR-VL rating should be close to ECR following Barroso’s principle TM8106 Optical Networking - Green Networking

38. Energy Bill Estimates • Cost of operation over projected lifetime • N is projected no. of years, CkWhj is cost of kWh in year of operation • This method gives estimate based on automatic (variable-load) energy management capabilities of reference system • Modest, but provide lossless operation under all conditions • Alternative cost estimate • Build upon static (idle-load) energy saving capabilities of device • Example: a switch downgraded to 25% at night and 10% on Sun. • Both systems are complementary & work on different time intervals • Relative slow configuration change, not applicable to burstyenv. • Requires personnel to take care of policy-driven degradation TM8106 Optical Networking - Green Networking

39. Site Planning • Providers use Agency Labels provided by vendors • Ill-suited because call for energy reservation at highest end • Makes it confusing whether device will work in case on-site power rating is less that this value • Useful resources for site planning • Value E100 • describes average SUT energy consumption under highest possible load • Provides upper boundary for energy requirements • If E100 not provided • can be approximated from sum of required components • Provided that component footprint Fci were published TM8106 Optical Networking - Green Networking

40. Functional Compliance • Tests 4&5 are intended for functional compliance testing • Only pass/fail criteria • In test 4, externally measured consumption values (E100, P50, P25, P10) should be compared to (R100, R50, R25, R10) reported by SUT • Adjusted result should be in acceptable range of effective consumption (±10% or less) • Upon vendor’s approval an offset R can be added to compensate electronics not covered by embedded power monitoring circuitry • Externally recorded load L should correlate to utilization U • Where Lmax = 100% SUT Utilization • Static offset can also be added in this case, upon vendor’s approval • Knowledge of system w.r.t time crucial for planning static energy management routines • Test 5 describes value-added functionality that might be in SUT TM8106 Optical Networking - Green Networking

41. Reporting Format • Results shall be fully reproducible by any laboratory • Thus has to include at least the following documentation • All SUT software versions, hardware board revisions and device configurations used during the test. • All commands applied for static config to SUT or run-time queries • Traffic generator tool passports, actual voltage in power feeds and ambient (environmental) conditions at test side • The test setup should fully described, including topology, the choice of offered load structure and test actions within a range of possible choices. TM8106 Optical Networking - Green Networking

42. TM8106 Optical Networking - Green Networking