Masking Failures from Application Performance in Data Center Networks with Shareable Backup

1. Masking Failures from Application Performance in Data Center Networks with Shareable Backup DingmingWu+,Yiting Xia+*, XiaoyeStevenSun+, XinSunny Huang+,SimbarasheDzinamarira+, T. S. Eugene Ng+ +Rice University, *Facebook, Inc.

2. Data Center Network Should be Reliable but…

3. NetworkFailuresareDisruptive • Median case of failures: 10% less traffic delivered • Worst 20% of failures: 40% less traffic delivered Gill et al. SIGCOMM 2011

4. Today’sFailureHandling---Rerouting • Fast local rerouting  inflatedpathlength • Global optimal rerouting  highlatencyofroutesupdates • Impact flows not traveling trough the failure location

5. Impact on Coflow Completion Time (CCT) • Facebookcoflowtrace • k=16Fat-treenetwork • Globaloptimal rerouting

6. DoWeHaveOther Options? • Restores network capacity immediately after failure • Be cost efficient • --Small pool of backup switch • How do we achieve that?

7. Circuit Switches • Physicallayerdevice • Circuitcontrolledbysoftware C A • Examples • --optical 2D-MEMS switch, 40us, $10 per-port cost • --electrical cross-point switch, 70ns, $3 per-port cost B D

8. IdealArchitecture Circuit Switch … … … BackupSwitch Servers Regularswitches • Entirenetworksharesonebackupswitch • Unreasonablehighport-countofcircuitswitch • Replaceanyfailedswitchwhennecessary • Singlepointoffailure

9. How to Make It Practical • Feasibility • -small port-count circuit switches • Scalability • -partition network into failure groups • -distribute circuit switches across the network • Low cost • -small backup pool • -share backup switches per failure groups

10. ShareBackupArchitecture AnoriginalFat-treewith k=6 • Partitiontheswitchesintofailuregroups;eachwithk/2switches. Corelayer • Addbackupswitchesperfailuregroups Agg.layer Edgelayer

11. EdgeLayer Edge switches Backup Switch 0 1 2 Circuit switches 1 0 2 0 2 1 Servers i

12. AggregationLayer Backup switch Agg. switches 0 1 2 ? 1 0 2 1 2 Circuit switches 0 1 2 0 1 2 0 ? Edge switches Backup switch 0 1 2

13. Core Layer Core switches 0 3 6 1 4 7 2 5 8 Circuit switches Aggregation switches Backup switch 0 1 2 0 1 2 0 1 2

14. Recover First, Diagnose Later • FailureRecovery • --switchfailurereplacedbybackupsviacircuitreconfiguration • --linkfailureswitchesonbothsidearereplaced • Automatic failure diagnosis performed offline • -details in the paper

15. Live Impersonation of Failed Switch Backup switch Edge switches 0 1 2 Routing Table of Every Edge Switch Routing Table 0 VLAN 0 Routing Table 1 VLAN 1 Routing Table 2 VLAN 2 Servers

16. Live Impersonation of Failed Switch Backup switch Edge switches 0 1 2 Routing Table of Every Edge Switch Routing Table 0 VLAN 0 Routing Table 1 VLAN 1 Routing Table 2 VLAN 2 0 Servers

17. Live Impersonation of Failed Switch Edge switches Backup switch 0 1 2 Routing Table of Every Edge Switch Routing Table 0 VLAN 0 Routing Table 1 VLAN 1 Routing Table 2 VLAN 2 0 Servers

18. What does control system do? • Collects keep-alive messages & link status reports from switches • Reconfigures circuit switches under failures • Performs offline failure diagnosis • Implications • -needs to talk to many circuit switches and packet switches • -keeps a large amount of states of circuit/switch/link status

19. DistributedControl System • Onecontrollerforafailuregroupofk/2switches --configuresthecircuitswitchesadjacent toswitchesinthegroup • Maintainsonlylocalcircuitconfigurationsinitsgroup • --doesnotsharestateswithothercontrollers • Talkstocircuitswitchesusinganout-of-bandcontrolnetwork

20. Summary • FastFailureRecovery • --asfastastheunderlyingcircuitswitchingtechnology • LiveImpersonation • --Traffic is redirected to the backups in physical layer • --Switchesinafailuregrouphavesameroutingtables,useVLANidfordifferentiation • --Regular switches recovered from failures become backup switchesthemselves Fastfailurerecovery,nopathdilation,noroutingdisturbance

21. Evaluation • Bandwidth Advantage • --Iperf throughput on testbed • Application performance • --MapReduce job completion time

22. Bandwidth Advantage • 4racks,8 servers,12switches • 8 iPerf flows saturate the network core ShareBackup restores network to full capacity regardlessoffailurelocations

23. Application Performance 1.2X MapReduce Sort w/ 100GB input data 4.2X ShareBackup preservesapplicationperformanceunderfailures!

24. ExtraCost • Smallport-countcircuitswitches---veryinexpensive • --e.g.$3per-portcostforcross-pointswitches • Smallbackupswitchpool • --1backupperfailuregroupisusuallyenough • --k = 48 fat-tree with 27648 servers ~6.7%extranetworkcost • Partialdeployment • --failuresmoredestructiveatedgelayer • --employbackuponlyforToRfailures

25. Conclusion • ShareBackup:anarchitecturalsolutionforfailurerecoveryinDCNs • --usescircuitswitchingforfastfailover • --is aneconomicalapproachofusingbackupsinnetworks • --preservesapplicationperformanceunderfailures • Keytakeaways: • --reroutingisnotthe only approach forfailurerecovery • --fast,transparentfailurerecoveryispossiblethroughcarefulbackupplacements&fastcircuitswitching

26. Backup---ControlSystemFailures • Circuitswitchsoftwarefailure/controlchannelfailure • --circuitswitchesbecomeunresponsive • --keepexistingcircuitconfigurations,dataplaneisnotimpacted • --fallbacktorerouting • Hardware/powerfailure • --controllerwillreceivelotsfailurereportsinashorttime • --callforhumanintervention • Controllerfailure • --state replication on shadow controllers

27. Backup---Offline Failure Diagnosis 0 0 0 Aggregation switch ? ? • Recycle healthy switch - Only one switch has failed - Back to normal after reboot • Chain up circuit switches using side ports Circuit switches 0 0 0 ? ? Edge switches 17

28. Backup---Offline Failure Diagnosis 0 0 0 Aggregation switch Circuit switches 0 0 0 Edge switches 18