14-760: Advanced Real-World Networks

1. 14-760:Advanced Real-World Networks Designing a Network for Gaming Lecture 21 * Spring 2019 * Kesden • Source: • https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-i • https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-ii • https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-iii

2. General Requirements for the Internet • Requirements: • Cheap • Bandwidth • Not generally required • Low latency • Low jitter

3. Why Isn’t Latency Generally A Concern? • Human scale • We want web pages to load “fast” • But, that allows a timescale of dozens of milliseconds • Maybe up to 100ms • Buffering • Where we care about latency we can often buffer in advance • Consider streaming video • Tricks • Where we can’t buffer, we can often times make it good enough • Use compression and send multiple copies, using first one to arrive

4. Why Is Latency A Concern for Gaming? • Can’t buffer moves that haven’t been played yet • If different players see different universes, they aren’t playing the same game • If the universe updates faster for some players than others, they have an advantage • Latency ruins the interaction and the enjoyment of playing

5. Latency Example

6. Imposing Latency: Backbone • Backbone is not designed for low latency • Packets are routed according to minimize financial cost • Contracts among collaborating competitors

7. Imposing Latency: Routers • Process packets on a per packet basis • Not per byte • Favors bulk senders with larger packets • More overhead per packet (headers, etc) , fills buffers faster • Games don’t work like streaming • Many small updates • Not large blacks of data

8. Imposing Latency: UDP vs TCP • UDP isn’t typically resent when lost • Losing it imposes no future cost on network • TCP resends when lost • If not sent now, it’ll come back and waste bandwidth again later • Consequence: • Many routers drop UDP in favor of TCP • Gaming: • Usually uses UDP for many things • No reason to resend old updates when new ones supersede them. • TCP HOL blocking for resends can back up the whole pipeline, not just single update

9. Now Back to that BGP Backbone thing • Crazy route means more routers to drop gaming traffic

10. Problems (And Solutions) Not Unique to Gaming • Some organization buy private connectivity • Leased lines between data centers or campuses • Google Fiber • Etc

11. Riot Games Solution: Hardware (Easy Part) • Buy routers • Least co-location space • Deploy routers • Can implement specially tuned policies • Peer with ISPs that are also co-located • Remember that peering is directly connecting • Avoid sending data upstream and then back down stream • Avoid higher up and backbone • Different, negotiated cost model • Obviously takes a lot of agreements and contracts • Not just plugging in

12. Riot Games Solution: Connectivity • Co-locating and peering provides good access to gamers • ISPs local presence is close to their customers, network wise. • But, how to connect various points? • Lease portions of bandwidth along existing routs • Lease “dark fiber” bringing online new capacity • Each connection is different • Just like two roads that connect two points • Different lengths and characteristics • Different numbers of routers, repeaters, etc.

13. Can’t Just Contract with Provider? • ISPs don’t specialize in this • Different geographic concentration by provider • Not necessarily the business relationships (with competitors) to optimize across boundaries • Have to reach all players equally, not benefit those on certain ISPs. • Internal architecture and tuning for bulk traffic, not latency and jitter • Their networks can be fast or cheap, but not both • Adding a customer of sorts doesn’t change this • One customer, e.g. Riot, can’t reasonable pay enough to make whole network fast for everyone • And it isn’t clear that would meet Riot’s needs, e.g. they have competitors • Basically, one can’t contract to buy what doesn’t exist as a product • Someone has to make it

14. How Did it work out? Impact of Peerings

15. How Did it work out? Impact of Server Move

16. Moving Forward: Inside the Box • Routers are products designed for the common problem • Cheap bandwidth • Monitoring at commodity scale • What about optimizing for the small packet traffic, etc? • Want new features, more than minor tuning • Or, what about more responsive monitoring? • Find bad play right now

17. Moving Forward: Inside the Box • Build own routers from commodity computers • Open source routing code • DPDK: Data Plane Development Kit