NetGrove: New Tools for Network Optimization

1. NetGrove: New Tools for Network Optimization Tomas Vykruta Software Development Engineer XNA Publishing Platform Microsoft

2. Speed of Light Still Constant

3. Latency • Speed of light = 186,282 miles per second • Seattle to L.A. • 960 miles = 5 milliseconds • Seattle to England • 4,799 miles = 26 milliseconds • Speed of light in fiber or copper slows to 60% • Each router adds 5 to 50 ms • DSL or cable modem adds 10 ms

4. Bandwidth on the Rise Thanks, Bungie!

5. Also on the Rise • Player expectations • More players in a session together • More voice, video, and background downloads • Network-using features in your game, such as data-feeds and spectator modes • Network-using features in the platform • Other home devices consuming bandwidth

6. Building a successful title • Optimizing game networking is still very important! • Have a Plan: • Define max bandwidth goal • Monitor bandwidth during development **DO NOT LEAVE UNTIL SUBMISSION**

7. Defining Maximum Bandwidth • Host = worst case scenario • Let’s target top 80% of users • 80% have 175ms or better latency • And 176kb/s down, 226kb/s up or better bandwidth • From “The Real World: Console Latency and Bandwidth on Xbox LIVE” white paper on xds.xbox.com • Subtract reservation (i.e. voice chat) • Divide remaining bandwidth by “max # of players – 1”

8. Bandwidth Calculation 5 player game 68kbps CLIENT 14kbps 17kbps CLIENT 14kbps HOST 56kbps 17kbps 14kbps CLIENT 17kbps 14kbps CLIENT 17kbps

9. Max Bandwidth cont’d • Scenario: theoretical 16 player game with 30 kbps reservation for voice chat • Host is updating 15 players • And receiving data from 15 players • Max Bandwidth = 9 kbps down, 13 kbps up per client max • Also simulate 175 ms latency for playtests • Monitor regularly!

10. Max Bandwidth Worksheet

11. Bandwidth Data 2007-2008 Table 2: 2007 Bandwidth Table 1: 2007 Latencies

12. Latency and Bandwidth Data • 2008 bandwidth stats same as 2007 • Median latency 84 ms • Only 10% of consoles are 32 ms or less • Worst case latency500 ms • 336 kb/s median downstream bandwidth • 352 kb/s median upstream bandwidth • Typical packet loss 2% • Worst case packet loss 10%

13. The Real World • From real-world title network reviews • Best case = 4 kbps per client in AAA FPS title • Worst case = 370 kbps+ for the host in a 4 player game of a different AAA title!

14. From a real network review • “Stopped moving” message sent every frame • Many invisible objects simulated • Entire animation state sent (100 bytes+) • Uncompressed data • Position + Orientation + LookAt sent each frame = 12 + 36 + 36 bytes = 84 bytes • Host not coalescing data • NPCs simulated like players • Many more… • End result after optimizations = 5 kbps per client!

15. Run-time Bandwidth Monitoring • Fps, Memory Usage, why not peak bandwidth? • Display on screen, always present • Visual alert to indicate packet loss

16. Tools • Instrumentationingamecode • Subjectiveplaytestexperience • Packetsniffing • Triedandtrue • Blackboxandlowbarriertoentry • Manysophisticatedsniffersavailable

17. Network Monitor 3.2 • Process Tracking to identify rogue applications • New engine for improved capture rate in high-speed networks • Find conversations to view TCP streams, HTTP flows etc. • Extensive parsers for over 300 protocols! • Network Monitor API: Create your own applications that capture, parse, and analyze network traffic

18. Network Monitor 3.2

19. Quick NetMon Tour • Conversations • Find frame (CTRL+F) • Display and capture filters • Declarative language with Intellisense • Color filters • Ex: XSP (by itself) filters down to Xbox LIVE traffic • NMCAP for command-line captures

20. NetMon Protocols • Extensive script-based protocol parser language • Documented in NetMon\Help\ParserLanguage.doc • Create your own! • Trade with your friends! • Sure would be nice if the LIVE protocol…

21. LIVE Secure Packet Parsing • Xbox 360 SDK and Games for Windows – LIVE SDK both provide a NetMon Parser • XSP.NPL • Automatically installs with XDK • Encrypted data is still encrypted • XNET_STARTUP_DISABLE_PEER_ENCRYPTIONto send unencrypted data while debugging • XSP parser + NetMon filtering + no encryption = see what’s actually happening on the wire

22. DISABLING ENCRYPTION • XNET_STARTUP_BYPASS_SECURITY added in November 2008 • Disables encryption, view unencrypted XSP data in NetGrove and NetMon • Flag must be enabled on all consoles • Silently disabled on RETAIL hardware • Stop using XNET_STARTUP_BYPASS_SECURITY • Disables authentication and XSP completely

23. LIVE Server Traffic Traffic to LIVE Presence Server Traffic to LIVE Statistics / Profile Server

24. PIX – System Monitor PIX provides counters for LIVE server traffic Traffic to LIVE Statistics Server

25. So What Do You Do with This? • Debug pathological network conditions • Excessive or unexpected LIVE traffic (TCR) • Mapping LIVE API calls to actual network traffic • Optimize and tune? • Not really • No great analysis features • Easy to lose serious problems in the weeds

26. Enter the NetGrove • Ships in the XDK as of March 2008 • Coming soon for Games for Windows – LIVE • Uses capture files from NetMon 2.0+ • Sophisticated analysis of capture files • Flexible and functional GUI • Warning system

27. NetGrove Key Features • Interactive network topology view • Packet traffic view • Enhanced packet information display • Graphical analysis of packet statistics • Text-based output and command-line usage

28. NetGrove Window

29. Network Topology View

30. Packet Traffic View

31. Packet Information

32. Graphical Analysis of Packet Statistics

33. Warnings Window

34. Text-based output

35. But wait, there’s more! • Includes built-in tutorial and sample capture file

36. So What Do You Do with This? • Seriously! • First • Some • Discussion • About • Network • Optimization • (An aside in many parts)

37. The Story of Goldilocks • The Goldilocks principle of packet size • 50 < Payload Size < 1264 • Coalesce! • The Goldilocks principle of frequency • This frequency is JUST RIGHT: 15–20 packets/s • Varies slightly by game type

38. Other Naïve Optimizations • Avoid XSP padding! • This includes using port 1000 • Avoid struct padding! • VDP always, UDP rarely, TCP almost never • Don’t send packets you don’t need to send • For example, don’t send muted voice

39. Slightly More Complex Choices • Do not lock send rate to framerate • Use peer-to-peer networking when feasible • Don’t need authoritative server • Can’t rely on one server to take high CPU and network bandwidth load • Definitely send voice peer-to-peer

40. Voice and Video • Always use peer-to-peer networking • Don’t send muted voice • Partition traffic by • Proximity to player • Team channel • “Importance” rating

41. Compressing Data • Quantize data • Pack Booleans as bit flags • Use polar coordinates (2 parameters) rather than 3D position (3 parameters) • Use random number seed to send simulation state • Do not send strings! • For larger chunks, use lossless compression

42. Modern Implementations • Delta compression • Client-side prediction • Lag correction and interpolation • Contextual bandwidth scaling • Separate guaranteed and non-guaranteed data

43. Sequential Packet Delivery • Use with great caution • Separate guaranteed and non-guaranteed data into different packets and queues • Can cause serious lag even under ideal network conditions when a packet is dropped • Also applies to in-house reliable packet delivery protocols

44. Sequential Packet Delivery PACKET 1 TIMEOUT HOST CLIENT PACKET 0 PACKET 1 DROPPED PACKET 1 (R) PACKET 2 PACKET 3 PACKET 4 PACKET 5 REQUEST RESEND PACKET 1 STALL GAME

45. Setup Diagram * *Series of Tubes

46. Living With Latency • Embrace quantum uncertainty • You can never know where an object is • Only where it used to be • ... and how fast it was moving • The current state is a probability field • Is the cat alive or dead? • Each player has their own parallel universe • Our goal is to keep the universes similar • No red pill

47. Prediction Relativity • Remember the parallel universes • Send information relative to the recipient • “Fired toward position (x, y, z)” • What if players are in different places? • “Shot at Shawn, missed 10° to the left” • Robust even if player positions differ

48. Prediction Paradox • Larger packets = less bandwidth • Position + velocity + controller input • More data makes prediction work better • Allows lower packet send rate • Fewer packet headers

49. Setup • Download and install NetMon 3.2 • Run NetMon as Administrator • Enable P-Mode • Use a hub! Switches and routers do not work.

50. Profiling Your Title • Start with a 2 player capture, then gradually add more players • Capture with voice and/or video enabled • Capture lobby, eliminate redundant traffic • Capture gameplay when all players are standing still, look for redundant packets • Capture worst-case scenario (heavy battle)