1 / 56

NetGrove: New Tools for Network Optimization

NetGrove: New Tools for Network Optimization. Tomas Vykruta Software Development Engineer XNA Publishing Platform Microsoft. Speed of Light Still Constant. Latency. Speed of light = 186,282 miles per second Seattle to L.A. 960 miles = 5 milliseconds Seattle to England

brian
Download Presentation

NetGrove: New Tools for Network Optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. NetGrove: New Tools for Network Optimization Tomas Vykruta Software Development Engineer XNA Publishing Platform Microsoft

  2. Speed of Light Still Constant

  3. Latency • Speed of light = 186,282 miles per second • Seattle to L.A. • 960 miles = 5 milliseconds • Seattle to England • 4,799 miles = 26 milliseconds • Speed of light in fiber or copper slows to 60% • Each router adds 5 to 50 ms • DSL or cable modem adds 10 ms

  4. Bandwidth on the Rise Thanks, Bungie!

  5. Also on the Rise • Player expectations • More players in a session together • More voice, video, and background downloads • Network-using features in your game, such as data-feeds and spectator modes • Network-using features in the platform • Other home devices consuming bandwidth

  6. Building a successful title • Optimizing game networking is still very important! • Have a Plan: • Define max bandwidth goal • Monitor bandwidth during development **DO NOT LEAVE UNTIL SUBMISSION**

  7. Defining Maximum Bandwidth • Host = worst case scenario • Let’s target top 80% of users • 80% have 175ms or better latency • And 176kb/s down, 226kb/s up or better bandwidth • From “The Real World: Console Latency and Bandwidth on Xbox LIVE” white paper on xds.xbox.com • Subtract reservation (i.e. voice chat) • Divide remaining bandwidth by “max # of players – 1”

  8. Bandwidth Calculation 5 player game 68kbps CLIENT 14kbps 17kbps CLIENT 14kbps HOST 56kbps 17kbps 14kbps CLIENT 17kbps 14kbps CLIENT 17kbps

  9. Max Bandwidth cont’d • Scenario: theoretical 16 player game with 30 kbps reservation for voice chat • Host is updating 15 players • And receiving data from 15 players • Max Bandwidth = 9 kbps down, 13 kbps up per client max • Also simulate 175 ms latency for playtests • Monitor regularly!

  10. Max Bandwidth Worksheet

  11. Bandwidth Data 2007-2008 Table 2: 2007 Bandwidth Table 1: 2007 Latencies

  12. Latency and Bandwidth Data • 2008 bandwidth stats same as 2007 • Median latency 84 ms • Only 10% of consoles are 32 ms or less • Worst case latency500 ms • 336 kb/s median downstream bandwidth • 352 kb/s median upstream bandwidth • Typical packet loss 2% • Worst case packet loss 10%

  13. The Real World • From real-world title network reviews • Best case = 4 kbps per client in AAA FPS title • Worst case = 370 kbps+ for the host in a 4 player game of a different AAA title!

  14. From a real network review • “Stopped moving” message sent every frame • Many invisible objects simulated • Entire animation state sent (100 bytes+) • Uncompressed data • Position + Orientation + LookAt sent each frame = 12 + 36 + 36 bytes = 84 bytes • Host not coalescing data • NPCs simulated like players • Many more… • End result after optimizations = 5 kbps per client!

  15. Run-time Bandwidth Monitoring • Fps, Memory Usage, why not peak bandwidth? • Display on screen, always present • Visual alert to indicate packet loss

  16. Tools • Instrumentationingamecode • Subjectiveplaytestexperience • Packetsniffing • Triedandtrue • Blackboxandlowbarriertoentry • Manysophisticatedsniffersavailable

  17. Network Monitor 3.2 • Process Tracking to identify rogue applications • New engine for improved capture rate in high-speed networks • Find conversations to view TCP streams, HTTP flows etc. • Extensive parsers for over 300 protocols! • Network Monitor API: Create your own applications that capture, parse, and analyze network traffic

  18. Network Monitor 3.2

  19. Quick NetMon Tour • Conversations • Find frame (CTRL+F) • Display and capture filters • Declarative language with Intellisense • Color filters • Ex: XSP (by itself) filters down to Xbox LIVE traffic • NMCAP for command-line captures

  20. NetMon Protocols • Extensive script-based protocol parser language • Documented in NetMon\Help\ParserLanguage.doc • Create your own! • Trade with your friends! • Sure would be nice if the LIVE protocol…

  21. LIVE Secure Packet Parsing • Xbox 360 SDK and Games for Windows – LIVE SDK both provide a NetMon Parser • XSP.NPL • Automatically installs with XDK • Encrypted data is still encrypted • XNET_STARTUP_DISABLE_PEER_ENCRYPTIONto send unencrypted data while debugging • XSP parser + NetMon filtering + no encryption = see what’s actually happening on the wire

  22. DISABLING ENCRYPTION • XNET_STARTUP_BYPASS_SECURITY added in November 2008 • Disables encryption, view unencrypted XSP data in NetGrove and NetMon • Flag must be enabled on all consoles • Silently disabled on RETAIL hardware • Stop using XNET_STARTUP_BYPASS_SECURITY • Disables authentication and XSP completely

  23. LIVE Server Traffic Traffic to LIVE Presence Server Traffic to LIVE Statistics / Profile Server

  24. PIX – System Monitor PIX provides counters for LIVE server traffic Traffic to LIVE Statistics Server

  25. So What Do You Do with This? • Debug pathological network conditions • Excessive or unexpected LIVE traffic (TCR) • Mapping LIVE API calls to actual network traffic • Optimize and tune? • Not really • No great analysis features • Easy to lose serious problems in the weeds

  26. Enter the NetGrove • Ships in the XDK as of March 2008 • Coming soon for Games for Windows – LIVE • Uses capture files from NetMon 2.0+ • Sophisticated analysis of capture files • Flexible and functional GUI • Warning system

  27. NetGrove Key Features • Interactive network topology view • Packet traffic view • Enhanced packet information display • Graphical analysis of packet statistics • Text-based output and command-line usage

  28. NetGrove Window

  29. Network Topology View

  30. Packet Traffic View

  31. Packet Information

  32. Graphical Analysis of Packet Statistics

  33. Warnings Window

  34. Text-based output

  35. But wait, there’s more! • Includes built-in tutorial and sample capture file

  36. So What Do You Do with This? • Seriously! • First • Some • Discussion • About • Network • Optimization • (An aside in many parts)

  37. The Story of Goldilocks • The Goldilocks principle of packet size • 50 < Payload Size < 1264 • Coalesce! • The Goldilocks principle of frequency • This frequency is JUST RIGHT: 15–20 packets/s • Varies slightly by game type

  38. Other Naïve Optimizations • Avoid XSP padding! • This includes using port 1000 • Avoid struct padding! • VDP always, UDP rarely, TCP almost never • Don’t send packets you don’t need to send • For example, don’t send muted voice

  39. Slightly More Complex Choices • Do not lock send rate to framerate • Use peer-to-peer networking when feasible • Don’t need authoritative server • Can’t rely on one server to take high CPU and network bandwidth load • Definitely send voice peer-to-peer

  40. Voice and Video • Always use peer-to-peer networking • Don’t send muted voice • Partition traffic by • Proximity to player • Team channel • “Importance” rating

  41. Compressing Data • Quantize data • Pack Booleans as bit flags • Use polar coordinates (2 parameters) rather than 3D position (3 parameters) • Use random number seed to send simulation state • Do not send strings! • For larger chunks, use lossless compression

  42. Modern Implementations • Delta compression • Client-side prediction • Lag correction and interpolation • Contextual bandwidth scaling • Separate guaranteed and non-guaranteed data

  43. Sequential Packet Delivery • Use with great caution • Separate guaranteed and non-guaranteed data into different packets and queues • Can cause serious lag even under ideal network conditions when a packet is dropped • Also applies to in-house reliable packet delivery protocols

  44. Sequential Packet Delivery PACKET 1 TIMEOUT HOST CLIENT PACKET 0 PACKET 1 DROPPED PACKET 1 (R) PACKET 2 PACKET 3 PACKET 4 PACKET 5 REQUEST RESEND PACKET 1 STALL GAME

  45. Setup Diagram * *Series of Tubes

  46. Living With Latency • Embrace quantum uncertainty • You can never know where an object is • Only where it used to be • ... and how fast it was moving • The current state is a probability field • Is the cat alive or dead? • Each player has their own parallel universe • Our goal is to keep the universes similar • No red pill

  47. Prediction Relativity • Remember the parallel universes • Send information relative to the recipient • “Fired toward position (x, y, z)” • What if players are in different places? • “Shot at Shawn, missed 10° to the left” • Robust even if player positions differ

  48. Prediction Paradox • Larger packets = less bandwidth • Position + velocity + controller input • More data makes prediction work better • Allows lower packet send rate • Fewer packet headers

  49. Setup • Download and install NetMon 3.2 • Run NetMon as Administrator • Enable P-Mode • Use a hub! Switches and routers do not work.

  50. Profiling Your Title • Start with a 2 player capture, then gradually add more players • Capture with voice and/or video enabled • Capture lobby, eliminate redundant traffic • Capture gameplay when all players are standing still, look for redundant packets • Capture worst-case scenario (heavy battle)

More Related