Optimistic Load Balancing in a Distributed Virtual Environment

1. Optimistic Load Balancing in a Distributed Virtual Environment Roman Chertov and Sonia Fahmy Department of Computer Science Purdue University {rchertov, fahmy}@cs.purdue.edu http://www.cs.purdue.edu/~rchertov/ http://www.cs.purdue.edu/~fahmy/

2. Distributed Virtual Environment u2 ISP 2 u1 u5 Serv2 u1 u4 u2 u3 Serv1 Internet u3 u4 Serv2 Serv1 DVE Servers ISP1 u5

3. Massive Virtual Environments • Massive Distributed Virtual Environments (DVEs) are hard to create. • How to design a DVE architecture capable of: • High update rates • Large scale • Simple yet effective load balancing • Non-partitioned unified world

4. Approaches Peer-to-Peer • Scales Well • No single point of failure • No latency guarantees • Not very dynamic Distributed Servers • No single point of failure • Not very dynamic Server Cluster • Can be highly dynamic (hot spots) • Latency can be guaranteed • Single point of failure

5. Related Work: Architecture • Peer to Peer • S. Hu and G. Liao, “Scalable peer-to-peer networked virtual environment,” NetGames 2004 • T. Iimura et al., “Zoned federation of game servers: A peer-to-peer approach to scalable multi-player online games,” NetGames 2004 • Distributed Servers • A. Bharambe et al., “A Distributed Architecture for Interactive Multiplayer Games,” Tech Report 2005 • J. Lui and M. F. Chan, “An Efficient Partitioning Algorithm for Distributed Virtual Environment Systems,” IEEE Transactions on Parallel Distributed Systems 2002 • B. Ng et al., “Multiserver support for large-scale distributed virtual environments,” IEEE Transactions on Multimedia 2005 • Server Cluster • T. Duong and S. Zhou, “A dynamic load sharing algorithm for massively multiplayer online games,” ICON 2003 • D. Min et al., “A Load Balancing Algorithm for a Distributed Multimedia Game Server Architecture,” ICMCS 1999 • M. Hori et al., “Scalability issues of dynamic space management for multiple-server networked virtual environments,” PACRIM 2001 • D. Bauer et al, “Network infrastructure for massively distributed games,” NetGames 2002

6. Related Work: Load Balancing • Game space can be divided into a grid and servers manage the regions. • B. Ng et al., “Multiserver support for large-scale distributed virtual environments,” IEEE Transactions on Multimedia 2005 • M. Hori et al., “Scalability issues of dynamic space management for multiple-server networked virtual environments,” PACRIM 2001 • T. Duong and S. Zhou, “A dynamic load sharing algorithm for massively multiplayer online games,” ICON 2003 • Glen Deen, “Making Quake II Massively Multiplayer with OptimalGrid,” ICPP 2005 Keynote • Game space can be divided into cells and a server can manage cells that are not adjacent in game space • B. Vleeschauwer et al., “Dynamic microcell assignment for massively multiplayer online gaming,” NetGames 2005

7. Our Approach • Server Cluster with stateless servers. • Server coverage is dynamic (move, resize) • Servers perform simple greedy actions to achieve global balanced state.

8. How it Looks

9. System Architecture • Load balancing Core runs the virtual environment and performs data exchange and validation • Gateways • Aggregate client data • Provide fast connection to the Core via MPLS • Filter updates from the Core for individual clients

10. System Architecture (cont’d) • Servers are stateless to avoid migration costs. • If update state is close to total state this will work. (action games) • Application specific routing is used to deliver messages to the correct server. • Gateways and Servers run in sync. • Gateways aggregate client data and push it to the Core • Core pushes new data to the gateways

11. Load Balancer Properties Properties of a good load balancer • Achieve even load distribution • Efficiently handle sparse environments • Allow dynamic points of interest Observation: • In DVEs, clients tend to cluster around points of interest

12. Game Space Partitioning • It is possible to use rectangles to partition game space into non-overlapping regions. • Rectangles are easier and faster to use than more complex polygons. Server 2 Server 1 Server 3

13. Server 2 Server 2 Server 1 Server 1 Load Mitigation • If the server load exceeds a Threshold,then it has to migrate some client to another server Server 2’s Threshold of 3 is exceeded Balance is reached

14. Server 2 Server 1 Server 3 Problems with Rectangle Bounds • When using rectangles for boundaries inefficiencies will occur!

15. Server 2 Server 2 Server 1 Server 1 Intersection Resolution • Servers can migrate their clients to other servers if their coverage areas overlap and MaxClients will not be exceeded in the target server. Server 2 and Server 1 overlap! Overlap is resolved

16. Additional Rules • Client transfer is rejected if coverage of the server will exceed MaxArea (configurable parameter) • If coverage of the server exceeds MaxArea then set Threshold to 0 to start migrations

17. Tradeoff • If MaxClients >> Threshold then load variance will increase; however, conflicts will be easier to resolve.

18. Single Server Tasks • NP to process the physics and cheat detection for N clients. • N2 to search for local client interactions. • Total run time = NP + N2 i.e.O(N2) N2can be reduced to N1.5 by using quad trees. For large enough N, the interaction search will take more time than computing physics!

19. Tasks of a Distributed Server i • NiP to process the physics and cheat detection for its own clients. • Ni2 to search for local client interactions. • M-1 to determine the K intersecting and adjacent rectangles that are managed by other M-1 servers. • Ni ΣKk=0Nk to determine the clients that potentially can interact with its clients. • Ni+K to determine the cilent that is closest to the closest server if migration is required. • KNi to determine which current clients are within other servers’ coverage so that they can be migrated there, provided the max capacity is not reached.

20. Runtime Complexity Single Server Complexity O(N1.5) • N total number of clients Distributed Server Complexity O(Ni1.5 + (Ni∑Kk=0Nk)1.5) • K is the number of overlapping servers • Ninumber of clients at one server It is essential to keep Ni as close to N/ServerNum and K as small as possible!

21. Experimental Setup • 10x10 Km map with 64 points of interest (evenly spaced) • 64 Servers • 100-2000 Clients (increased by 100) • Servers can do only 3 migrations when trying to reach Threshold. • 30 minute run time • Parameters • Threshold = ClientNum / 64 • MaxClients = Threshold * 1.5 • MaxArea = 10 Km x 10 Km / 64

22. Client Behavior • A client chooses a point of interest in MapLen / 3 radius and goes there. • Client chooses a speed at random with 5 m/s being the max. • While en-route, it can choose another point with a 0.001 probability (tested every cycle) • When it arrives at the point of interest, it traverses a 50 m x 50 m region for 300 m total. Then it chooses another destination.

23. Client Behavior (cont’d) • Random Way Point model is not considered accurate for some games: • S. Tan et al., “Networked game mobility model for first-person-shooter games,” NetGames 2005 • In our case, the map has no geographical obstacles, clients can change direction while en-route, and clients prefer way points that are closer. • Hence, the client mobility model we chose is adequate.

24. Load Balancing of Clients • Figure (a) shows that the average is close to the ideal distribution. • Property (i) is satisfied (good balancing). • Figure (b) shows the increase in Std. deviation as the number of clients increases. This is because of the balancing tradeoff.

25. Total Covered Area • When the number of clients is small, the server cover a small percentage of the total area. • Property (ii) is satisfied (sparse areas)

26. Coverage Overlap • Number of overlaps is K from analysis. • K increases very slowly as number of clients increases. • Since K is small and the load is close to even, our distributed system will outperform a single server.

27. Dynamic Points of Interest • We have conducted 10 runs with 2000 clients. • The configuration is the same for all the runs except the seeds. • The points of interest are randomly chosen on the map at the start.

28. Dynamic Points of Interest (cont’d) • Small variance between the runs. • Average Std. deviation was 8.6 • Average server overlap count was 1.83 This shows that the system can handle dynamic points of interest that are unknown ahead of time. Property (iii) is satisfied

29. Large Number of Clients • 10000 clients. • 100 servers. • 100 points of interest. • Result • Std. deviation of 35.2 • Average K 2.67 This shows potential that the system can scale to very large number of clients.

30. Conclusions • Our load balancer takes simple greedy actions. • Good load balancing properties are satisfied. • Each Server can in turn be a Core, thus leading to good hierarchical scaling properties! • Future work: Adjust system parameters as clients join and leave.