Multi-Agent Systems:

1. Multi-Agent Systems: The Tao of Soccer A tutorial presented at SFU Surrey Yu Zhang March 8th, 2005

2. Overview • Multi-Agent Systems. • Intelligent Agents, agent designs • multi-agent systems. • The Tao of Soccer. • Background • Client-Server Architecture • GUI client and Java3D view • TOS AI programming guide • TOS Demo • The End • Questions & Answers

3. What is an agent? • An agent is an entity that perceives its environment through sensors and acts upon that environment through effectors.[1] • For each possible percept sequence, an ideal rational agent should do whatever action is expected to maximize its performance measure, on the basis of the evidence provided by the percept sequence and whatever built-in knowledge the agent has.[1]

4. Design Intelligent Agents • The job of AI is to design the agent program that implements the agent mapping from percepts to actions.[1] • In order to design an agent program, we need to know the possible percepts and actions, what goals or performance measure that agent is supposed to achieve, and what sort of environment it will operate in. • We will consider three basic types of intelligent agents: Simple reflex agents, agents that keep track of the world and goal-based agents.

5. Simple reflex agents • A simple reflex agent[1] stores a set of condition-action rules. For a TOS soccer player, we can have rules like: • If have-ball and close-to-opponent-goal then shoot • The simple reflex agent works well only if the correct decision can be made on the basis of the current percept. • Problem: But certain states can not be determined only by the current percept. • A soccer player is dribbling the ball.

6. Agents that keep track of the world • This type of agents maintains some internal state information in order to better determine world states and conditions.[1] • The agent program also maintains two more types of knowledge. • How the world evolves independently of the agent. • How’s the agent’s own actions affect the world. • Problem: Reflex agents are not flexible. The rules has to be re-written to adapt to a new goal or a new environment.

7. Goal-based agents • How to set goals? • The designer manually set the goals for the agent. • The agent sets its own goals by defining a utility function. • Utility is a function that maps a state to a real number, which represents the associated degree of happiness. • Goal-based agents[1] store goal information and the information about the results of possible actions in order to choose actions that achieve the goal. • Goal-based agents reasons about the future.

8. Layered agent architecture • A layered agent architecture is modeled as a hierarchy of interactive sub agent modules.[3] • The higher layers are usually event-driven, goal-based. • The lower layers are fixed-sample-time driven, simple-reflex-based. • Each layer is a sub-agent with its own inputs and outputs. • Inputs: controls signals from the higher layer, the sensing signals from the environment and the states from the lower layer. • Outputs: controls signals to the lower layer and the states to the higher layer. • Control commands flow down and the sensing information flow up.

9. Multi-agent systems • A multi-agent system is a group of agents that interact to solve problems that are beyond the individual capabilities.[2] • Multi-agent system design is more complicated than a single agent design. It presents many new cooperation issues. • A multi-agent system solves the big problem by “divide and conquer”. This presents us… • Cooperation issue 1 - Task allocation: how to assign responsibility to a single agent? • Solution: each agent is assigned a role within the team with its own behavior expectations and goals. • Now agents may have conflicting knowledge, actions and goals. This presents us… • Cooperation issue 2 – Resolving Conflicts: How to resolve conflicting knowledge, actions and goals among agents. • Solution: assign authority levels to roles, communication and negotiation among agents. • Now agents need to communicate, this presents us… • Cooperation issue 3 – Communication: How to understand each other? • Solution: develop a language and a communication system that can accept and interpret this language.

10. The Tao of Soccer :Tao direction; way; method; road; path; principle; truth; reason; skill; method; Tao (of Taoism); a measure word; to say; to speak; to talk

11. Background and History • 1992 – Prof. Alan Mackworth proposed robotic soccer as a test bed for multi-agent system research and started the world's first soccer-playing robots dynamite. • 1997 – The first RoboCup was held during the International Joint Conference on Artificial Intelligence (IJCAI), in Nagoya, Japan. • Soccer Server is the official simulator for the RoboCup Simulation League. • Written in C++, complicated client/server protocol, no human playing ability, only runs on UNIX type machines. • 2001 – The Tao of Soccer was created. • Written in pure Java, similar to RoboCup but with simplified client/server protocol, and it can be played by human as well.

12. TOS: a soccer environment simulator • For each simulation step, TOS calculates and gives each agent its percept, gets an action from each agent, and then updates the environment. • TOS Simulator has five main classes. • SoccerWorld: represents the state of TOS environment. A soccer field, two teams of players, one group of viewers and a ball. • Host: a thread waits for connection requests and actions from agents. • HeartOfWorld: a thread calculates and gives each agent its percept, updates the SoccerWorld at every simulation step. • SoccerRules: simulates and calculates the soccer rules, such as off-side, goal scores. • SoccerPhysics: simulates the movement of player and the ball.

13. Soccer Physics simulation • Soccer field is 2D rectangular, 100 x 65 meters. The center of the soccer field is set to (0,0), Y goes up, X goes right. • The players and the ball are treated as circles. Movements of these objects are simulated stepwise for every 50 milliseconds. (24steps/sec) • At each simulation step, movement of each player is calculated in the following manner: • P1 = P0 + V0; • V1 = V0 + A0; • A1 = FORCE * K1 - V0 * K2; • movement of the ball is calculated as: • P1 = P0 + V0; • V1 = V0 + A0; • If (kicked by a player) { A1 = KICKFORCE * K1; V1 = 0; } else A1 = -FRICTIONFACTOR * V0; • Collisions: When more than two players overlap, all of them are moved back until they do not overlap. Then their velocities are multiplied by -0.1.

14. Soccer Rules simulation • Once the server starts, it enters a 4-period match cycle repeatedly until it is turned off. • Pre Game: The referee is not activated. No score is recorded. • First Half: The game starts. The referee is activated to enforce the soccer rule. • Half Time: The referee is deactivated again. • Second Half: The game restarts. The referee is reactivated to enforce the soccer rule. • A kick-off is a way of starting or restarting play at the start of the match, or after a goal has been scored or at the start of the second half of the match. When the kick-off mode is on, the opponents of the team taking the kick-off are at least 9 meters from the ball until the ball is touched by an opponent player. • Other implemented soccer rules are: • Goal kick • Corner kick • Throw in • Offside, the offside rule can be turned off by a command-line switch

15. TOS: client-server architecture • TOS runs on internet using the client-server architecture. • TOS simulator runs as the server to provide percepts to the agents and receives actions from them on the internet. • TOS agents run as the clients to receives percepts from the server and send actions to the server on the internet. • Percepts and actions are encoded as LISP-like text messages, packed and transmitted as UDP/IP packets between the server and client. • A client can be implemented in any language. • A Java client can use built-in API soccer.common to communicate with the server. • There are two types of TOS clients (agents), Player and Viewer.

16. Client to Server: Connect • First, a client has to join the game by sending a connection message to the server. • (c ClientType SideType). For example (c p l) • To do this in Java, we use API in soccer.common // step 0: setup network communication, usually only done once Transceiver transceiver = new Transceiver( false ); // step 1: fill the data ConnectData connect = new ConnectData(ConnectData.PLAYER,ConnectData.LEFT); // step 2: create the UDP packet by providing data and server address Packet packet = new Packet( Packet.CONNECT, connect, address, port); // step 3: send the packet transceiver.send( packet ); • What if the connection message got lost during the transmission? Or the team is full? • Solution: the server sends back an acknowledge packet.

17. Server to Client: Initialize • The sever responds the connection request from the client by sending an initialization message back. • (i ClientType Number) • ClientType can be one of l, r, v or f.l means left, r means right, v means viewer and f means full. In the case of left or right, Number is returned as the player's player number. full means connection fails because of the full team. • To make the connection, the client can send a connection packet every 60 seconds until • it receives an initialization packet from the server, which means a successful connection. • or it reaches the maximum number of tries, and fails the connection.

18. Server to Client: Player Percepts • After the client-server connection is established, the server begins to send visual information to the client. • If the client is a player, it gets see packet • (s Time Side Id X Y Dir Status BallX BallY BallControllerType BallControllerId PlayerInfo PlayerInfo ...); • PlayerInfo ::= (PlayerSide PlayerId X Y Dir) • If the client is a viewer, it gets view packet • (v Time BallX BallY BallControllerType BallControllerId PlayerInfo PlayerInfo ...); • The difference between see and view packet is: view packet does not have visual data for viewer himself. • We use soccer.common API to process visual data. packet = transceiver.receive(); if(packet.packetType == Packet.SEE) { see = (SeeData) packet.data; distance2Ball = see.player.position.distance( see.ball.position ); direction2Ball = see.player.position.direction( see.ball.position ); }

19. Server to Client: Player Percepts • The server also sends referee information to the client whenever the situation on the field is changed. • (r Time Period Mode LeftName LeftGoal RightName RightGoal) • Time ::= simulation step; • Period ::= 0 | 1 | 2 | 3; 0 means preGame, 1 means firstHalf, 2 means halfTime, 3 means secondHalf. • Mode ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11;0 means beforeKickOff, 1 means kickOffL, 2 means kickOffR, 3 means throwInL, 4 means throwInR, 5 means cornerKickL, 6 means cornerKickR, 7 means goalKickL, 8 means goalKickR, 9 means offsideL, 10 means offsideR, 11 means playOn. • We use soccer.common API to process referee data. packet = transceiver.receive(); if(packet.packetType == Packet.REFEREE) { referee = (RefereeData)packet.data; gamemode = referee.mode; gamePeriod = referee.period; }

20. Client to Server: Player Actions • A player can move around by sending the drive command to the server. • (d Dir Force) • -180 < Dir < 180 • -30 < Force < 100 • To do this in Java, we use API in soccer.common // step 0: setup network communication, usually only done once Transceiver transceiver = new Transceiver( false ); // step 1: fill the data DriveData driver = new DriveData( direction, force ); // step 2: create the UDP packet Packet packet = new Packet( Packet.DRIVE, driver, address, port); // step 3: send the packet transceiver.send( packet );

21. Client to Server: Player Actions • If the player is close enough to the ball, he can kick it by sending the kick command to the server. • (k Dir Force) • -180 < Dir < 180 • -30 < Force < 100 • To do this in Java, we use API in soccer.common // step 0: setup network communication, usually only done once Transceiver transceiver = new Transceiver( false ); // step 1: fill the data KickData kicker = new KickData( direction, force ); // step 2: create the UDP packet Packet packet = new Packet( Packet.KICK, kicker, address, port); // step 3: send the packet transceiver.send( packet );

22. Client to Server: Player Actions • A player can broadcast a message to everybody on the field by sending the message to the server. • (t Message) • The length of the Message <= 30 • At most one Message every 30 second • To do this in Java, we use API in soccer.common // step 0: setup network communication, usually only done once Transceiver transceiver = new Transceiver( false ); // step 1: fill the data TalkData talker = new TalkData( “Attack! Attack! Attack!”); // step 2: create the UDP packet Packet packet = new Packet( Packet.TALK, talker, address, port); // step 3: send the packet transceiver.send( packet );

23. Server to Client: Player Percepts • After the server receives a broadcast message from a client , it sends audio information to all clients in the game. • (h Time Side Id Message) • We use soccer.common API to process audio data. packet = transceiver.receive(); if(packet.packetType == Packet.HEAR) { heard = (HearData) packet.data; String message = heard.message; int speakerID = heard.id; int speakerSide = heard.side; }

24. SoccerMaster: a TOS GUI client • SoccerMaster can be run in three modes. • In the VIEW mode, it connects to the server as a viewer client. • The user observes two AI teams play against each other. • In the PLAY mode, it connects to the server as a player client. • The user plays the game himself. • In the REPLAY mode, it reads a log file and redisplays the match. • The user reexamines a recorded match. • The user can hot-switch Java2D and Java3D view any time, no application restart is needed.

25. TOS Java 3D View • The java 3D view is implemented in package soccer.client.view.j3d. • FieldJ3D is the main class for displaying the 3D scene. In this class, we add content nodes to the Java3D scene graph tree; such as soccer players, the ball, the soccer field, the goal poles, etc. • Robot contains the manually coded player 3D model using spheres and cylinders. • MouseRotateXZ and MouseZoomOnRightClick are used to control the camera view. It’s very easy to add existing 3D models to TOS 3D scene. For example, to load a balloon into the 3D scene using Xj3D loader library: VRML97Loader modLoader = new VRML97Loader(); URL modURL = getClass().getResource("/model/balloon.wrl"); Scene model = modLoader.load(modURL); BranchGroup balloon = model.getSceneGroup(); scene.addChild(balloon);

26. SFU Default AI Soccer Team • In the package edu.sfu.soccer.agent, there are 4 classes. AIPlayers, Formation, Robot, and WorldModel. • AIPlayers is the main class, it maintains one or two teams of threads, each thread controls an independent TOS player. • Robot contains the logic for controlling the TOS player. • WorldModel contains the internal states, actions and team formation. • Formation contains the team formation data. • Each Robot is a goal-based agent, uses three main methods, sense(), plan() and execute().

27. AIPlayers: the main class • AIPlayers maintains one or two teams of Robots. Each team can have 0-11 players. The size of each team is given at the command line. For example: • To control 11 players of the left team. java -cp soccer.jar edu.sfu.soccer.agent.AIPlayers -l 11 -r 0 • To control 11 players of the right team. java -cp soccer.jar edu.sfu.soccer.agent.AIPlayers -l 0 -r 11 • Each player is run as a thread. Why do we use threads? • To soccer server, each separate UDP socket represents a unique soccer client. There’s no difference between a process client and a thread client. • A thread uses less computing resources than a process. • One Java process can control 22 AI players. It’s easier for us to use and to maintain the soccer AI.

28. WorldModel and Formation • WorldModel is the memory bank for our soccer agent, it stores: • How the world evolves. • 1, The static information about the soccer field, such as length and width. • 2, The static information about the soccer physics simulation, such as the maximum speed of players and the ball. • Internal states. • 1, The low level percepts such as the positions of moving objects on the field. • 2, The high level knowledge such as my relative distance and direction to the ball, if I’m interested in getting the ball, if I’m stuck, etc. • Multi-agent. My current role in my team formation. • The high level actions I’m going to execute, such as shoot, move, chase, avoid stuck and avoid offside. • Formation stores the formation information of a soccer team. A soccer team can have different style of formation, such as “433”, “523” and “424”.

29. Robot: a goal-based soccer agent • Robot implements a goal-based agent. • Sense(): receives UDP packets from the server about its current percepts, updates its world model. • Plan(): determines the high level actions, such as shoot, dribble, move and chase. • Execute(): translates the high level actions into a sequence of low level actions that can be accepted by the soccer server, such as kick and drive. • Plan() method calls a list of decision-making methods to determine the next high level action. • ShouldIScore(): determines if I should shoot the goal. • ShouldIDribble(): determines if I should dribble. • ShouldIPass(): determines if I should pass the ball to a teammate. • DetermineWhereToMove() & DeterminePlayerPos(): determines where should I move when I don’t have the ball

30. Create your own soccer team • Approach 1: Make changes to the default SFU soccer team. • Change the Formation class, experiment with different team formations. • Improve the decision-making methods in the Robot class. These are ShouldIPass(), ShouldIScore(), ShouldIDribble(), DetermineWhereToMove() and DeterminePlayerPos(): . • Approach 2: Create your own AI team from scratch. • If you have a great idea that is hard to implement in the default SFU team. • If you want to experiment with some other agent architecture. One example of such alternative is Dynamo98 that uses constraint nets robotic architecture. • Dynamo98 is in package com.graviton.dynamo98. You may also build your team on this AI implementation as well. • Approach 3: Add new AI methods into the existing AI implementations. • Include a rule-based engine, such as a Prolog or Fuzzy logic, to handle high level planning tasks, instead of hard-coding the decision-making methods. • Add learning capabilities, such as neural networks, decision trees, reinforcement learning and genetic algorithms.

31. TOS DEMO: Play TOS as a game • Start the TOS server. Double click soccerserver.bat • Start the TOS GUI. Double click soccer.bat • Play the game with the coach ability on. • Move the player around with the left mouse click. • Open a DOS window, type sfu_team.bat both to start AI. • Click space to chase the ball. • When you have the ball, right click to kick the ball, left kick to dribble, and space to shoot. • Step forward the game, continue, step forward. • Forward the game period, play again. • Toggle the player ID display. • Toggle music. • Toggle 3D display. • Left click to rotate the 3D view, right click to zoom in/out the 3D view. • Reset the top-down 3D view. • Stop the game.

32. TOS DEMO: TOS as a simulation • Start the TOS server. Double click soccerserver.bat • Open a DOS window, type dynamo98.bat both to start AI. • Start the TOS GUI. Double click soccer.bat • View the game with the coach ability on. • Step forward the game, continue, step forward. • Forward the game period, play again. • Toggle the player ID display. • Toggle music. • Toggle 3D display. • Left click to rotate the 3D view, right click to zoom in/out the 3D view. • Reset the top-down 3D view.

33. TOS DEMO: TOS Replay • Start the TOS GUI. Double click soccer.bat • Load the log file. • Play the log file. • Fast forward, normal, fast forward. • Pause, play, pause, forward, pause, play • Backward play, play, backward play, fast backward. • Toggle 3D display. • Left click to rotate the 3D view, right click to zoom in/out the 3D view. • Reset the top-down 3D view.

34. The End and the beginning. How to get help and to help others? • Visit TOS website at http://soccer.sourceforge.net/soccer/ • Join the mailing list athttp://games.groups.yahoo.com/group/tao_of_soccer/ • Post your questions and answers to the TOS forums or to the TOS mailing list. • Make your AI team publicly available on the TOS web site. • Join the TOS development. • Find and fix bugs. • Add new functions to the server and the GUI. • Improve the 3D view. • Make 3D models. • Improve the documentation. • Help to maintain the TOS website. • Thank you and Good Luck!

35. References • Artificial Intelligence, A Modern Approach. 1995, By Stuart Russell and Peter Norvig • Multiagent Systems. 1998, By Katia Sycara. AI Magazine 19(2) • A Constraint-Based Robotic Soccer Team. 2002, By Yu Zhang and Alan Mackworth. Constraints, 7, 7-28.

36. Questions & Answers