Distributed Interactive Simulation 101: The Basics

1. Distributed Interactive Simulation 101:The Basics Don McGregor (mcgredo@nps.edu) Don Brutzman (brutzman@nps.edu) MOVES Institute, Naval Postgraduate School

2. Learning Objectives • What standards are used in distributed simulations in the military? • What types of networks? • What needs to be standardized? • How can we use dead reckoning to reduce network traffic? • What coordinate systems are used?

3. Topics • What’s distributed simulation? • Military modeling & simulation distributed simulation standards • Underlying TCP/IP networks • DIS: Entity State PDUs • DIS: Entity Types and Entity IDs • DIS: Coordinate Systems • DIS: Shooting • DIS: Research Topics • Bibliography

4. Distributed Simulation • A distributed simulation runs on multiple cooperating hosts on the network • One host may control the position and behavior of dozens of simulated entities, and communicate the position and state of those entities to other hosts • Live, Virtual, Constructive: • Live: Real people, real systems • Virtual: Real people, simulated systems (human in loop) • Constructive: simulated people, simulated systems (AI controlled)

5. Live, Virtual, Constructive

6. What Do We Want to Do? • People who want to learn about Distributed Interactive Simulation (DIS) usually are in the “virtual” or “constructive” domains • They want to simulate tanks or other entities in the world, controlled by humans or AI • We need to exchange state information between hosts on the network about entities in the world • To view someone else in the virtual world, we need to know their position, orientation, and other state information • Typically entities are controlled by different hosts that are connected via a network. The state information is usually exchanged via TCP/IP

7. State Information in Distributed Simulations Helo Tank State Information Location, Orientation, Velocity, &c over the TCP/IP network

8. State Information Exchange • The tank & helicopter in the simulation are controlled by different hosts. The simulation running the tank needs to know about the helicopter’s position, and the helicopter needs to know about the tank’s position • We have to come to all sorts of agreements to exchange information. It’s harder than it looks! • But why do we want to standardize, rather than let every vendor do his own thing? • Vendor lock-in considered harmful

9. Simplest Possible Distributed Simulation Case • We want to describe a simulated tank in a virtual world. What do we need? • We need to send state information such as position and orientation from the host controlling the entity to one or more other hosts participating in the simulation (X, Y, Z) Host Receiving Information Controlling Host

10. Simplest Possible Distributed Simulation Case • We assume that some host has authoritative information about the entity state information. Typically this is the host controlling the entity via a joystick, keyboard, AI, etc. We want to inform other hosts of the location of that entity • We also have to send information about orientation, articulated parts, velocity, acceleration, and so on • We transmit the state information to another host, which often displays a graphical representation of the entity

11. What’s So Hard? • For performance reasons the state message is usually sent in binary format. The format of binary messages needs to be exactly specified. • What coordinate system is being used? You need one that works well for ground and air systems, can handle curvature of the earth issues, and everyone has to agree on one • Network issues: what happens if a message is dropped? • Scalability issues: how can we get a reasonable number of entities to participate? • Latency: can we keep the message delay reasonable?

12. Distributed Interactive Simulation • DIS was the first standard to tackle these problems in a systematic way • Originated in the SIMNET project in the 80’s. DARPA supported converting the SIMNET research into a standard; SISO developed the standard and took it to IEEE for approval. • This means anyone can get the standard from IEEE, implement it, and participate in a simulation • Development of standard is continuing; updated DIS standard version 7 just approved

13. Distributed Simulation Standards • There are many ways to exchange the state information, but we want a standard way so we can interoperate with simulations from many vendors rather than being locked in • In the case of defense modeling and simulation, the big three are • TENA: Test and Training Enabling Architecture • HLA: High Level Architecture • DIS: Distributed Interactive Simulation • We will see that TENA and HLA borrow many semantic concepts from DIS. Understanding DIS has many carry-overs to other standards

14. TENA • Used on ranges; often the “L” in Live-Virtual-Constructive simulations • Designed for real time and embedded systems, real sensor systems, etc • In effect it is thinly disguised CORBA distributed objects with some new features to help it work in a simulation environment • Can gateway it to other standards, such as DIS or HLA • See http://tena-sda.org

15. High Level Architecture • HLA is very general and intended to cover most of the defense modeling domains including training, analysis and engineering in addition to virtual worlds • Participants communicate via an agreed-upon Federation Object Model (FOM) and an API associated with a Run-Time Infrastructure (RTI) • Specification is maintained by SISO (http://sisostds.org), is IEEE standard 1516and has implementations by MAK (http://www.mak.com), Pitch (http://www.pitch.se), Portico (older version) (http://porticoproject.org) and others • http://www.pitch.se/hlatutorial is a good introduction to HLA

16. Distributed Interactive Simulation • DIS is the original distributed simulation standard for military M&S • Targeted at virtual environments modeling entities in the world, such as tank/helicopter combat • Many of the concepts worked out in DIS are borrowed in TENA and HLA. For example, the most widely used HLA FOM, RPR-FOM, is patterned after DIS • Maintained by SISO, which presents it to IEEE for approval as 1278.x • Substantial commercial support, open source implementations, many home grown implementations of portions of the standard

17. Distributed Interactive Simulation • What is it? • A standardized way to exchange messages about entities in a virtual world • Common semantics for things like coordinate systems, and how to describe specific entities • Common practices to ensure interoperability

18. Networking: The Protocol Stack • DIS defines the format of the messages, but doesn’t specify how to get the messages from one host to another. Almost always this is done via TCP/IP DIS Implementations Application TCP UDP How messages are exchanged between hosts IP Link and Physical

19. TCP/IP • TCP/IP is the underlying substrate for almost all network communications. Web servers, streaming video, email and almost everything else you use on the internet all rely on TCP/IP • Users typically implement things at the “application layer” at the top of the stack. This is where DIS lives • Somewhat confusingly, the DIS protocol is considered part of the “application layer” in addition to what users think of as the “application”, the visual simulation • The lower layers of the stack are responsible for delivering messages. We can usually ignore this part; it’s all handled by operating system vendors • We interact with the TCP/IP stack via either the User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) APIs.

20. TCP/IP • TCP has higher latency, higher jitter (variation in latency), and doesn’t scale to large numbers of hosts well • UDP has lower latency, lower jitter, scales to large numbers of participants better, but is unreliable • DIS typically uses UDP • Hold on—UDP is unreliable? What’s up with that? • Individual UDP messages may be dropped by the network; there’s no guarantee that a UDP message will be delivered. This is a tradeoff to achieve lower latency and jitter and more scalability • This is not as big a deal as it might seem. If we’re getting position updates from an entity every 1/30th of a second, does it matter if we drop one? We have better information coming along shortly, so why try to resend the dropped message?

21. Byte Endianess • Back in the elder days of computing different CPU makers made different decisions about how to arrange bytes in numbers. TCP/IP does not hide these decisions from you! • Consider a 4-byte binary integer on the wire • DIS chose to use “big endian” or “network byte order” to represent binary numbers. Intel CPUs use little endian byte order. Beware! Big Endian: Most significant bit on the left A B C D D C B A Little Endian: Most significant bit on the right

22. Unicast, Broadcast, and Multicast • UDP messages are like postcards: they contain a fairly small amount of data, and we need to address them to a destination • We can send them to one host (based on IP number): called unicast • We can send them to all hosts on a local network: called broadcast • We can send them to those hosts, on the local network or not, that are interested in the message: called multicast • DIS messages are contained inside of UDP packets and can use any one of these addressing schemes. Broadcast is very common. Multicast is better.

23. DIS Messages • We know what we want to do: send a message that tells another host what the state of an entity is • We know how we do this: we send the message via TCP/IP, typically in a UDP message over broadcast or multicast • What are the actual messages? • This is what is specified by the DIS standard. • There are dozens of possible messages to send, relating to everything from logistics to electronic warfare to radio communications

24. DIS Messages • A partial list of the messages (PDUs) and Families—several dozen PDUs in total • Entity Information/Interaction PDU Family • Entity State PDU • Collision PDU • Warfare PDU Family • Fire PDU • Detonation PDU • Radio Communications PDU Family • Transmitter PDU • Receiver PDU • Logistics PDU Family • Resupply

25. DIS Messages: Entity State PDU • We use ESPDU when want to inform other hosts of the position of an entity we control—the most common PDU by far • Contains information such as: • Entity ID: a unique identifier that describes one entity in the world • Entity Type: a series of enumerations that specify what this entity is. For example, an M1A2 tank, AH-64 attack helicopter, German F-16C, and so on • Position, orientation, velocity, acceleration, angular velocity, and dead reckoning algorithm to use • Timestamp • All messages are in binary format

26. DIS: Entity IDs • The entity ID is a unique identifier for each entity in the world. Before we can tell an entity “hey you, do that” we need a way to differentiate between entities • In DIS this is done via a triplet of three numbers: the Site, Application, and Entity

27. DIS: Entity IDs • Example: before the simulation starts we agree on the following arbitrary numbers

28. DIS: Entity IDs • An tank entity controlled by the YoYoDyne simulator at China Lake may have an entity ID of (42, 112, 17) and no other entity in the simulation should have the same entity ID • An UCAV entity controlled by a ACME simulator at Norfolk would have an EID of (17, 417, 18), and no other entity should have the same ID • Note that this depends on all the participants agreeing to use the same arbitrary numbers! We’ll see the same phenomenon later with Entity Types. • This concept of Entity IDs is also used many places in HLA such as the RPR FOM and in some places in TENA

29. Entity IDs Tank EID: (42, 112, 18) at (x0, y0, z0) EID: (42, 112, 19) at (x1, y1, z1) Helo EID: (17, 417, 18) at (x2, y2, z2) ACME simulator at Norfolk controls an aircraft YoYoDyne simulator at China Lake controls two distinct tanks

30. DIS: Entity State PDU • So a simulator issues an ESPDU, saying “This is the entity’s unique ID, and this is where it is.” • We may need some other information. What if we want to draw this entity? We need to know what this is—a tank, a helicopter, a ship? • This is done via something called the Entity Type, which in turn depends on a SISO document called the “Enumeration and Bit Encoded Values” (EBV). • The EBV document is a long listing of standardized enumerations that lets us identify military hardware

31. Entity Type From EBV Document

32. Entity Type • The Entity Type is a collection of several numbers that precisely define what piece of hardware this is • (1, 1, 225, 1, 1, 6) is an entity type that defines a vehicle, on land, from the US, that is a tank, specifically an M1, and even more specifically an M1A2 with DU armor. • (1, 4, 225, 1, 3, 10) defines a vehicle, undersea, from the US, that is an SSBN, of the Ohio class, that is the USS Pennsylvania (SSBN-735) • (1, 1, 222, 2, 1, 1) defines a vehicle, on land, from Russia, that is an armored fighting vehicle, a BMP-1, basic model • Everyone should agree on the EBV document version used! • In reality this may be hard, and you may need gateways to rewrite these values to make simulations interoperate • These enumerations are also often used in HLA and TENA

33. Entity Type { Kind: 1 (entity) Domain: 1 (Land) Country: 225 (US) Category: 1 (Tank) Subcategory:1 (M1) Specific: 6 (M1A2) } Whenever a DIS message has a record with the above settings we know it’s referring to an M1A2 tank

34. Entity State PDU: Position • What does it mean if we say an entity is at (x, y, z)? This has no meaning without a coordinate system. We need to agree on one, and where the origin is • DIS chose to use a geocentric coordinate system

35. ESPDU: Coordinate Systems • If you want to specify the location of an entity in DIS, you have to specify it in terms of how far from the center of the earth it is. You must also agree on the shape of the earth (eg WGS-84) and terrain • Using geocentric coordinates may seem strange, but DIS has to deal with all sorts of entities, and has to also account for curvature of the earth issues if the simulation demands it. The geospatial people have done the math to convert geocentric to things like (lat, lon, alt) or MGRS. The geocentric coordinate system is often used in TENA and HLA • The folks at SEDRIS have provided an open source framework for dealing with these issues in their Spatial Reference Model (http://www.sedris.org/hdr1trpl.htm)

36. DIS: Coordinate Systems • Very often you set up a local, flat coordinate system for convenience and use that for simulations that cover up to perhaps 20KM. When you actually send the ESPDU you need to convert back to global coordinates; SEDRIS SRM can do this

37. Coordinate Systems The local coordinate system origin set at {lat, lon, alt}. The simulation does all its calculations in this coordinate system, because it’s convenient. Before sending entity information out in a DIS ESPDU, convert from local to geocentric coordinates Y (lat, lon, alt) X

38. DIS: Dead Reckoning • So far, we can send an ESPDU message that has an entity ID, entity type, and its position. If we’re running a visual simulation, how often do we need to send these? • If running at frame rate, about every 1/30th of a second. • If we have 500 entities in a simulation, this can work out to 15,000 UDP messages per second. If you’re doing other computation on the host this will tax your CPU budget • Do we really need to send that fast? If we know how fast and in what direction we’re moving, we can dead reckon in between receiving an ESPDU and draw our entity there. • Sure, we could be lying to the user. Got a problem with that? If our DR is wrong, we just correct our position stealthily when we get better info on the next packet

39. DIS: Dead Reckoning • What DR algorithm is best? It depends! In some situations we might want to include acceleration or angular velocity, but not in others. The ESPDU sender specifies what DR algorithm to use • Typically the sender also performs its own DR to determine what the recipients are seeing. If the sender decides the clients are probably wrong in their guess about where the entity is, it can issue another ESPDU with better location information

40. DIS: Learning About the World and Heartbeat • If we’re running a DIS simulation, how do we learn about all the other entities in the world? • One design choice DIS could have chosen is to use a server. Instead, the designers chose for DIS to be peer-to-peer; there is no central server, and hosts simply talk to one another directly • In DIS all we have to do is listen for ESPDUs. The ESPDUs contain what we need to know: entity type, location, velocity, orientation. This is simple and avoids a single point of failure, and makes configuration easy

41. DIS: Heartbeat • To make this work, every entity has to periodically send and ESPDU even if its state hasn’t changed. This is called a heartbeat • This is simple and robust but affects scalability. Especially so since we are often sending redundant information in the ESPDU, such as the entity type • Typically an entity must send an ESPDU about once every 5 seconds, which means you should learn about every entity in the world within 5-15 seconds. (Remember, UDP packets may be lost because UDP is unreliable)

42. DIS: Timestamp • One of the oddities of UDP is that UDP packets may be duplicatedduringTCP/IP routing; we send a single packet, but two arrive • UDP packets may also arrive out of order. We send packets in order A, B, C, but they arrive in order C, A, B • This creates problems for position updates! • To detect this problem DIS includes a timestamp field • The field typically represents time since the start of the hour • If the next packet we receive has a timestamp before the last packet we processed, we can discard it; it’s old information that has been obviated by new data

43. DIS: Timestamp T=23 Position: {10, 11} T=24 Position: {10, 12} Host A Sends: T=23 Position: {10, 11} T=24 Position: {10, 12} Host B Receives: Host B receives the green ESPDU packet first, then the red packet. We should discard the red packet, because we know it’s older than the green packet

44. DIS: Timestamp • Careful! If you don’t update the timestamp field when sending ESPDUs the receiver may conclude that it is receiving duplicate ESPDU messages and discard them • It is useful to have coordinated time on the hosts, perhaps via Network Time Protocol (NTP), though this is not required • Receivers have to handle roll-over in the field at the top of the hour

45. DIS: Entity State PDU • So far we’ve looked at an ESPDU, which contains • Entity ID • Entity Type • Position, orientation, velocity, etc • Specifies a DR algorithm for the receiver to use • Timestamp • We’ve also seen that we need to agree upon a coordinate system, and agree on the enumerations that describe things like the entity type and portions of the entity ID.

46. DIS: API • DIS doesn’t have an API. This seems strange to people coming from HLA or TENA, but reflects common practice in networking protocols • The standardized part is the format of the messages on the wire. The standard is silent about how to create or receive those messages • Different DIS vendors have different APIs, but all produce the same format messages. This is in contrast to HLA, which has a standard API, but is silent about the format of messages on the wire. As a result, different HLA RTI vendors usually use different message formats for exchanging information • TENA standardizes the API, and there is a single approved implementation of the RTI equivalent; this sidesteps the wire standard problem because there is only one approved RTI equivalent

47. DIS: Message Format Standardized JCATS Sim Code ACME Sim Code Vendor A API for DIS Vendor B API for DIS Standard Format Messages Network Vendor C API for DIS ONE-SAF Sim Code

48. HLA: API Standardized JCATS Sim Code ACME Sim Code HLA RTI API HLA RTI API RTI Vendor-Specific Format Messages Network HLA RTI API ONE-SAF Sim Code While the API is standard, implementations of the HLA RTI API from different vendors will produce different format messages

49. DIS: API • The implications of this are that while HLA has a standardized API, RTIs from different vendors can’t typically talk to each other. This makes changing vendors easy, but makes getting RTIs from different vendors talking to each other hard--you need to use a gateway • DIS in contrast makes changing vendors hard (since it involves changing the API your simulation code uses) but talking between vendors easy (since all the messages on the wire are in the same format) • The lack of an API can help when using unusual languages, such as Objective-C, C#, and Python. Since there’s no API, just make one up. As long as they produce standard messages, it doesn’t affect anyone else

50. DIS: PDU Format • Remember, all this information is being sent in binary format in (typically) a UDP packet • The exact format that an ESPDU must have on the wire is specified in the DIS standard. This includes byte order • For example, the EntityID field starts 12 bytes into the ESPDU message, is in the order (site, application, entity), and each field entry is 16 bits long in network byte order, and unsigned