Overview of the OMG Data Distribution Service

1. Overview of the OMGData Distribution Service Douglas C. Schmidt & Jeff Parsons {schmidt,parsons}@dre.vanderbilt.edu http://www.dre.vanderbilt.edu/~schmidt/ Professor of EECS Vanderbilt University Nashville, Tennessee

2. Past R&D Successes: Platform-centric Systems From this design paradigm… Air Frame • Legacy DoD systems are designed to be: • Stovepiped • Proprietary • Brittle & non-adaptive • Expensive to develop & evolve • Vulnerable Nav WTS AP FLIR GPS IFF Cyclic Exec Problem: Small changes can break nearly anything & everything

3. Utility “Curve” “Broken” “Works” Utility Resources “Harder” Requirements Past R&D Successes: Platform-centric Systems …and this operation paradigm… • Real-time QoS requirements for platform-centric DoD systems: • Ensure end-to-end QoS, e.g., • Minimize latency, jitter, & footprint • Bound priority inversions • Allocate & manage resources statically Problem: Lack of any resource can break nearly anything & everything

4. Past R&D Successes: Network-centric Systems …to this design paradigm… • Today’s leading-edge DoD systems are designed to be: • Layered & componentized • More standard & COTS • Robust to expected failures & adaptive for non-critical tasks • Expensive to evolve & retarget • Vulnerable Air Frame AP Nav WTS Event Channel Replication Service GPS IFF FLIR Object Request Broker Problem: Unanticipated changes can break many things

5. Middleware Middleware Past R&D Successes: Network-centric Systems …and this operational paradigm… Applications Applications Interceptor Interceptor Sys Cond Sys Cond Sys Cond Sys Cond Mechanism & Property Managers { } Workload & Replicas Workload & Replicas Connections & priority bands Connections & priority bands Local Resource Managers Local Resource Managers QoS Contracts QoS Contracts CPU & memory CPU & memory Network latency & bandwidth Network latency & bandwidth Endsystem Endsystem Adaptive Real-time Middleware

6. Desired Utility Curve “Working Range” Utility Resources “Softer” Requirements Past R&D Successes: Network-centric Systems …and this operational paradigm… Problem: Network-centricity is an afterthought in today’s systems

7. Case Study: QoS-enabled Publish/Subscribe Technologies for Tactical Information Management Feedback & Control Coordination Of Multiple UAVs Dynamic Mission Replanning Image Processing & Tracking DARPA PCES Capstone demo, 4/14/05, White Sands Missile Range

8. Asynchronous Event Handling Asynchronous Transfer of Control Physical Memory Access Synchronization Memory Management Scheduling Case Study: QoS-enabled Publish/Subscribe Technologies for Tactical Information Management Feedback & Control Coordination Of Multiple UAVs Dynamic Mission Replanning Image Processing & Tracking Aspect Languages Modeling Tools Real-time JVMs Model Checking Real-time ORBs

9. Real-time Info to Cockpit Real-time Event Service Object Request Broker Tactical Network & RTOS Limitations with Demo’d PCES Information Management Technologies • Shooter information management was based on platform-centric Real-time CORBA & Real-time CORBA Event Service • Well-suited for point-to-point & staticpub/sub command processing over wireline networks • e.g., statically provisioned QoS policies • Poorly suited for dynamic pub/sub info dissemination to multiple nodes in MANETs • e.g., too many layers, excessive time/space overhead, inflexible QoS policies & pub/sub model, etc. Problem: Net-centricity is afterthought in platform-centric technologies

10. Limitations with Demo’d PCES Information Management Technologies Track Processing • C2 & C4ISR information management based on J2EE & Java Messaging Service (JMS) • Best suited for operational enterprise environments • e.g., large data centers connect via wireline networks • Poorly suited for tactical environments • e.g., lack of QoS policies & RTOS integration, extremely high time/space overhead Java Messaging Service J2EE Middleware Enterprise Network & OS Problem: Enterprise technologies are ill suited for tactical systems

11. Our R&D Goal: Evaluate QoS-enabled Info Brokering for Tactical Systems • Solutions must function properly where • Communication bandwidth is limited/variable • Connectivity is intermittent • Connections are noisy • Processing & storage capacity are limited • Power & weight limits affect usage patterns • Unanticipated workflows are common • Dynamic network topology & membership changes are frequent • Ideally, solutions should be COTS, standard, & integrate with heterogeneous legacy systems Goal is “better than best-effort,” subject to resource constraints…

12. Overview of the Data Distribution Service (DDS) • DDS is an highly efficient OMG pub/sub standard • e.g., fewer layers, less overhead Topic R Data Writer R Data Reader R Publisher Subscriber RT Info to Cockpit & Track Processing DDS Pub/Sub Infrastructure Tactical Network & RTOS

13. Overview of the Data Distribution Service (DDS) • DDS is an highly efficient OMG pub/sub standard • e.g., fewer layers, less overhead • DDS provides meta-events for detecting dynamic changes Topic R NEW TOPIC Data Writer R Data Reader R NEW SUBSCRIBER Publisher Subscriber NEW PUBLISHER

14. Overview of the Data Distribution Service (DDS) • DDS is an highly efficient OMG pub/sub standard • e.g., fewer layers, less overhead • DDS provides meta-events for detecting dynamic changes • DDS provides policies for specifying many QoS requirements of tactical information management systems, e.g., • Establish contracts that precisely specify a wide variety of QoS policies at multiple system layers Topic R HISTORY RESOURCE LIMITS Data Writer R S1 Data Reader R S2 S3 S4 S5 Publisher Subscriber S6 S7 LATENCY X S7 S7 S6 S5 S4 S3 S2 S1 COHERENCY RELIABILITY

15. Overview of the Data Distribution Service (DDS) • DDS is an highly efficient OMG pub/sub standard • e.g., fewer layers, less overhead • DDS provides meta-events for detecting dynamic changes • DDS provides policies for specifying many QoS requirements of tactical information management systems, e.g., • Establish contracts that precisely specify a wide variety of QoS policies at multiple system layers • Move processing closer to data DESTINATION FILTER Topic R Data Writer R S1 Data Reader R SOURCE FILTER S2 S3 S4 S5 Publisher Subscriber S6 S7 TIME-BASED FILTER

16. DDS Architectural Elements Data-Centric Publish-Subscribe (DCPS) • The lower layer APIs apps can use to exchange topic data with other DDS-enabled apps according to designated QoS policies Data Local Reconstruction Layer (DLRL) • The upper layer APIs that define how to build a local object cache so apps can access topic data as if it were local • DDS spec only defines policies & interfaces between application & service • Doesn’t address protocols & techniques for different actors implementing the service • Doesn’t address management of internal DDS resources

17. DDS Application Architecture { The Application Application Application Application Application DLRL DLRL DLRL DLRL DCPS Communication

18. Domain 3 Domain 2 Domain 1 DDS Domains & Domain Participants • The domain is the basic construct used to bind individual applications together for communication • Like a VPN 2 3 1 1 Node Node Node 3 2 1 1 Node Node Node DomainParticipant

19. DCPS Entities • Data can be accessed in two ways • Wait-based (synchronous calls) • Listener-based (asynchronous callbacks) • Sophisticated support for filtering • e.g., Topic, Content-FilteredTopic, or MultiTopic • Configurable via (many) QoS policies DCPS Entities include • Topics • Typed data • Publishers • Contain DataWriters • Subscribers • Contain DataReaders • DomainParticipants • Entry points Topic Topic Topic Domain Participant Data Writer Data Reader Data Reader Data Reader Data Writer Data Writer Publisher Subscriber Publisher Subscriber Data Domain

20. QoS Policies Supported by DDS • DCPS entities (i.e., topics, data readers/writers) configurable via QoS policies • QoS tailored to data distribution in tactical information systems • Request/offered compatibility checked by DDS • DEADLINE • Establishes contract regarding rate at which periodic data is refreshed • LATENCY_BUDGET • Establishes guidelines for acceptable end-to-end delays • TIME_BASED_FILTER • Mediates exchanges between slow consumers & fast producers • RESOURCE_LIMITS • Controls resources utilized by service • RELIABILITY (BEST_EFFORT, RELIABLE) • Enables use of real-time transports for data • HISTORY (KEEP_LAST, KEEP_ALL) • Controls which (of multiple) data values are delivered • DURABILITY (VOLATILE, TRANSIENT, PERSISTENT) • Determines if data outlives time when they are written • … and many more …

21. Best Effort Reliability QoS in DDS QoS Reliability = BEST_EFFORT Subscriber Subscriber Publisher Subscriber Notification of new data objects timeout deadline time-based filter no notification notification time • Very predictable • Data is sent without reply • Publishers and subscribers match and obey QoS Deadline policy settings • Time-based Filter QoS policy gives bandwidth control

22. Reliable QoS in DDS QoS Reliability = RELIABLE Topic R history Data Writer R S1 Data Reader R S2 S3 S4 S5 Publisher Subscriber S6 S7 S7 S6 S5 S4 S3 S2 S1 Ordered instance delivery is guaranteed

23. Tradeoff Between Reliability & Determinism QoS Reliability = BEST_EFFORT • Can’t have reliability and determinism. • Best effort mode for streaming data. • No retries of dropped packets. • Minimizes latency. • Reliable mode for commands & events. • Retry dropped packets until timeout. • Every packet received in order. • Speculative cache mechanism. • DDS reliability is tunable. • Can be adjusted per subscription. X QoS Reliability = RELIABLE X X

24. Deadline Destination Order Durability Entity Factory Group Data History Latency Budget Lifespan Liveliness Ownership Ownership Strength Partition Presentation Reader Data Lifecycle Reliability Resource Limits Time-Based Filter Topic Data Transport Priority User Data Writer Data Lifecycle All QoS Policies in DDS

25. Single vs. Multiple Domain Systems

26. Data Writers & Publishers • Data Writers are the primary access point for an application to publish data into a DDS data domain • The Publisher entity is a container to group together individual Data Writers • User applications • Associate Data Writers with Topics • Provide data to Data Writers • Data is typically defined using OMG IDL • It can therefore be as strongly or weakly typed as you desire

27. Data Readers & Subscribers • A Data Reader is the primary access point for an application to access data that has been received by a Subscriber • Subscriber is used to group together Data Readers • Subscribers & Data Readers can have their own QoS policies • User applications • Associate Data Readers with Topics • Receive data from Data Readers using Listeners (async) or Wait-Sets (sync)

28. Types & Keys • A DDS Type is represented by a collection of data items. • e.g. “IDL struct” in the CORBA PSM struct AnalogSensor { string sensor_id; // key float value; // other sensor data }; • A subset of the collection is designated as the Key. • The Key can be indicated by IDL annotation in CORBA PSM, e.g., #pragma DDS_KEY AnalogSensor::sensor_id • It’s manipulated by means of automatically-generated typed interfaces. • IDL compiler may be used in CORBA PSM implementation • A Type is associated with generated code: • AnalogSensorDataWriter // write values • AnalogSensorDataReader // read values • AnalogSensorType // can register itself with Domain

29. Topics • A DDS Topic is the connection between publishers & subscribers. • A Topic is comprised of a Name and a Type. • Name must be unique in the Domain. • Many Topics can have the same Type. • Provision is made for content-based subscriptions. • MultiTopics correspond to SQL join • Content-FilteredTopics correspond to SQL select. Domain ID 35 Topic name “MySensor” Type name “AnalogSensor” [ key ] string sensor_id value float

30. Topic-Based Publish/Subscribe Pressure • DataWriter is bound (at creation time) to a single Topic • DataReader is bound (at creation time) with one or more topics (Topic,ContentFilteredTopic, or MultiTopic) • ContentFilteredTopic & MultiTopic provide means for content-based subscriptions & “joins”, respectively Temperature

31. application Data Reader Data Reader Data Reader QoS Topic 1 Topic 2 Topic n QoS subscriber subscriber QoS Subscription = Topic + DataReader

32. Content-based Subscriptions • ContentFilteredTopic (like a DB-selection) • Enables subscriber to only receive data-updates whose value verifies a condition. • e.g. subscribe to “Pressure” of type AnalogData • where “value > 200” • MultiTopic (like a DB-join operation) • Enables subscription to multiple topics & re-arrangement of the data-format • e.g. combine subscription to “Pressure” & “Temperature” & re-arrange the data into a new type: struct { float pres; float temp; };

33. DDS Content-Filtered Topics Topic Instances in Domain Instance 1 Value = 249 Instance 2 Value = 230 Content-Filtered Topic Instance 3 Value = 275 Topic Instance 4 Value = 262 Filter Expression: Value > 260 Value = 258 Instance 5 Instance 6 Value = 261 Expression Params Instance 7 Value = 259 Filter Expression and Expression Params determine which Topic instances the Subscriber receives.

34. DDS MultiTopic Subscriptions Topic Topic Topic Topic MultiTopic Domain Participant Domain Participant Data Reader Data Reader Data Reader Data Reader Subscriber Subscriber Subscriber MultiTopics can combine, filter, and rearrange data from multiple Topics.

35. Example: Create Domain Participant • DomainParticipantobject acts as factory for Publisher, Subscriber, Topic and MultiTopic entity objects // used to identify the participant DomainId_t domain_id = ...; // get the singleton factory instance DomainParticipantFactory_var dpf = DomainParticipantFactory::get_instance (); // create domain participant from factory DomainParticipant_var dp = dpf->create_participant (domain_id, PARTICIPANT_QOS_DEFAULT, NULL);

36. Example: Create Topic …… // register the data type associated with the topic FooDataType foo_dt; foo_dt.register_type (dp, “Foo”); // create a topic Topic_var foo_topic = dp->create_topic (“Foo_topic”, //topic name “Foo”, // type name TOPIC_QOS_DEFAULT, // Qos policy NULL); // topic listener

37. Example: Create Subscriber and DataReader …… // create a subscriber from the domain participant SubscriberQos sQos; dp->get_default_subscriber_qos (sQos); Subscriber_var s = dp->create_subscriber (sQos, NULL); // create a data reader from the subscriber // and associate it with the created topic DataReader_var reader = s->create_datareader (foo_topic.in (), DATAREADER_QOS_DEFAULT, NULL); FooDataReader_var foo_reader = FooDataReader::_narrow (reader.in ());

38. Example: Create Publisher and DataWriter …… // create a publisher from the domain participant PublisherQos pQos; dp->get_default_publisher_qos (pQos); Publisher_var p = dp->create_publisher (pQos, NULL); // create a data writer from the publisher // and associate it with the created topic DataWriter_var writer = p->create_datawriter (foo_topic.in (), DATAWRITER_QOS_DEFAULT, NULL); // narrow down to specific data writer FooDataWriter_var foo_writer = FooDataWriter::_narrow (writer); // publish user-defined data Foo foo_data; foo_writer->write (foo_data);

39. How to Get Data (Async Listener-based) Listener_var subscriber_listener = new MyListener(); foo_reader->set_listener(subscriber_listener); MyListener::on_data_available(DataReader reader) { FooSeq_var received_data; SampleInfoSeq_var sample_info; reader->take(received_data.out (), sample_info.out (), ANY_SAMPLE_STATE, ANY_LIFECYCLE_STATE); // Use received_data …… }

40. How to Get Data (Sync Wait-based) Condition_var foo_condition = reader->create_readcondition(ANY_SAMPLE_STATE, ANY_LIFECYCLE_STATE); WaitSet waitset; waitset->attach_condition(foo_condition); ConditionSeq_var active_conditions; Duration_t timeout = {3,0}; waitset->wait(active_conditions.out (), timeout); ... FooSeq_var received_data; SampleInfoSeq_var sample_info; reader->take_w_condition(received_data.out (), sample_info.out (), foo_condition); // Use received_data

41. Benchmark Scenario • Two processes perform IPC in which a client initiates a request to transmit a number of bytes to the server along with a seq_num (pubmessage), & the server simply replies with the same seq_num (ackmessage). • The invocation is essentially a two-way call, i.e., the client/server waits for the request to be completed. • The client & server are collocated. • DDS & JMS provides topic-based pub/sub model. • Notification Service uses push model. • SOAP uses p2p schema-based model.

42. Testbed Configuration • Hostname blade14.isislab.vanderbilt.edu • OS version (uname -a) Linux version 2.6.14-1.1637_FC4smp (bhcompile@hs20-bc1-4.build.redhat.com) • GCC Version g++ (GCC) 3.2.3 20030502 (Red Hat Linux 3.2.3-47.fc4) • CPU info Intel(R) Xeon(TM) CPU 2.80GHz w/ 1GB ram

43. Test Parameters • Average round-trip latency & dispersion • Message types: • sequence of bytes • sequence of complex type • Lengths in powers of 2 • Ack message of 4 bytes • 100 primer iterations • 10,000 stats iterations // Complex Sequence Type struct Inner { string info; long index; }; typedef sequence<Inner> InnerSeq; struct Outer { long length; InnerSeq nested_member; }; typedef sequence<Outer> ComplexSeq;

44. Roundtrip Latency Results (1/2)

45. Roundtrip Latency Results (2/2)

46. Roundtrip Latency Results Analysis • From the results we can see that DDS has significantly better performance than other SOA & pub/sub services. • Although there is a wide variation in the performance of the DDS implementations, they are all at least twice as fast as other pub/sub services.

47. Encoding/Decoding Benchmarks • Measured overhead and dispersion of • encoding C++ data types for transmission • decoding C++ data types from transmission • DDS3 and GSOAP implementations compared • Same data types, platform, compiler and test parameters as for roundtrip latency benchmarks

48. Encoding/Decoding Results (1/2)

49. Encoding/Decoding Results (2/2)

50. Encoding/Decoding Results Analysis • Slowest DDS implementation is compared with GSOAP. • DDS is faster. • Almost always by a factor of 10 or more. • GSOAP is encoding XML strings. • Difference is larger for byte sequences. • DDS implementation has optimization for byte seq. • Encodes sequence as a single block – no iteration. • GSOAP always iterates to encode sequences. • Jitter discontinuities occur at consistent payload sizes.