Benchmarking Results from the Tycho Performance Tests

1. Benchmarking Results from the Tycho Performance Tests Matthew Grove DSG Seminar 27rd January 2006

2. Outline • Introduction. • Changes made to the Tycho design from last time (June 2005). • Example Tycho setup. • Tycho benchmark motivations and methodology. • Some performance results. • Summary.

3. Introduction • Tycho is a reference implementation of a combined extensible wide-area messaging framework with a built in distributed registry. • Tycho is deigned to reuse existing infrastructure where ever possible. • Tycho aims to simplify and speed application development by freeing developers from the need to use a combination of middleware to provide service discovery and messaging.

4. What’s new for Tycho • We have changed the name from jGMA to Tycho, this lets us focus on the presenting the strengths of Tycho, removing the dependency to explain our motivation in terms of GMA and the Grid. • The architecture has remained the same but we have completed the initial implementation.

5. Tycho’s Architecture

6. Example Tycho Setup

7. Implementation Changes • There are 24 items in the change log, the API has remained the same, highlights are: • The IRC VR-connector has been completed. • Extended the HTTP handler to provide VR boot strapping using a known web site. • Support for JDBC data-stores (like HSQLDB). • Added support for SQL keywords ‘LIKE’ and ‘LIMIT’. • Added caching for inter-VR responses.

8. Tycho Benchmarks We performed two rounds of benchmarking: • Message Passing: These tests measured the performance of inter-client and inter-mediator communications for Tycho and NaradaBrokering. • Virtual Registry: These tests measured the performance of the VR (query routing and processing) in a range of different configurations: • These results were compared to other tests which measured performance of Hawkeye, R-GMA and MDS2 and 4.

9. Message Passing Tests (A) • We measured the latency of communication for LAN and simulated WAN messaging between two clients varying message size (ping-pong tests). • Then we measured the affect on scalability of message latency using many producers publishing to a single consumer and single consumer consuming from multiple producers. • We used Starbug to run the tests.

10. Test A - Configuration

11. Ping-Pong - Latency Latency vs Message Length

12. Ping-Pong - Bandwidth Bandwidth vs Message Length

13. Scalability Results - Producers Bandwidth vs Message Length

14. Scalability Results - Consumers Bandwidth vs Message Length

15. Message Passing Tests - Summary • Tycho achieves comparable or lower latency than NaradaBrokering in all of these tests. • The performance of both systems does not drop off after a producer or consumer has become saturated. • NaradaBrokering needs the JVM heap size to be increased as the number of clients increases (due to internal buffers): • Tycho used the default heap for all of the tests.

16. Virtual Registry Tests (B) • We used 4 different configurations to measure the performance of the Tycho VR. • For each test we varied one or more aspect of the configuration and measured the effect on client query latency: • Number of records in a mediator. • Number of simultaneous client queries. • Numbers of mediators in the VR. • Different VR-interconnects between mediators. • In order to make some comparisons between Tycho and some other systems we attempted to reproduce a test used in a related paper.

17. Configuration

18. Variables • The comparison tests with the other systems used the test 2 configuration. Test Parameter Matrix; the components that are varied in each test are highlighted. S1 queries select a single record, S2 queries select every record.

19. Number of Records Response Time vs Number of Records

20. Simultaneous Client Queries Response Time vs Number of Clients

21. Number of Mediators Response Time vs Number of Mediators

22. VR-Interconnects Response Time vs Number of Mediators

23. Virtual Registry Tests - Summary • The Simple store performs best for small numbers of records, but HSQLDB and MySQL scale better with response size. • The response time increases linearly with number of clients. • For routing queries in the VR, IRC has better performance than HTTP. • As response size increases HTTP performs much better than IRC, this is due to the IRC code having to send multiple messages.

24. Comparison Tests • The tests measure the impact of increasing the number of clients simultaneously querying the registry. • We had to use 5 compute nodes instead of 8 to make the test as close a match as possible. • Each system has different sized responses which we matched by adding padding to Tycho registry information. • The main difference is CPU (2.8Ghz vs 1.1Ghz).

25. Comparison Tests (2K) Response Time vs Number of Clients (2Kbyte Responses)

26. Comparison Tests (10K) Response Time vs Number of Clients (10Kbyte Responses)

27. Comparison Tests (19K) Response Time vs Number of Clients (19Kbyte Responses)

28. VR Comparison Tests – Summary • Tycho shows linear increase in response times regardless of message size and number of clients. • Tycho had lower response times than R-GMA, MDS4 and MDS2 without caching. • MDS2 with caching is basically OpenLDAP - which is mature and performs well. • Hawkeye performs better than Tycho for less than 100 clients, but it does not scale well with larger number of clients.

29. Future Work • Some performance improvements: • Caching of local mediator queries to reduce response times. • The Simple store could be improved by making it more intelligent. • We could use a hybrid VR-interconnect to use IRC for query routing and HTTP for transporting large responses. • More functionality can be added to provide services that are more advanced: • HTTPS and SSL Sockets for secure comms. • Support for transactions. • Producers and consumers for computational steering.

30. Links • Project Web page: • http://dsg.port.ac.uk/projects/tycho/ • The DSG Web page: • http://dsg.port.ac.uk/