Building High Throughput, Multi-threaded Servers in C#/.NET

1. Building High Throughput, Multi-threaded Servers in C#/.NET David Buksbaum

2. Who Am I? • Wall Street Wage Slave • Developer – I still write code • Tech-Lead – My bugs are harder to find • Pointy Haired Manager – I live by the KISS principle

3. What have I built? • User Time to Near Real Time Systems • User Time: event time is based on user interaction • Near Real Time: event timing is as fast as possible without guarantee • Server to Server • Client - Server • Small Scale: 1-25 concurrent users • Large Scale: 250+ concurrent users • Distributed Systems (now called SOA)

4. Agenda • C# Crash Course • Application Servers • Concepts • Connection Stack • Conclusion • Further Concepts

5. C# Crash Course • [external]

6. Application Servers • What I mean by App Servers • A stateful server to service client requests for high-level business functions in near user time • RRE Messaging Model (request, response, event) • Middle Tier Intra-Net Systems • May contain 1 or more logical servers • What I do not mean by App Servers • Not the back-end infrastructure systems • Broadcast Servers (pricing, curve servers, etc…) • Bus Servers (messaging platforms like Jabber) • Database Servers

7. Concepts • Endian-ness • Does not matter unless your network is exactly ½ little and ½ big • Pick the model used more and make the other convert • Only the smaller group will have the performance hit • Most clients to your servers will be little-endian • Many middle-tier systems are Intel based • Solaris is being moved further back or out due to costs • Typical configurations will be WinTel clients with either WinTel or LinTel servers

8. Concepts – cont. • Threading • Your server functionality will determine threading or not • Guideline: If you threads on WinTel, then thread count target should be (2 * CPU’s) + 1 • Explained in detail by Jeffery Richter in "Programming Server Side Applications for Windows 2000" • Exceptions: Lots of I/O bound threads for example, so use your judgment • Burst servers should be guided towards single threaded per connection (min concurrency) • Servers with minimal state, fast response time, and no persistent connections • HTTP 1.0 defaulted to a burst server (no keepalives) • What is a burst server? echo, time, etc… • Pattern: UDP or short lived TCP

9. Concepts – cont. • Threading – cont. • Stateful servers need threads – so use them • State operations are slower than network operations, so minimize it in the network thread • If you use threads, use thread pools • Creating threads are expensive, so don’t do it • .NET has a built in thread pool • System.Threading.ThreadPool • ThreadPool.QueueUserWorkItem(new WaitCallback(myThreadFunc), myContextObject); • In large systems, use multiple specialized pools • Different Priorities, Order Guarantees, etc…

10. Concepts – cont. • Queues • Rule 1: If you use threads – you use queues • Queues break performance coupling between two independent tasks • Task 1: Read / Write Data From / To Network • Task 2: Do Something with the Data • Queues allow for complex tasks to be sub-divided into simpler sub-tasks • These sub-tasks can be worked on concurrently (parallelism or grids) or synchronously • Reusability of sub-tasks between different parent tasks • eg: fetch from database, calculate distance between two points on a map, execute a trade • Simpler code is easier to build, debug, & maintain

11. Concepts – cont. • Synchronicity • Blocking Synchronous I/O • Read or Write w/o timeouts • Select w/o timeouts • Non-Blocking Synchronous I/O • Read or Write w/ timeouts • Select w/ timeouts • Polling • Asynchronous I/O • BeginReceive or BeginSend • I/O Completion Ports & Overlapped I/O

12. Concepts – cont. • Synchronicity – cont. • Which do you use? • Rule 2: Never, Ever, Use Blocking I/O • Server Threading model defines synchronicity strategy • Thread per connection  Non-Blocking Synchronous I/O • Thread pools  Asynchronous I/O • Clients can usually use Non-Blocking Synchronous I/O in a dedicated thread

13. Concepts – cont. • State • Definition • Data and Methods encapsulating a logic block related information • An App Server has different types of state • Server State • State managed on a per logical server basis • Used anywhere the server needs to self administer itself • Usually the container of all other state in the server • Connection State • State managed on a per user connection basis • Used only in a context where the connection needs to be known • Reading and Writing to the network • Verifying credentials

14. Concepts – cont. • State – cont. • Business State • Contains all the cached data used to support the business functionality • Provides access to all non-cached data transparently • eg: BusinessState.GetAllAccounts() may go to the cache or some external store. Follow good OO procedures here. • State Rules • Minimize surface area (less public methods), but no side effects • Everything public MUST be reentrant or thread-safe • Encapsulate – no public variables • Design for threading

15. Concepts – cont. • State – cont. • Rule 3: State will dominate server design & performance • Questions to ask when defining state • Who needs access to it? • When do they need access to it? • Special Cases – frequent & pervasive • Logon, Logoff • Credentials Check

16. Concepts – cont. • State – cont. • Techniques • Synchronized Collections • Equal punishment to readers & writers • Reader / Writer Locks • Favors reads over writes • External Data Store • Push the locking burden on an expert • Embedded • All state is passed between the client and server in every message • eg: URL’s, Cookies, etc… • Tokens • A token for looking up the state is passed between the client & server in every message • Usually used to add state to a stateless design (eg: HTTP) • Hybrids • Combinations of the above • eg: Tokens, Embedded, & External are used by many web servers

17. Concepts – cont. • Performance • User Time vs. Actual Time • User Time is the time between the initiation of a process by the user on the client and when the result of that action is visible to the user • Actual Time is the actual time it takes to perform the given action on the server from receipt of a request to the sending of a response • Throughput of the server is often defined in terms of actual time (messages per second) • Rule 4a: The user does not care about Throughput • Rule 4b: User Time is all that matters in interactive systems

18. Concepts – cont. • Performance – cont. • Users vs. Actions • How many concurrent users? • How many concurrent actions? • Similar to Time vs. Memory tradeoff • eq: 1 user with 100 actions or 100 users with 1 action

19. Concepts – cont. • Case Example: Massive Multiplayer Online Role-Playing Games (MMORPG) – Everquest 2 • 2500 concurrent users per server • Each user requires significant state management • Location in the game world • Character statistics • Inventory • Relationships to other characters, non-player characters, organizations • State is a hybrid combining tokens & external data store • User time is between 11 & 140 ms, with the bulk of the users being at around 100 ms – derived from frame rates • Actual time peak is 25000 / second (2500 * 10) • Assumes all 2500 are not just concurrent, but active

20. Questions?

21. Architecture @ 50,000 Feet User Time Queue P1 Network Thread Pool P2 P3 Task Thread Pool Network Initial Response Time Actual Time • P1, P2 and P3 represent the areas that have the possibility for high contention due to locking or congestion • A fast initial response time, followed by incremental updates (streaming or events), can give the illusion of faster user time

22. Protocol Stack @ 50,000 Feet Application • Wire • TCP, UDP, or other network protocol • Transport • Envelope to encapsulate the application message • Fields: • Length, Checksum, Flags, Data • Application • Message ID • Message Data Transport Wire

23. Wire Protocols • UDP • Not reliable • Small blocks of data • Connectionless • TCP • Reasonably reliable • Any reasonable amount of data • No information about content

24. Wire Protocols – cont. • Message Queues • Queue to Queue communications • Each end point has a queue • Messages are moved from one queue to another • A centralized queue can have 1 or more listeners in non-exclusive mode • A centralized queue in exclusive mode can have auto-failover • Reliability level is controllable • Products • JMS – Java Messaging Service • .NET bindings exists • Derivative works like Tibco’s EMS is a very performant version • Deployable on top of most network infrastructures • MSMQ • MS has .NET bindings in the System.Messaging namespace • Not available on all Windows platforms by default • Many organizations have restricted this product due to tight integration with the Windows platform

25. Wire Protocols – cont. • Message Queues – cont. • Alternative Systems • Book: Message Passing Server Internals by Bill Blunden • Contains the design and implementation of Bluebox; an enterprise ready messaging queuing system in Java Others • Tibco Rendezvous • Basically a serialized hash table (key, item) • Jabber • Java framework originally geared for Instant Messaging, but now supporting robust bus based communications • Still XML based, so not a great fit for high performance

26. Wire Protocols – cont. • Why not UDP? • Packet size limitations • Connectionless • Reliability • Why not TCP? • Reasonable reliability is not always good enough • Performance in broadcast scenarios (1 to many listeners)

27. Wire Protocols – cont. • Why messaging systems? • Reliable & recoverable queues are great for when the message absolutely must get to its destination • Trading systems • Banking systems • Transactional • Queues can be used as the back bone of a system that support distributed functionality with distributed transactions (XA) • Your app sever can participate in a transaction that spans all systems between the client and the database

28. Wire Protocols – cont. • Why messaging systems? – cont. • Support tools • Many messaging systems provide rich tools for sniffing and/or capturing the messages • Captured messages can be used for playback • Robust infrastructure • Messaging systems have been built by networking experts to accommodate a number of different infrastructures, topologies, and geographical idiosyncrasies • You have someone to scream at when it goes wrong • Rule 5: If you can, use a commercial transport

29. Transport Protocol • Why do you need a protocol between the application and the wire? • Wire Insulation: The wire may change • TCP  Message Queues • UDP  TCP • The application protocol may not play nicely into a wire protocol • MB sized files • Streaming data • XML • Variable length records

30. Transport Protocol – cont. • Remember our goal at the wire? • Get the data off the wire and onto the queue as fast as possible • We need information as we get the data off the wire to speed up processing • Length of the total message • Message data • Smallest transport protocol (4 bytes) • 4 byte length | n byte data block

31. Transport Protocol – cont. • What else do we need? • We have: • Length • Data • We need (loosely used): • Message type information • Message ID • Sequencing information • Beginning • Middle • End • Flags or Options

32. Transport Protocol – cont. • DIME Lite • Protocol created by Microsoft to allow for easier transmission of binary data over SOAP • Can be used independently of SOAP as a unidirectional transport block

33. Transport Protocol – cont.

34. Application Protocol • What is it? • It is anything meaningful to your servers business logic • Hash Tables • Schema’s can be used to define required keys • eg: Default schema may mandate key 100 in all messages • Key 100 might be the schema id for the rest of the message • Allows for complex record based pseudo-objects with mandatory, optional, default fields • Compatible cross language / platform • Efficient • Client objects can wrap hash table

35. Application Protocol – cont. • What is it? – cont. • Serialized Objects • Complex message object or hierarchy of objects • Very rich distributed object functionality • Limited to .NET environment and possible subset of languages within the .NET world • eg: Java RMI, some transport for .NET Remoting • SOAP, Raw XML, or other text based transport • Schema or well defined set of tag / value pairs • String based • Changes require new memory block for the whole message since strings are immutable • Slow to parse (~350 / sec) [validating, msxml] • Even the fastest is less than 1000 / sec

36. Application Protocol – cont. • What is it? – cont. • Hybrids • Serialized objects encoded in hash tables • Flexibility of hash tables • Expandable through new keys • Cross platform / language support • Versionable objects • Main limitation is the primitive types supported • Requires well known schemas

37. Complete Example Simple RPC Client / Server

38. Questions?

39. Further Thoughts • Lock / Wait – Free Data Structures • Used in graphics pipelines and media streaming • Reduces contention • Aimed at small # of writers and any number of readers (where you would use Reader / Writer locks) • Protocols • Current trends: • Ignore wire performance and go for B2B protocols (SOAP, et al) • Text based protocols, such as ASCII encoded key / value pairs • Serialization • No simple way to work cross language / platform • Primitives are not always portable • Most research is focused on B2B

40. Further Thoughts – cont. • Memory Pools • Pre-allocate pools of objects that can are reused rather than freed

41. Rules Summary • Rule 1: If you use threads – you use queues • Rule 2: Never, Ever, Use Blocking I/O • Rule 3: State will dominate server design & performance • Rule 4a: The user does not care about Throughput • Rule 4b: User Time is all that matters in interactive systems • Rule 5: If you can, use a commercial transport

42. References & Links • HTTP 1.0 – RFC 1945 • http://www.ietf.org/rfc/rfc1945.txt • HTTP 1.1 – RFC 2616 • http://www.ietf.org/rfc/rfc2616.txt • Echo Protocol – RFC 862 • http://ftp.rfc-editor.org/in-notes/rfc862.txt • Time Protocol – RFC 868 • http://ftp.rfc-editor.org/in-notes/rfc868.txt • Superseded by Network Time Protocol (NTP) • http://en.wikipedia.org/wiki/Network_time_protocol

43. References & Links – cont. • Endianness • http://en.wikipedia.org/wiki/Endian • .NET Thread Pools • MSDN Docs for ThreadPool Class • http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemthreadingthreadpoolclasstopic.asp • Stephen Toub’s Managed Thread Pool • http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=bf59c98e-d708-4f8e-9795-8bae1825c3b6 • Enhanced version of Stephen Toub’s Thread Pool • http://www.codeproject.com/csharp/SmartThreadPool.asp

44. References & Links – cont. • Overlapped I/O • http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/synchronization_and_overlapped_input_and_output.asp • I/O Completion Ports • http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/i_o_completion_ports.asp

45. References & Links – cont. • Message Queues • Message Passing Server Internals by Bill Blunden • http://www.amazon.com/gp/product/0071416382/103-6338215-6207008?v=glance&n=283155 • Jabber • http://www.jabber.org/ • DIME • http://msdn.microsoft.com/library/en-us/dnglobspec/html/draft-nielsen-dime-02.txt

46. Lock / Wait Free Data Structures • http://www.audiomulch.com/~rossb/code/lockfree/ • http://appcore.home.comcast.net/

47. Extra Slides

48. using System; using System.Text; using System.Net; using System.Net.Sockets; namespace EchoServer { class Program { static void Main(string[] args) { try { . . . } catch(Exception x) { Console.WriteLine(x.ToString()); } } } } UDP Echo Server – the shell

49. UDP Echo Server – the code try {// create a udp server on port 8666 UdpClient server = new UdpClient(8666); // create an end point that maps to any ip and any port IPEndPoint endPoint = new IPEndPoint(IPAddress.Any, 0); // receive data from our end point byte[] data = server.Receive(ref endPoint); // convert from bytes to string string msg = Encoding.ASCII.GetString(data); // display the message and who it is from Console.WriteLine( "Received message from {0} ==> {1}", endPoint.ToString(), msg); }

50. UDP Echo Client in Java public static void main(String[] args) { try { // set up our locals InetAddress host = InetAddress.getByName("127.0.0.1"); int port = 8666; String msg = "Hello World from Java"; // get the bytes for string byte[] data = msg.getBytes(); // create a udp client DatagramSocket client = new DatagramSocket(); // create our data packet DatagramPacket packet = new DatagramPacket(data, data.length, host, port); // send the data client.send(packet); } catch(Exception x) { System.out.println(x.toString()); }; }