Using Java for a High Performance Infrastructure Service

1. Using Java for a High Performance Infrastructure Service Tom Dimock Cornell University December 9, 2002

2. Agenda • Some Ancient History • In which the Cornell environment will be described • Some Recent History • In which the gory details of our melt-down will be revealed • The Re-write • In which the architecture and implementation will be discussed • Epilog • In which we will disclose the success or failure of the project Using Java for a High Performance Infrastructure Service

3. Ancient History • Project Mandarin • Began in the early 90’s • Allowed secure access to mainframe data • Not standards based, because there were no relevant standards • Deployed at more than 20 Universities • Performance was nothing to brag about, but beat the old off-line systems Using Java for a High Performance Infrastructure Service

4. Mandarin Architecture IBM Mainframe Mac OS ProbeMaker CUSSP CUSSP Natural Dispatcher KFRONT CUSOCKET TCP/IP TCP/IP Windows C Visual Basic Using Java for a High Performance Infrastructure Service

5. Recent History • The staff who had designed and implemented ProbeMaker were no longer at Cornell • The ProbeMaker and Visual Basic clients were getting a bit long in the tooth • The decision was made to re-implement the client side in Java, using the same code base for both Mac and Windows, and hopefully Unix as well • Other cross-platform projects had gone well, so we were not expecting trouble Using Java for a High Performance Infrastructure Service

6. Common Client Code Base CUSSP Natural Dispatcher KFRONT CUSOCKET CUSSP TCP/IP Java TCP/IP C Using Java for a High Performance Infrastructure Service

7. MELTDOWN!!! • Most students saw “Connection Failed” messages • Over and over and over again….. • Eventually, the students got enrolled, but it was a very bad experience all around Using Java for a High Performance Infrastructure Service

8. The Post Mortem (technical) • The old clients had code to just quietly re-try an indefinite number of times if the connection failed • The queuing code in Kfront was not working at all • The depth of the TCP/IP connection queue on the mainframe could be set to any value, as long as that value was 5 • The new client would re-try a failed connection several times, but would then give up and reflect an error message back to the user Using Java for a High Performance Infrastructure Service

9. Post Mortem (social) • It had been announced to the students the class sizes were going to be more strictly enforced, so if you wanted that small but popular class you should be ready to pre-enroll as soon as the system was available • The start of pre-enrollment was midnight on a Friday night, when ALL of the students are awake, and NONE of the support staff are at work • 3000 students were poised with their fingers over the enter key, watching the second hand approach midnight • Due to the previously mentioned technical problems, all but 30 were going to get bounced with an error message Using Java for a High Performance Infrastructure Service

10. Post Mortem (solutions) • Improve the efficiency of the main-frame dispatcher support • Done, but was clearly not enough to solve the problem • Get IBM to fix the TCP/IP queuing so that queue lengths other than 5 were supported • Dead end • Fix queuing in the mainframe Kfront implementation • Code was poorly documented spaghetti code • Original developer had left Cornell • No one with mainframe C experience (or at least no one who would admit it…) Using Java for a High Performance Infrastructure Service

11. Post Mortem (solutions) • Move Kfront to Unix and Java • The Unix TCP/IP stack is significantly more robust that the mainframe stack, and properly supported connection queuing • Our bottleneck (if we got the queuing fixed) was going to be mainframe cycles for the dispatchers, so moving load off the mainframe would help • The available developer was much more competent in Java than in C • Java has relatively easy to use multi-threading, and a straightforward TCP/IP API Using Java for a High Performance Infrastructure Service

12. The Re-Write • Design goals • Performance, performance, performance • Good behavior under adverse conditions • Manageable • Platform independent Using Java for a High Performance Infrastructure Service

13. Performance • Use threads effectively • Separate different tasks into their own threads • Allow each type of task to have multiple threads • Minimize creation of new objects • Object pooling • StringBuffers • Minimize synchronized code • The only synchronized operations are queue and dequeue, both of which are very fast Using Java for a High Performance Infrastructure Service

14. Initial Architecture Cleanup Q Stats Q Listener Q Dispatch Q Kerb Using Java for a High Performance Infrastructure Service

15. Queue management • Queues are linked lists • They do not use any library objects or methods • Queue and dequeue operations are very fast • They are the ONLY synchronized code • Support for both FIFO and LIFO queueing • Administrative operations are queued LIFO Using Java for a High Performance Infrastructure Service

16. Listener • Listener thread listens for new connections • New socket is put into a command object and queued for further processing • Run as the highest priority thread • Original implementation had one listener task, current has a second dedicated to administrative use Using Java for a High Performance Infrastructure Service

17. Kerb • Kerb task parses the incoming command, decrypts the Kerberos ticket and queues the command for further processing • Since ticket decryption is resource intensive, mutiple copies of this thread are run • Command parsing is done by a very fast finite state automaton • Thread.yield() called between command parse and ticket decryption • Administrative commands are identified and queued for processing locally Using Java for a High Performance Infrastructure Service

18. Dispatch • Each command processor on the mainframe has its own Dispatch object running in a thread • The TCP/IP connection between the Dispatch object and its mainframe command processor is kept open, reducing the stress on the mainframe TCP/IP stack • Error recovery is a BIG deal • “Broken Pipes” are a major pain Using Java for a High Performance Infrastructure Service

19. Cleanup • Cleanup task resets the command object fields to their default status and queues the command for re-use • Original implementation had the cleanup task performing administrative functions – these have subsequently been moved to their own task Using Java for a High Performance Infrastructure Service

20. Stats • Stats task runs in its own thread, accepting statistics and periodically logging statistics records • For performance reasons, the statistics buckets are updated by other threads without synchronization • Although this has the potential to corrupt the statistics, it was felt that the performance hit of synchronizing all statistics updates (there are a LOT of them) was not warranted Using Java for a High Performance Infrastructure Service

21. First production… • Scene: • Senior Management Meeting • VP for Information Technologies (very worried): • Oh, dear. Isn’t Course Enrollment coming up soon? • Director of Integration & Delivery (very smug): • Yes, it started last Monday…. • E-mail from student: • “You guys are no fun, you made Course Enroll boring….” Using Java for a High Performance Infrastructure Service

22. Since then…. • Kfront has been stable for months at a time • Major outstanding problem has been broken TCP/IP connections between Kfront and the mainframe • Because we don’t know whether the command was processed or not on the mainframe, we don’t know whether or not to retry • In the most recent Course Enroll period we experienced peak loads of 8,000 commands processed per minute • Response time went as high as 15 seconds, but had dropped back down to sub-second within 10 minutes of the start of the enrollment period • Bottleneck is the mainframe – we can easily push it to 100% CPU for extended periods Using Java for a High Performance Infrastructure Service

23. Epilog • The Kfront re-write was a huge success • Java is very definitely appropriate for implementation of high-performance infrastructure services Using Java for a High Performance Infrastructure Service

24. Where to now? • Investigate conversion to the “New I/O” architecture in Java • Add support for “Native” Java routines • Experiments have shown that a Java routine accessing ADABAS via TCP/IP may be as much as 10 times as fast as the current NATURAL routines • Re-architect to facilitate recovery from “broken pipes” • Sigh – this one will require getting back into the IBM assembler code… Using Java for a High Performance Infrastructure Service

25. Questions?? Using Java for a High Performance Infrastructure Service