Java Projects

1. Java Projects Tools and techniques Mike Godfrey Information Technology Services / The University of Texas at Austin

2. Outline Look at three java projects • Different architectures adopted • Problems encountered and lessons learnt • Tools and other useful techniques

3. Who are we? • Remains of centralized data processing • Part of the unit providing IT services across campus • Traditionally mainframe based applications with web front-ends scripted in a simple in-house language • Changing mission as our administrative application development role is done by departments • Project list that requires a much more powerful language pushing adoption of Java • Changes are forcing developer to acquire new skills and ways of working – java camp.

4. The Systems • Clips – a web-based file system for distributing course materials • Jwebagent – a java runtime for our in-house scripting language • Tech lounge – a web-based developer documentation store

5. Clips – file system

6. Clips Requirements • A simple to use, intuitive interface • Handle all file types • Accessible from any location • Date controlled display of materials • Easy transfer of materials from class to class • Work well with existing mainframe class management system • Work within the UTDirect portal framework

7. Basic Design Choices • File system metaphor for user interface • Browser access for both students and faculty • Use native XML database to model the file system in a recursive schema • Store binary files in same XML database • JSP with scriptlets for display • clips interface

8. Use of XML for Meta Data • File system metaphor seems ideally suited to the tree structure of XML • Meta data structure based on two nodes types; directory-nodes and binary-nodes • One XML document contains all meta information needed to build file structure for a given class • Easy to re-build hierarchy of resources from XML tree • Database uses schema to specify stored document structure

9. Meta-Data Details • Directory nodes store information about a directory: name, creation time, description, etc • Binary nodes store information about a binary file: name, foreign key, size, content type, etc • Directory nodes can contain either binary nodes or other directory nodes (hierarchy) • clips meta data

10. mainframe Broker Natural Existing class management system Adabas Clips Architecture XML database (binaries and Meta-data) Helper Classes Tamino api JSP presentation layer User Broker api Authorization and class info

11. New Issues for us • Handling and auditing file uploads from web pages (multipart encoded forms) • Building and manipulating XML records • How to best make large scale use of JSPs in a complex Java system • How to simplify communication with mainframe

12. Useful Tools • Log4j – used for debugging and recording production exceptions (system written before Java 1.4) • JDOM – very user friendly api for manipulating the XML • cos.jar – used to parse the multipart encoded forms • Entire-X broker ACI for Java • Java stub generator for mainframe modules • Tamino - native XML database

13. Lessons Learnt • Be careful of scriplets in JSP; they expand to fill the available space! • Better to use a controller servlet to organize navigation through JSPs • XP and pair-programming works • A Java developer’s best friend is Goggle

14. View jsp Controller servlet Maintain jsp Update jsp Copy jsp Copy Task Copy Task Auth Task Auth Task Form Parser Form Parser Resource List Resource List Update Task Update Task mainframe mainframe XML store XML store Alternative JSP architecture View jsp Maintain jsp Update jsp Copy jsp

15. Jwebagent Jwebagent HTML file .WBX file

16. What is Webagent? • A scripting language with a syntax like Natural (mainframe language) • Scripts transformed into C++ objects forming a virtual machine which is serialized to a .WBX file • Runtime reads .WBX file, rebuilds the C++ virtual machine and executes it. • Pro: easy and productive for creating web front ends to mainframe applications. • Con: limited functionality

17. Why Jwebagent? • Same as Webagent but the rebuilding of the runtime is done in Java …plus • Enable java classes to run existing Webagent scripts • Enable Webagent scripts to use java classes • Improve execution speed – no process forking and script buffering

18. Main Runtime Elements • Servlet – handles all execution requests for a WBX file • Executor – coordinates loading file, running the VM, and handling errors for an individual file request • File loader – re-builds the serialized script objects • Context objects – handle data specific to that particular request • VmOpCodes – functional elements of the VM • Data objects – variables, literals

19. file request Servlet Executor vmOp vmOp broker vmOp broker vmOp broker vmOp vmOp WBX file Serial code object env http req http resp Request context Runtime Script context Builds then executes Data buffer

20. Useful Tools • Junit – unit testing framework (junit.org) • JProbe – application analyzer (Quest Software) • JMeter – load testing suite (Apache.org) • HttpExplorer – gui http request and response tool (in-house) • Jedit – text editor (Jedit.org) • IntelliJ IDEA – Java IDE (Jet Brains) • CVS (Tortoise, WinCVS, Putty for SSH) • Xvi32 – hex editor

21. Junit – unit testing • Provides easy way to set and teardown test situations • Automated running of many tests • Organize tests in suites and suites into meta-suites • Text or gui UI for test reporting • Refactor with confidence! (very valuable) • Trick is to know what tests to write • Good with Jwebagent as it is not database driven

22. JProbe – code analyser • Threadalizer – finds deadlocks and conflicts (did not show any problems for Jwebagent) • Memory Debugger – shows memory use by classes and can find hanging objects (useful – strings to byte arrays) • Profiler – tracks objects and the call pattern, records method time and number of calls (most useful tool –identify prime objects to optimize)

23. JProbe Code Optimizations • Strings to byte arrays – saved memory on string generation • io to nio gave 25% increase in through-put • In-lining of frequently used methods to reduce calling overhead • Combining of related classes when frequently constructed • Un-doing OOP improved performance!

24. JMeter – load testing • Build and store test plans • Control number of threads, ramp-up times and thread delay to simulate different testing scenarios • Test http apps, SOAP and web services, LDAP, JDBC, etc • Display test results in graphs, tables, files, etc • Build tests with logical flows to suit application • Apply pre and post requests during tests • Very valuable for performance tuning (with JProbe)

25. JVM Tuning • Changed default of client mode to server mode (20% improvement) • Other settings have given some improvements • Use incremental GC rather than ‘stop the world’ style • Uses the flags that control survivor ratio for the GC generations – we have short lived objects • Increased default memory allocation

26. Did we make it? • First tests showed consistently slower times than the C++ version ( up to 33* slower) • JProbe analysis and JVM tuning reduced this to about 2* slower on average • Through-put is slightly lower • JMeter shows a mean that is much faster than C++, but and average that is much higher • Problem appears to be in the multi-treading of Broker • Tried JNI against C++ broker and see same results

27. Lessons learned • Always build a test suite before refactoring • Never optimize until you know what to optimize • Do not assume GC defaults are optimal • Java needs lots of memory! • Allow for distractions in your project estimates (estimated time 9 months, actual time start to finish 2.5 years, actual time spent on project 10 months)

28. Tech Lounge

29. Project Goals • Nice, easy to use documentation store • Replace mainframe based system • Provide better editing and display options • Allow limited access to subsets of articles • Control the life-time of articles to ensure relevancy • Provide good search capabilities and cross-referencing options • Store multiple document formats

30. Design Choices • Store articles in native XML database • Article content stored against a subset of XHTML strict DTD – easy to display and construct • Each article has a meta-data XML file for owners, dates, related articles, number of reads, etc • Article display via XSLT for HTML and pdf • Article maintenance via jsps and custom tag libraries • tech lounge

31. Article Maintenance Architecture New Article jsp Controller servlet Maintain Article jsp Auth Task Form Parser Article Bean mainframe XML store

32. Useful Tools • Many things from Apache • DHTML • Graphic designer Created Tools/utilities • Tamino Wrappers – abstraction layer • UID (QUID-like foreign key generator)

33. Problems/Mistakes • Not using CVS well during first half of project • Not writing unit tests • Not using logging for debug messages

34. Evolution • Display of XML – moving towards all XSLT • JSP usage – moving away from interlinked pages towards controller server style • JSP structure – scriplets bad, custom tags good(?) • Building up institutional tools/utilities and standards that are increasing productivity • Individual experts towards wide spread Java use through in-house training courses

35. Review of our Java Experience Pros: • More flexible and capable • Greater abstraction possible, increased elegance • Many very powerful tools/utilities freely available Cons: • Requires higher competency of developers • Less productive for our typical applications • Needs larger machines

36. Other projects • PdaParser – code generation to ease communication with mainframe modules • Fatcookie – authentication • Virtual File System extension to JEdit to work with our unix server script editing proceedure Current • Jar management database (entirely XML/XSLT) • Web service wrapper for mainframe modules • Dynamic pdf generation from mainframe data

37. Questions?

38. Useful Project Management Tip Remember the Bullfighter!