Atropos

1. Atropos Comp 415 Project In partnership with JPMorganChase

2. Introductory Remarks • COMP 415 Class: • Brad • Chelsea • Dave • Felipe • Hubert • Matt • Sohum • Dr. Wong • Jacy Grannis

3. A Word From Our Sponsors... JacyGrannis has been our customer contact within JPMorganChase. He is a Rice Computer Science graduate and works in JPMorgan’s Houston office.

4. COMP 415—Summary • Software engineering • Real customer, real experience • Team environment • Communication • Organization

5. COMP 415—The benefits • Team programming experience is sought by employers and not often found in undergraduates • Saves on-the-job learning time • Industry exposure

6. COMP 415—The sequence • COMP 410 • COMP 415 • Teaching assistant for both courses • Need more courses

7. Atropos …semester in review

8. Atropos—Success • Correlation • Network View • Single Edge View • Remote communication

9. Atropos—Mistakes • Teams change • Individuals change • Team structure • Small team • Proxy pattern • Integration

10. System Requirements …an Atropos primer

11. System Requirements—Correlation • Track messages as they travel between servers • Different systems have different message formats • Messages have no fixed unique ID • Servers may split and merge messages

12. System Requirements—Scalability • System must handle thousands of messages per second • Components may run on separate machines • Concurrency • Asynchronous communications

13. System Requirements—Recoverability • Failure of one component should not bring the system down • Failure detection • Component restart • Restore component state • Replay dropped messages

14. System Design …the view from the top

15. Distributed system Service-oriented architecture (SOA) Component registration Atropos System Design—Unregistered

16. Distributed system Service-oriented architecture (SOA) Component registration Atropos System Design—Registered

17. Design—Abstraction and the Proxy Pattern • Allows flexible system configuration • Beneficial for testing purposes and agile development • Unbiased by transfer protocol

18. Design—Dynamic Correlation • Concepts of finding matches from message organization are logically distinct • Decouple algorithm from data • Correlation engines organize messages and use strategies to fine matches • Changing strategies at runtime enables intelligent load balancing and independence from JPM system configuration

19. Atropos Demo …which will totally work

20. Start Controller and Router • Initialize the controller • Service that manages the system • Router registers with controller

21. Start Correlation Engine, Data Client • Correlation Engines and Data Clients run in their own JVMs • On startup, they register with the controller • Controller configures them to connect them into the system

22. Starting View System • View Provider • View Client

23. Connect and Start Client • View Client connects to system • Requests data 

24. Changing Sources and Edges • Utilize IAdmin API to add sources • Dynamically add and remove edges • Check system status after updates

25. Single Edge Latency View • Access • Dialog • Clicking edge in Dashboard View • Display • Time-chunk average of latency • Scalable to different time periods • Static or updatable

26. Custom Latency View • See correlation during a specific time window

27. Uncorrelated Messages • Messages change, machine connections change, parts of the topography may disappear completely • Messages will stop correlating • View uncorrelated messages over a period of time • Updates live (currently every 30 secs)

28. Disconnect View Client • Disconnect a view client currently registered • View client is unregistered • Future updates are no longer sent

29. Remote Communication …bridging gaps

30. Remote Communication • No guarantee that components are on the same machine • Need a way to communicate between multiple components • Solution must be • Scalable • Flexible • Pluggable

31. Mule—Benefits • Provides a scalable package solution • Open source • Provides a flexible framework that can adapt to any architecture • Can tie into many messaging protocols

32. Mule—Hardships • Little support and documentation • Remote method invocation can cause problems • Needs to be scaled to work effectively

33. Functional Communication • Proxy pattern connects components • Controller as an object on the Mule server • Mule support behind remote calls abstracted out

34. Data Client …the beginning of it all

35. Data Client—Goals • Vehicle for getting messages into the system • Robust • Abstract • Recoverable • Extensible

36. Data Client—Message Sources • Entry point for messages • Accepts messages with a simple API • accept(message)... too easy? • Next step: router

37. Data Client—Abstract • Messages can come from anywhere • Log parsers • Applications producing the messages • Manual API calls • Abstracting out information about the source is crucial for interoperability

38. Data Client—Recoverability • Router dies... • What happens to messages that have been sent through the Data Client but are dropped? • Take a lesson from Tivo! • Store information about messages that have been seen and replay! • Controller detects failure in the system • Can ynchronize all the modules in the system by replaying the Data Client

39. Data Client—Extensibility • Multiple sources can send information to one data client • Require that incoming messages are be tagged with their source ID • Data Clients can send to other Data Clients • Available for common special processing that can be added at run time • Added and removed via registration at run time with the Controller

40. Router …the grunt work

41. Router—Role • Data Clients are only aware of Router • All messages simply forwarded • Router responsible for mapping Data Clients to Correlation Engines • Data Client to Correlation Engine mapping not 1:1 • One-to-many relationship

42. Router—Extensibility • Routers can make use of the composite pattern • Updated dynamically at runtime by the Controller • Whenever Data Clients or Correlation Engines register (or unregister), the router must reflect those changes

43. Correlation Engine …the workhorse

44. Correlation Engine—Role • Organizes messages received • Applies correct strategies between various sources, as determined by the system’s IAdmin • Writes correlation information to a memory store • Keeps a store of what messages have been received for replay purposes

45. Correlation Engine—Design Concerns • Configurability • Should handle addition of sources and new rules at runtime • Concurrency • Thread safety • Use of concurrency to take advantage of multi-cores

46. Correlation Engine—Implementation Details • Makes messages appear to have been received in order; facilitates strategy implementation • Configurable correlation strategies • Handle rules for correlation • Insulated from issues related to system configuration, message order, etc.

47. User Interface …the pretty side

48. View Client—Technologies • Eclipse RCP Application • Java Universal Network/Graph (JUNG) framework • JFreeChart framework

49. Eclipse RCP Application—Positives • Portability • Integration with current JPMorgan systems • Faster development on Eclipse IDE • Powerful framework simplifies component connections

50. Eclipse RCP Application—Negatives • SWT less popular than AWT • Fewer resources for help • Steep learning curve