1 / 58

Atropos

Atropos. Comp 415 Project In partnership with JPMorganChase. Introductory Remarks. COMP 415 Class: Brad Chelsea Dave Felipe Hubert Matt Sohum Dr. Wong Jacy Grannis. A Word From Our Sponsors.

avital
Download Presentation

Atropos

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Atropos Comp 415 Project In partnership with JPMorganChase

  2. Introductory Remarks • COMP 415 Class: • Brad • Chelsea • Dave • Felipe • Hubert • Matt • Sohum • Dr. Wong • Jacy Grannis

  3. A Word From Our Sponsors... JacyGrannis has been our customer contact within JPMorganChase. He is a Rice Computer Science graduate and works in JPMorgan’s Houston office.

  4. COMP 415—Summary • Software engineering • Real customer, real experience • Team environment • Communication • Organization

  5. COMP 415—The benefits • Team programming experience is sought by employers and not often found in undergraduates • Saves on-the-job learning time • Industry exposure

  6. COMP 415—The sequence • COMP 410 • COMP 415 • Teaching assistant for both courses • Need more courses

  7. Atropos …semester in review

  8. Atropos—Success • Correlation • Network View • Single Edge View • Remote communication

  9. Atropos—Mistakes • Teams change • Individuals change • Team structure • Small team • Proxy pattern • Integration

  10. System Requirements …an Atropos primer

  11. System Requirements—Correlation • Track messages as they travel between servers • Different systems have different message formats • Messages have no fixed unique ID • Servers may split and merge messages

  12. System Requirements—Scalability • System must handle thousands of messages per second • Components may run on separate machines • Concurrency • Asynchronous communications

  13. System Requirements—Recoverability • Failure of one component should not bring the system down • Failure detection • Component restart • Restore component state • Replay dropped messages

  14. System Design …the view from the top

  15. Distributed system Service-oriented architecture (SOA) Component registration Atropos System Design—Unregistered

  16. Distributed system Service-oriented architecture (SOA) Component registration Atropos System Design—Registered

  17. Design—Abstraction and the Proxy Pattern • Allows flexible system configuration • Beneficial for testing purposes and agile development • Unbiased by transfer protocol

  18. Design—Dynamic Correlation • Concepts of finding matches from message organization are logically distinct • Decouple algorithm from data • Correlation engines organize messages and use strategies to fine matches • Changing strategies at runtime enables intelligent load balancing and independence from JPM system configuration

  19. Atropos Demo …which will totally work

  20. Start Controller and Router • Initialize the controller • Service that manages the system • Router registers with controller

  21. Start Correlation Engine, Data Client • Correlation Engines and Data Clients run in their own JVMs • On startup, they register with the controller • Controller configures them to connect them into the system

  22. Starting View System • View Provider • View Client

  23. Connect and Start Client • View Client connects to system • Requests data 

  24. Changing Sources and Edges • Utilize IAdmin API to add sources • Dynamically add and remove edges • Check system status after updates

  25. Single Edge Latency View • Access • Dialog • Clicking edge in Dashboard View • Display • Time-chunk average of latency • Scalable to different time periods • Static or updatable

  26. Custom Latency View • See correlation during a specific time window

  27. Uncorrelated Messages • Messages change, machine connections change, parts of the topography may disappear completely • Messages will stop correlating • View uncorrelated messages over a period of time • Updates live (currently every 30 secs)

  28. Disconnect View Client • Disconnect a view client currently registered • View client is unregistered • Future updates are no longer sent

  29. Remote Communication …bridging gaps

  30. Remote Communication • No guarantee that components are on the same machine • Need a way to communicate between multiple components • Solution must be • Scalable • Flexible • Pluggable

  31. Mule—Benefits • Provides a scalable package solution • Open source • Provides a flexible framework that can adapt to any architecture • Can tie into many messaging protocols

  32. Mule—Hardships • Little support and documentation • Remote method invocation can cause problems • Needs to be scaled to work effectively

  33. Functional Communication • Proxy pattern connects components • Controller as an object on the Mule server • Mule support behind remote calls abstracted out

  34. Data Client …the beginning of it all

  35. Data Client—Goals • Vehicle for getting messages into the system • Robust • Abstract • Recoverable • Extensible

  36. Data Client—Message Sources • Entry point for messages • Accepts messages with a simple API • accept(message)... too easy? • Next step: router

  37. Data Client—Abstract • Messages can come from anywhere • Log parsers • Applications producing the messages • Manual API calls • Abstracting out information about the source is crucial for interoperability

  38. Data Client—Recoverability • Router dies... • What happens to messages that have been sent through the Data Client but are dropped? • Take a lesson from Tivo! • Store information about messages that have been seen and replay! • Controller detects failure in the system • Can ynchronize all the modules in the system by replaying the Data Client

  39. Data Client—Extensibility • Multiple sources can send information to one data client • Require that incoming messages are be tagged with their source ID • Data Clients can send to other Data Clients • Available for common special processing that can be added at run time • Added and removed via registration at run time with the Controller

  40. Router …the grunt work

  41. Router—Role • Data Clients are only aware of Router • All messages simply forwarded • Router responsible for mapping Data Clients to Correlation Engines • Data Client to Correlation Engine mapping not 1:1 • One-to-many relationship

  42. Router—Extensibility • Routers can make use of the composite pattern • Updated dynamically at runtime by the Controller • Whenever Data Clients or Correlation Engines register (or unregister), the router must reflect those changes

  43. Correlation Engine …the workhorse

  44. Correlation Engine—Role • Organizes messages received • Applies correct strategies between various sources, as determined by the system’s IAdmin • Writes correlation information to a memory store • Keeps a store of what messages have been received for replay purposes

  45. Correlation Engine—Design Concerns • Configurability • Should handle addition of sources and new rules at runtime • Concurrency • Thread safety • Use of concurrency to take advantage of multi-cores

  46. Correlation Engine—Implementation Details • Makes messages appear to have been received in order; facilitates strategy implementation • Configurable correlation strategies • Handle rules for correlation • Insulated from issues related to system configuration, message order, etc.

  47. User Interface …the pretty side

  48. View Client—Technologies • Eclipse RCP Application • Java Universal Network/Graph (JUNG) framework • JFreeChart framework

  49. Eclipse RCP Application—Positives • Portability • Integration with current JPMorgan systems • Faster development on Eclipse IDE • Powerful framework simplifies component connections

  50. Eclipse RCP Application—Negatives • SWT less popular than AWT • Fewer resources for help • Steep learning curve

More Related