GLStream Services Implementation

1. GL Stream ServicesImplementationNovember 26th, 2007Core Products - Trading Servers GL TRADE Internal Use Only

2. GL TRADE Internal Use Only Synchronous Communication Flow Control Local Remote caller callee caller callee Asynchronous Communication Flow Control Single-Threaded Program Single-Threaded Callee Multi-Threaded caller callee caller callee caller callee Synchronous vs. Asynchronous Communication • Synchronous communication • The caller sends a message to the callee • The callee may return data to the caller • If on the same process: the thread of the caller is lent to the callee • If on different process: the caller is halted until the callee returns • Equivalent to a telephone call • Asynchronous communication • The caller sends a message to the callee • The callee does not return any data to the caller • The caller proceeds once the message is sent • The callee runs in its own threading model • Input messages can be queued • Equivalent to an email • Two communication tools must be provided by GLStreamServices • Hooks and Stubs for synchronous communications • Queues for asynchronous communications

3. GL TRADE Internal Use Only Original Code Code once the Hook has been added GLHandlerP10::TreatCrossOrder(GLDataCmd *cmd) { switch (cmd->GetCommand()) { case cmdOrder: return TreatCrossCreate(); case cmdCancel: return TreatCrossCancel(); case cmdUpdate: return TreatCrossUpdate(); ... } } GLHandlerP10::TreatCrossOrder(GLDataCmd *cmd) { if (HOOK(OnTreatCrossOrder,cmd)==1) return; switch (cmd->GetCommand()) { case cmdOrder: return TreatCrossCreate(); case cmdCancel: return TreatCrossCancel(); case cmdUpdate: return TreatCrossUpdate(); ... } } HooksSynchronous Communications • The Hook is the output gate of a program • A hook is a place in the code where subroutines can be called • A program may have several hooks • Each hook belongs to only one program • The owner of the hook does not have any knowledge of the subroutines that are called (if any) • The owner of the hook specifies the contract (ie: input and output data definition) • Hooks are implemented as one simple function call • Uniquely identified by a logical name • Object instantiated only once at the first pass • By construction, hooks always return an integer whose value can be set by subroutines (default returned value is 0) • Minimal impact to the original code and performance • In the example above: • The hook name is “OnTreatCrossOrder” • Input data contains the received command (part of the contract) • If the returned value is 1, the original code will return immediately (also part of the contract)

4. GL TRADE Internal Use Only StubsSynchronous Communications • A Stub is a subroutine that can be called by a hook on behalf of a program thread • If the stub is available in the hook process space, it runs in the caller’s thread • Communications between hooks and stubs are handled by the GLStreamServices layer • Stubs run in the same address space as the caller (ie: the same process) • A stub should follow the contract specified by the hook • The stub knows beforehand what input data it should expect and what output data it can set • Returned and output data set by stubs may alter program paths of the caller owning the hook • If several hooks share the same contract, the same stub can be shared across these hooks • Several stubs can be associated to the same hook • Each stub has a default priority that can be overriden by a configuration file • Stub ordering is deterministic (ie: ordering based on priority) • Each stub has the ability at runtime to tell the hook whether the next stub should be called or not • Stubs are subroutines implemented as shared librairies (eg: .dll or .so) • In most cases, a stub knows the hook(s) it should be associated with • However, hook / stub associations can be set by a configuration file • To this end, each stub is uniquely identified by a logical name

5. GL TRADE Internal Use Only QueuesAsynchronous Communications • The Queue is the input gate of a program • A queue contains a list of messages waiting to be processed (FIFO) • Logical entity created upon program startup • A program may have several queues • Each queue belongs to only one process (ie: no physical distribution – reachable in only one memory space) • Any program can post messages into the queue via the GLStreamServices layer • Each queue is identified by a logical name • The queue owner is notified upon message arrival (ie: event driven) • The queue owner chooses the threading model • The owner associates a queue to a thread pool • The owner can set the size of thread pools (ie: from 1 to n threads) • Several queues can share the same pool • For example, to ensure single-threaded execution, a program associates all queues to a pool of 1 thread

6. GL TRADE Internal Use Only QueuesUsage Models • Service Queues • The name of the queue identifies the service • The queue owner specifies the contract • The contract specifies the data (type and domain values) expected in messages • All messages posted into the queue by other programs should follow the contract • Notification Queues • The name of the queue identifies a notification • The sender of notifications specifies the contract • All queues processing notifications should follow the contract • Similar to message broadcasting • Both models use the same communication tool but differently • Same class (STSQueue) • Same implementation • Same interface

7. GL TRADE Internal Use Only Message Received Message Received P10 Pre-process Message Pre-process Message Hook Call Hook «OnProcessMessage» Call Hook «OnProcessMessage» 1 1 Stop Stop Return Code? Return Code? 0 0 Message Checked Message Checked Process Message Process Message Y Y N N Queue Is Command Accepted? Is Command Accepted? Send Command to AP Send Command to AP Send Reject to Client Send Reject to Client Stop Stop Stop Stop Test Case - P10 alone • Out of the box, the P10 provides one hook and one input queue • The hook “OnProcessMessage” allows other programs to process incoming messages • The queue “MessageChecked” allows other programs to post validated or rejected commands • By default, the hook does not call any stub and any incoming message is forwarded to the AP • The P10 uses one thread to process the message Flow Control P10 Program Path

8. GL TRADE Internal Use Only Message Received Message Received P10 P10 Enter Enter Stub Pre-process Message Pre-process Message N N Is Uti Selectorized? Is Uti Selectorized? Y Y Call Hook «OnProcessMessage» Call Hook «OnProcessMessage» Send Cmd to Selector Send Cmd to Selector 1 1 Stop Stop Return Code = 1 Return Code = 1 Return Code? Return Code? 0 0 Message Checked Message Checked Process Message Process Message Return Return Y Y N N Is Command Accepted? Is Command Accepted? Send Command to AP Send Command to AP Send Reject to Client Send Reject to Client Stop Stop Stop Stop Test Case – P10 and Selector • The Selector delivers a stub which is going to plug itself into the P10 hook • The job of the Selector stub is to find out whether a uti is selectorized or not • If the uti is selectorized, the command is then forwarded to the Selector • If the uti is not selectorized, the program path remains the same • However, the Hook now calls the selector stub • The thread of the P10 is then lent for a short period of time to the selector stub Flow Control P10 Program Path Selector Stub

9. GL TRADE Internal Use Only Selector Message sent to the Selector Agent 1 Stop Return Code? Process Message Y N Is Command Accepted? Send Command to AP Send Reject to Client Stop Stop Test Case – P10 and Selector • If the uti happens to be selectorized, the command is sent by the stub to the selector • Asynchronous communication is used between the stub and the selector • In such a case, the selector stub sets the return code to 1 • Therefore, the P10 stops processing the command • Later on, once the selector has processed the command, it posts it into the P10 input queue • Once again, asynchronous communication is used • The P10 is then notified by the GLStreamServices layer and processes the command Flow Control P10 Program Path Selector Stub Message Received P10 Enter Stub Pre-process Message N Is Uti Selectorized? Y Call Hook «OnProcessMessage» Send Cmd to Selector Return Code = 1 0 Message Checked Return

10. GL TRADE Internal Use Only CLIENT HANDLER CLIENT HANDLER P10 msg RRA1 SELECTOR P10 RRA1 RRA2 SELECTOR PERSHINGRR RRA2 PERSHINGRR RRA3 RRA3 LINE HANDLER P10 P10 msg LINE HANDLER CLIENT HANDLER Flow ControlTest Case: Selector and Pershing • The client handler has a dedicated thread handling client connections • The client handler receives a client message • The client handler forwards the message to the P10 via a queue (asynchronous) • The P10 pre-processes the message and reaches its first hook • The hook calls the selector stub which finds out that the uti is selectorized • The stub then forwards the message to the selector (asynchronous) • The selector stub then returns a value telling the P10 not to process the message • The thread of the P10 then stops • Once evaluated, the message is posted to the P10 by the selector (asynchronous) • The P10 is then notified, processes the message and reaches its second hook • The second hook calls the pershing stub that finds out that Pershing routing rules apply • The stub then forwards the message to the Pershing RR program (asynchronous) • The stub then returns a value telling the P10 not to process the message • The thread of the P10 then stops • The Pershing Routing Rule program then processes the message • Once the destination has been set, a message is posted to the P10 (asynchronous) • The P10 is notified of the Pershing message arrival and processes it • The P10 then reaches its third hook which calls GL destination-check stub • The stub makes sure that the destination is valid • The stub then returns a value telling the P10 to keep on processing the message • The P10 then posts the client command to the Line Handler program (asynchronous) • The Line Handler program then forwards the command to the appropriate AP

11. GL TRADE Internal Use Only CLIENT HANDLER CLIENT HANDLER CLIENT HANDLER P10 P10 msg msg RRA1 RRA1 SELECTOR P10 RRA1 RRA2 RRA2 SELECTOR PERSHINGRR PERSHINGRR RRA2 PERSHINGRR RRA3 RRA3 RRA3 LINE HANDLER LINE HANDLER P10 P10 msg msg LINE HANDLER CLIENT HANDLER Flow ControlTest Case: Uti not selectorized • If the uti is not selectorized, the selector stub does not post any message to the selector • In such a case, it returns to the P10 as if nothing happened • The P10 resumes its original thread and keeps on processing the original message

12. GL TRADE Internal Use Only P10 New BackEnd Server modules msg RRA1 SELECTOR Selector modules Pershing modules GLStreamServices Communication Tools RRA2 PERSHINGRR RRA3 msg Logical Modules CLIENT HANDLER LINE HANDLER

13. GL TRADE Internal Use Only Physical Modules SelectorAgent Plugin Selector Agent Selector Stub Plugin GLStreamServices GLStream Plug-In Server Selector Stubs GL libs libc GLStreamServices GL libs GLStreamServices libc GL libs libc GLTSAgent Plugin GLStream Server GLTS Stub Plugin GLTS Agent GLStreamServices GLTS Stubs P12 Agent P10 Agent GL libs GLStreamServices GLStreamServices libc GL libs GL libs libc libc P11 Stub Plugin P11 Stubs P11Agent Plugin GLStream Plug-In Server GLStreamServices GL libs P11 Agent libc GLStreamServices GLStreamServices GL libs GL libs libc libc Business libraries Configuration File Process Space Technical libraries Shared libraries (.dll or .so) GLStream Server Inter-Process Communications (TCP/IP) Log Files GLStream Server Intra-Process Communications (Internal API) Executable binary files