1 / 12

Main Sphinx Design Concepts

Main Sphinx Design Concepts. There are two primary design components which comprise Sphinx The Database Warehouse The Control Process The Database Warehouse is the information depot for all of Sphinx All information is stored/retrieved via the database (or possibly multiple databases)

robertsray
Download Presentation

Main Sphinx Design Concepts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Main Sphinx Design Concepts • There are two primary design components which comprise Sphinx • The Database Warehouse • The Control Process • The Database Warehouse is the information depot for all of Sphinx • All information is stored/retrieved via the database (or possibly multiple databases) • There is no direct communication between different internal Sphinx modules • The API for all internal modules is SQL along (with the database schema) • The Control Process is the main point of activity • It is a loop which calls different Sphinx modules • All modules in the control process operate on stateful entities in the database, changing the state of an entity from one to another state • Sphinx operates as a Finite State Machine! • Uses data mining principles in the design: • The API for all modules is SQL along the database schema • Modules mine the database for information and then take some action.

  2. Version 1: Sphinx Server/Client Sphinx Server VDT Client Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Single-threaded Control Process Job Tracker Prediction Engine Job Submitter Site Tables Condor-G/DAGMan Job Execution Planner VDT Server Site Resource Job Table DAG Table Replica Table Information Gatherer Monitoring Service Message Tbl.

  3. More Details • The next slides show the message passing between different Sphinx components • The components are split (packaged) in the following way: • Sphinx Client • Incomming Messages • Outgoing Messages • Sphinx Server • Incomming Messages • Control Process • Outgoing Messages • See http://ufgrid.phys.ufl.edu:8080 for documentation • The next version of Sphinx (currently in progress) will split/package the components into a more logically distinct, modular scheme (yet to be decided)

  4. SendRequest MsgHandler DOMparser Version 1: Sphinx Client (outgoing message) Message ID Msg Type Msg Sender Message XML RPC XML message Destination URL Clinet ClinetOutModule ACK ACK ACK Files MsgID Msg Type Msg Sender Message Message Type Path to file(DAX) Path to User ID file XML message Msg strings Sender Info XMLDocUtils MsgID Msg Type Msg Sender Message XML message

  5. DOMparser Version 1: Sphinx Client (incoming message) JobWrapper job ID dag ID Message string SubmissionGenerator Msg string Work Dir ACK Super dag file ClientIncModule createSubmission execute XML message Msg type Message string condor_submit_dag XML message MsgHandler Msg ID Msg Type Msg Sender Message XML message

  6. Version 1: Sphinx Server (incoming message) DB Connector Msg Type Msg Sender Info Message XML message MessageWrapper InMessageWrapper IncModule ACK (Msg ID) Msg Id Msg Type Msg Sender Info Message checkIn XML message Dag MsgHandler message Msg Id Msg Type Msg Sender Info Message checkIn XML message checkOut remove(msg) Job DOMparser checkIn AbstractDagParser Dag Obj Job Obj Dag Info Job Info Dagwrapper Jobwrapper

  7. Version 1: Control Process Control Process UNREDUCED DagReducer Dag Table UNPREDICTED UNPREDICTED PredictionEngine UNFINISHED UNPREDICTED Planner SchedulingModule UNPLANNED Job, Strategy execute Job Table Sites Info UNPLANNED Exec. site info Input file info Output file info UNSENT Physical site info Site Tables JobWrapper Client ID Msg tpye Job info string Physical file info Message Table Check In / SUB_REQ Replica Table MessageWrapper OutMessageWrapper

  8. OutModule MsgHandler DOMparser Version 1: Sphinx Server (outgoing message) Msg ID Client Info Msg type message XML message Destination URL SOAP RPC OutgoingInterface SendRequest ACK ACK ACK MsgID Msg Type Client Info Message Next msg ID message XML message MessageWrapper Check Out MsgID Msg Type Client Info Message XML message Message

  9. Version 2 of Sphinx (current) • The current development cycle is represented in the next three slides • Essentially: • More planning tables have been added to the database warehouse • More planning modules have been added to the control process • A Data Management Component is being added to the Sphinx Server • Replica Management (via RLS and GridFTP) • The Monitoring Interface was upgraded to use MonALISA • We also want to investigate the possibility of making a “Clarens service” out of any arbitrary planning module in the control process • The Job Submitter, Job Tracking, and Condor-G/DAGMan (which currently form the Sphinx Client) is an example.

  10. Version 2:Sphinx Server with Sphinx Submission Client Sphinx Server VDT Client Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Job Tracker Job Predictor Policies DAG Predictor Job Submitter Accounting Job Admission Control Resource Prop. DAG Admission Control Condor-G/DAGMan Resource Hier. Resource Dist. DAG Data Repl. Planner Distance Job Execution Planner Entity-Name VDT Server Site Entity Hier. Resource Job-Entity Prop. Data Management Job DAG Tracker DAG Rel. Data Rel. Appl. Repl. Loc. Serv. Information Gatherer Job Tracking Grid Weather Monitoring Service Schedule Prediction Message

  11. Version 2:Sphinx Server with Sphinx UI Client Sphinx Server/VDT Client UI Machine Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Job Predictor Policies DAG Predictor Accounting Job Admission Control Resource Prop. DAG Admission Control Resource Hier. Resource Dist. DAG Data Repl. Planner Distance Job Execution Planner Entity-Name Condor-G/DAGMan Job Submitter VDT Server Site Entity Hier. Resource Job-Entity Prop. Job Tracker Data Management Job DAG Tracker DAG Rel. Data Rel. Appl. Repl. Loc. Serv. Information Gatherer Job Tracking Grid Weather Monitoring Service Schedule Prediction Message

  12. Version 2: Finite State Machine Control Process DAG: unreduced JOB: unpredicted Await Mesg. no yes DAG: unpredicted JOB: unpredicted DAG: unfinished JOB: unplanned Reduce DAG Plan Dag no no yes yes DAG: unpredicted JOB: unaccepted DAG: unfinished JOB: unsent Predict Job Plan Job no no yes yes DAG: unaccepted JOB: unaccepted DAG: unfinished JOB: remove Predict DAG Submit Job no no yes yes DAG: unaccepted JOB: unplanned DAG: remove JOB: remove Admit Job Remove Job no no yes yes DAG: unplanned JOB: unplanned Remove DAG Admit DAG no yes

More Related