1 / 7

Main Sphinx Design Concepts

Main Sphinx Design Concepts. There are two primary design components which comprise Sphinx The Database Warehouse The Control Process The Database Warehouse is the information depot for all of Sphinx All information is stored/retrieved via the database (or possibly multiple databases)

reid
Download Presentation

Main Sphinx Design Concepts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Main Sphinx Design Concepts • There are two primary design components which comprise Sphinx • The Database Warehouse • The Control Process • The Database Warehouse is the information depot for all of Sphinx • All information is stored/retrieved via the database (or possibly multiple databases) • There is no direct communication between different internal Sphinx modules • The API for all internal modules is SQL along with the database schema • The Control Process is the main point of activity • It is a loop which calls different Sphinx modules • All modules in the control process operate on stateful entities in the database, changing the state of an entity from one to another state • Sphinx operates as a Finite State Machine! • Uses data mining principles in the design: • The API for all modules is SQL along the database schema • Modules mine the database for information and then take some action.

  2. More Details • The components are currently split (packaged) in the following way: • Sphinx Client • Clarens Web Service (XML-RPC) + VDT-Client • Sends a Chimera DAX to Sphinx Server • Receives individual “concrete Jobs” from Sphinx Server • Submits “Job” (= “micro DAG”) to Condor-G/DAGMan • “micro DAG” (for now…could use Grid-shell or equivalent in the future) • Job to create environment • Job to move input data locally (if required) • Job to run application • Job to move output data (if required) • Job to register data in RLS • Job to clean-up environment • Sphinx Server • Clarens Web Service (XML-RPC) • Receives a Chimera DAX • Returns individual concrete “Jobs” (=“micro DAG”) • Provides heuristic algorithms and Satisfies constraints for: • policies, • input data existence, • required resources • grid weather

  3. Version 1 (current configuration):Sphinx Server with Sphinx Submission Client Sphinx Server VDT Client Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Job Tracker Job Predictor Policies DAG Predictor Job Submitter Accounting Job Admission Control Resource Prop. DAG Admission Control Condor-G/DAGMan Resource Hier. Resource Dist. DAG Data Repl. Planner Distance Job Execution Planner Entity-Name VDT Server Site Entity Hier. Resource Job-Entity Prop. Data Management Job DAG Tracker DAG Rel. Data Rel. Appl. Repl. Loc. Serv. Information Gatherer Job Tracking Grid Weather Monitoring Service Schedule Prediction Message indicates current place-holder implementation

  4. Version 1: Sphinx Control Process – Simple Finite State Machine DAG: unreduced JOB: unpredicted Await Mesg. no yes DAG: unpredicted JOB: unpredicted DAG: unfinished JOB: unplanned Reduce DAG Plan Dag no no yes yes DAG: unpredicted JOB: unaccepted DAG: unfinished JOB: unsent Predict Job Plan Job no no yes yes DAG: unaccepted JOB: unaccepted DAG: unfinished JOB: remove Predict DAG Submit Job no no yes yes DAG: unaccepted JOB: unplanned DAG: remove JOB: remove Admit Job Remove Job no no yes yes DAG: unplanned JOB: unplanned Remove DAG Admit DAG no yes

  5. Examples of how planning modules can be moved inside/outside the Sphinx Server/Client Sphinx Server VDT Client Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Job Tracker Job Predictor Policies DAG Predictor Job Submitter Accounting Job Admission Control Resource Prop. DAG Admission Control Condor-G/DAGMan Resource Hier. Resource Dist. DAG Data Repl. Planner Distance Job Execution Planner Entity-Name VDT Server Site Entity Hier. Resource Job-Entity Prop. Data Management Job DAG Tracker DAG Rel. Data Rel. Appl. Repl. Loc. Serv. Information Gatherer Job Tracking Grid Weather Monitoring Service Schedule Prediction Message

  6. Examples of how planning modules can be moved inside/outside the Sphinx Server/Client Sphinx Server/VDT Client UI Machine Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Job Predictor Policies DAG Predictor Accounting Job Admission Control Resource Prop. DAG Admission Control Resource Hier. Resource Dist. DAG Data Repl. Planner Distance Job Execution Planner Entity-Name Condor-G/DAGMan Job Submitter VDT Server Site Entity Hier. Resource Job-Entity Prop. Job Tracker Data Management Job DAG Tracker DAG Rel. Data Rel. Appl. Repl. Loc. Serv. Information Gatherer Job Tracking Grid Weather Monitoring Service Schedule Prediction Message

  7. Examples of how planning modules can be moved inside/outside the Sphinx Server/Client Sphinx Server/VDT Client VO Specific External Planning Service Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Job Predictor Job Execution Planner Policies DAG Predictor Accounting Job Admission Control Resource Prop. DAG Admission Control Resource Hier. Resource Dist. DAG Data Repl. Planner Distance Entity-Name Condor-G/DAGMan Job Submitter VDT Server Site Entity Hier. Resource Job-Entity Prop. Job Tracker Data Management Job DAG Tracker DAG Rel. Data Rel. Appl. Repl. Loc. Serv. Information Gatherer Job Tracking Grid Weather Monitoring Service Schedule Prediction Message

More Related