Sphinx Server

1. Current Sphinx Client/Server Sphinx Server Sphinx Client Data Warehouse Message Interface Message Interface Multi-threaded Control Process DAG Reducer Prediction Engine Submitter Planner Generic Grid Site Resource Monitoring Service Information Gatherer

2. Sphinx Client/Server with V2 Components Sphinx Server Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Submitter Admission Control Planner Generic Grid Site Resource Data Management Monitoring Service Information Gatherer

3. Stand alone Sphinx Server with light Client Sphinx Server Sphinx Client Data Warehouse Message Interface Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Generic Grid Site Submitter Resource Data Management Monitoring Service Information Gatherer

4. Data Warehouse Sphinx Server • Message Tables • Policy Information Tables • Entity Accounting Tables • Work (DAG, Job, etc) Tables • State • Unreduced • Unpredicted • Unaccepted (not admitted) • Unplanned (not yet feasible) • Unsent (assigned) • Unfinished (executing) • Remove (accounted or rejected) • Entity • Resource Property Requirements • Dependency Requirements • Do not start before “event” • I/O • QoS Requests • Resource Assignment for next k steps • Application Tables (in general, distributed…) • Transformation Catalogue • Profiles (CPU, Disk, Bandwidth requirements) • History (CPU, Disk, Bandwidth, actual use) • Data Tables (in general, distributed…) • Replica Catalogue (from RLS) • Profiles (Size, Bandwidth requirements) • History (Access rate, etc) • Resource Property Tables • CPU, Disk, Network Connection, middleware configuration, etc • Grid Weather Tables • CPU View • Storage View • Bandwidth View Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer What about including Virtual Data Tables?

5. Input (incoming) DAG Requirements QoS requests Status requests Output (incoming) DAG Table (“unreduced”) Job Table (“unreduced”) application Requirements QoS Input (outgoing) Message Output (outgoing) Status updates Message Interface Component Sphinx Server Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer

6. DAG Reducer (Decomposer?) Sphinx Server • Input • DAG (“unreduced”) • Available (existing) data • Action • Remove DAG nodes for which input data exists • Add Jobs to Job table • Output • Pruned DAG (“unpredicted”) • Jobs (“unpredicted”) Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer

7. Prediction Engine Component Sphinx Server • Input: • “unpredicted” Job • Requirements (if available) • CPU hours • Storage hours • Bandwidth • Profile/History • Output: • Estimated Resource Usage • “unaccepted” job Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer

8. Admission Control Component Sphinx Server • Input • Entity Account Information • “unaccepted” DAG Request • Requirements • QoS • Predictions of Jobs (decomposed DAG) Resource Usages • Policy Constraints • Current schedule for next k steps • Output • Reject DAG Request • “Remove” DAG • “Remove” Jobs • Accept DAG with Estimated QoS • “Unplanned” DAG • “Unplanned” Jobs Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer

9. Planning Component Sphinx Server • Job Planner (independent “work” that can be fully resolved onto resources at time t) • Input • “unplanned” Job Information • Requirements • Dependencies (I/O) • Job QoS Requests • Entity Account Information • Policy Constraints • Available Data • Available Applications • Available Resource Properties • Prediction of Job Resource Usage • Output • QoS Deliverable • “unsent” Job(s) schedule for the next k steps • DAG Planner (intra-dependent “work” that can not all be fully resolved onto resources at time t) • Input • “unplanned” DAG Information • Requirements • DAG QoS Requests • Current Job Schedule for next k steps • Output • QoS Deliverable • Modified Job Schedule for next k steps Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer

10. Job Submission Component (also part of Sphinx Client) Sphinx Server • Input • Fully planned Job: • Job state “unsent” • Assigned gatekeeper and jobmanager • I/O data movement locations • Action • Construct Job DAG • Set up environment • Import input data (if necessary) • Publish to DMC (if necessary) • Run application • Export output data (if necessary) • Publish to DMC (if necessary) • Cleanup • environment • Local I/O data (if necessary) • Submit to Condor-G/DAGMan • Job state “unfinished” • Monitor Job statistics • Output • Exit status (if available) • Final Job statistics (resource usage) • CPU • Storage • Bandwidth • Entity account update • Job state “remove” Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer

11. Data Management Components Sphinx Server • “User Demanded Movement” Module • Input • Source(s) • Destination(s) • Output • DMC update • “Pre-Planned Movement” Module (most important to Sphinx) • Input • Available Data • Available Resources • Prediction of Data Access Patterns • DAG (“reduced”) • Jobs (“unsent”) • Job(s) schedule for next k steps • Output • Data Movement • DMC update • Prediction Module • Input: • Historical Data Access Patterns • Output: • Future Data Access Patterns Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer

12. Information Gathering Module Sphinx Server • Resource Property Module • Input • Grid Information Systems (such as MDS using GLUE Schema, etc) • Output • Resource Property views • Job Tracking Module • Input • Job Monitoring information (such as Condor-G, etc) • Output • Work Views • Grid Weather Module • Input • Grid Monitoring systems (such as GEMS, MonALISA, etc) • Output • Grid Weather Views • CPU • Storage • Bandwidth • Grid Weather Prediction Module • Input: • Grid Weather History • Output: • Grid Weather Forcast for next k steps Data Warehouse Message Interface DAG Reducer Multi-threaded Control Process Prediction Engine Admission Control Planner Submitter Data Management Information Gatherer