Disys 2.0 Transparent Remote Task Execution (044160)

1. 2.0 Disys 2.0Transparent Remote Task Execution(044160) June 2010

2. Supervisors & Staff • Supervisor: • OvedItzhak • Developers: • SaeedMhameed • Hani Ayoub

3. Reminder • Dysis: Generic distributed system, to execute tasks remotely With Focus on: • Transparency • Generality • Simplicity

4. Project Goal • Improve the system! • Make the system more General • Performance improvements • Improve management • Reliability • Consider • Backward compatibility

5. Improvements • Multitask support • Improved Scheduler • Executers queues monitoring and management • Full fault tolerance

6. Multitask support • Currently • Support only one type of tasks simultaneously • Disadvantage • Not exploiting execution power • Limiting the developer • Less generic • Goal • Give the user the ability to execute any task without changing the current system state • Example: • Task1: Crawl WebPages • Task2: Download WebPages • Task3: process the result

7. Multitask support (cont.) Result1 Result2 Result1 Result2 Base Executer 1 Base Executer 1 Final Result Final Result Current System Crawl Crawl Crawl Client Download Download Download Task 1 Task 2 Process Process Process

8. Multitask support (cont.) Result1 Result2 Result1 Result2 Base Executer 1 Base Executer 1 Final Result Final Result Future System Crawl Crawl Crawl Client Download Download Download Task 1 Task 2 Process Process Process

9. Improved Scheduler • Currently • Simple RR • Consider buffer load • Goal • Considers • Buffer Size • Executer machine power • How? • New scheduling algorithm based on:

10. Improved Scheduler (Cont.) • Calculating executer power using • # of processors • Processor strength • Benchmark • Calculate Effective buffer size using • Buffer size • Executer power Choose best executer to have minimum effective buffer utilization

11. Monitoring and management • Currently: • No monitoring and management provided of tasks queue. • Disadvantage: • User cannot modify or monitor tasks in real-time • Goal: • Give the user the ability to: • Add • Remove • Manipulate • Change priority • Monitor • Reorder any waiting task.

12. Full Fault Tolerance • Currently: • Partial fault tolerance. i.e. the system reliable from executer failures and network disruption • Goal: • Make the system fully fault tolerance, so it’ll fulfill: Each task submitted to the system will eventually be executed and result will be received by the corresponding client. • How? • If task is not pending or executer is dead for a predefined timeout -> reschedule task.

13. Performance and reliability measurement • The main improvements of the system are: • Performance • Reliability • How will we measure these improvements? • For performance: Run predefined benchmark on both current and future system, then compare results. • For reliability: Create unreliable environment, then make sure all tasks are executed.

14. Q&A 2.0

15. Backup – Improved Scheduler - HOW? • Definitions: • Buffer Size (BS): current number of tasks • Buffer Capacity (BC): number of tasks the buffer can hold • Buffer Efficiency (BE): actual efficiency of tasks buffer BE = (BS \ BC) • Executer Power (EP): Remote-machine strength Determined by: • Processors number • Processor frequency • Benchmark time EP = (Processors_Number*(Processor_Frequency/1000) ) Banchmark_Time

16. Backup – Improved Scheduler - HOW? (Cont.) • Definitions (cont.): • Max Executer Power (Max_EP): Max_EP = Max { EPs } • Power Percentage (PP):  PP = (EP \ Max_EP) • Effective Buffer Capacity (Eff_BC): Eff_BC = (BC * PP) • Effective Buffer Efficiency (Eff_BE): Eff_BE = BS \ Eff_BC Scheduler(T) <= Max { PP( Min { Eff_BE } ) }