1 / 37

KARMA with ProActive Parallel Suite 12/01/2009 Air France, Sophia Antipolis

KARMA with ProActive Parallel Suite 12/01/2009 Air France, Sophia Antipolis. Solutions and Services for Accelerating your Applications. ProActive Parallel Suite .  Speed-up applications by distributing them over a set of resources .  The result of 9 years of R&D at INRIA.

dafydd
Download Presentation

KARMA with ProActive Parallel Suite 12/01/2009 Air France, Sophia Antipolis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. KARMA with ProActive Parallel Suite12/01/2009Air France, Sophia Antipolis Solutions and Services for Acceleratingyour Applications

  2. ProActive Parallel Suite Speed-up applications by distributing them over a set of resources The result of 9 years of R&D at INRIA

  3. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  4. ProActive Technology Integration • Simple and Seamlessintegration • Installation : one directory • Deployment : one XML file (GCM Deployment) • Possible automated agent-based deployment • Support for common standards • Connection/Acquisition: Ssh, Rsh, LSF, PBS, SGE,… • Communication: RMI, RMI/SSH, HTTP, Ibis,… • Export any activity as a WebService • J2EE (in and out) • Validated on common JDKs 1.5, 1.6 and 1.7 • Activities are monitored through standard JMX • Compliant with common monitoring tools (TIBCO Hawk)

  5. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  6. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  7. ProActive Scheduling Managed environment for distributed computation

  8. Scheduler / Resource Manager Overview • Multi-platform Graphical Client (RCP) • File-based or LDAP authentication • Static Workflow Job Scheduling, Native and Java tasks, Retry on Error, Priority Policy, Configuration Scripts,… • Dynamic and Static node sources, Resource Selection by script, Monitoring and Control GUI,… • ProActive Deployment capabilities : Desktops, Clusters, ProActive P2P,…

  9. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  10. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  11. Parallelism with ProActive Scheduling • Computations are defined as Jobs made of Tasks • Bag of Tasks • Tasks Flow • ProActive application Task 2(input 2) Task 1(input 1) Task 1(input 1) res2 res1 Task 3(res1,res2) Functional // Technical // Task 1(input 1) Task 2(input 2) Task N(input N) …

  12. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  13. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  14. Defining Tasks Tasks can be defined in Java or as Native Application • Dynamic Task creation through Java API JavaTaskaTask = new JavaTask(); aTask.setName("task 1"); aTask.setRerunnable(2); aTask.setRestartOnError(RestartMode.ELSEWHERE); aTask.addArgument("foo",new Boolean(true)); aTask.setExecClassName("com.activeeon.Compute"); aTask.setPreScript("/path/to/script_file"); • Can attach pre/post script to a task • Any JSR-223 supported language (JS, Python, Ruby,…) • Used for configuration and data transfer  Data are handled separately

  15. Defining Jobs Jobs can be defined in Java or XML • Dynamic job creation through Java API TaskFlowJob job = new TaskFlowJob(); job.setName("job_name"); job.setPriority(JobPriority.NORMAL); job.setCancelOnError(true); job.addTask(new NativeTask(cmd));  Can easily develop tools that generate and submit jobs • Specialized • Automated • Static description in XML descriptor  Can be defined by any non specialized user

  16. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  17. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  18. Executing Jobs • Submission through… • Java API • Graphical interface • Command line • Recovery is fully configurable with User/System level distinction • Host failure : automatically re-dispatched with needed input • Number of retries bounded by the administrator • Taskfailure : can be either… • Automatically re-dispatched with needed input • Number of retries bounded by the user • Aborted (Task) : exception is transmitted as task result • Aborted (Job) : job is fully canceled • Scheduler itself is recoverable from underlying database

  19. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  20. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  21. Monitoring Jobs • System and ProActive level monitoring • Through standard JMX • Provide generic tool for monitoring and reporting • Compliant with common monitoring tool

  22. Monitoring Jobs • System and ProActive level monitoring • Through standard JMX • Provide generic tool for monitoring and reporting • Compliant with common monitoring tool • Scheduler level monitoring • On-going porting from MVC to JMX

  23. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  24. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  25. Controlling Jobs • Fine grain control through GUI and API Job level (User) Scheduler level (Admin)

  26. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  27. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  28. Specializing the Scheduler • Business Level specialization and automation • ActiveEon provides business specific interface for the Scheduler • Bio-technology (Gold, MapRead) • Finance • MatLab/Scilab

  29. Specializing the Scheduler • Business Level specialization and automation • ActiveEon provides business specific interface for the Scheduler • Bio-technology (Gold, MapRead) • Finance • MatLab/Scilab • User-level hooks to (very) easily specialize the Scheduler • Plug-in for RCP client • e.g. ActiveEon’s Gold module • Result preview mechanism • Can preview result in the scheduler GUI (textual/graphical) • Can attach any external application to a task for result display

  30. Specializing the Scheduler • Business Level specialization and automation • ActiveEon provides business specific interface for the Scheduler • Bio-technology (Gold, MapRead) • Finance • MatLab/Scilab • User-level hooks to (very) easily specialize the Scheduler • Plugin for RCP client • e.g. ActiveEon’s Gold module • Result preview mechanism • Can preview result in the scheduler GUI (textual/graphical) • Can attach any external application to a task for result display

  31. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  32. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  33. ProActive Resource Manager • Rely on ProActive Programming features • GCM deployment standard • Agent based deployment • Support for many LAN / WAN / Grid / Cloud standards… Cluster MultiCPU/Core Clusters / Grille Clouds / RAS Desktop Grid Straightforward Upgrade

  34. ProActive Resource Manager • Rely on ProActive Programming features • GCM deployment standard • Agent based deployment • Support for many LAN / WAN / Grid / Cloud standards… • Deal with resources heterogeneity with Selection Scripts • User defined : no need for system reconfiguration • Any JSR-223 supported language (JS, Python, Ruby,…) • Can rely on dynamic decision or on static host description • Optimized (probabilistic selection of candidates)

  35. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  36. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

  37. Thank you for your attention • Demo

More Related