Towards a Generic Deployment Framework for J2EE, OSGi, Web Services (or Everything)

1. Towards a Generic Deployment Framework for J2EE, OSGi, Web Services(or Everything) JACQUARD JACQUARD Tuesday, February 28th, 2006 ITEA S4ALL Task 3.5 Meeting, INT, Evry, France Philippe Merle Email: Philippe.Merle@inria.fr Jacquard Project - INRIA Futurs, Laboratoire d‘Informatique Fondamentale de Lille (LIFL), Université des Sciences et Technologies de Lille (USTL), Villeneuve d’Ascq, France

2. Un brin d’histoire • OpenCCM • Plate-forme de composants répartis CORBA • Automatisation du déploiement • Tout est spécifique au CCM • Identifier des abstractions logicielles pour construire des plates-formes de déploiement • Qu’est ce qui peut être factoriser en concepts/codes entre le déploiement OpenCCM et le déploiement Fractal ? • Voir articles [DECOR 2004], [LMO 2005] • Le travail présenté aujourd’hui est dans la poursuite de cette recherche du Graal

3. Agenda • Context • Deployment of distributed services and systems • Motivation • No generic deployment approach/tool/framework exists! • Objectives • Build a generic deployment framework • Principles • A fine grain component-based framework with Fractal • Illustrations • Some of the Fractal-based components • Work plan • Conclusion and perspectives

4. Context - Multiple Services and Providers • The user-centric approach considers the users’ services as gathering of the many services provided by the all-around world. • Within each involved platform, deploying these services is made easy. • Why not everything deployed in once? Banking Facilities Monitoring OSGi Gateway J2EE Server Directories Extra Stuff <insert yours> Web Server

5. We Would Like To Have Something Which… • Starts various processes on remote hosts • Enables complex configurations for the launched processes • Deploys applications’ services inside the hosting platforms • Adds some test / control over this deployment • Relays on nothing more than existing facilities • Operating systems, remote shell accesses, etc. • Via an easy to use, automation, extensible, and efficient approach!

6. But Heterogeneity At Every Layer JOnAS Oscar JOnAS JOnAS Application Server SSH Telnet ProActive OSGi Remote Access Linux Windows Mac OS Towards Users Service Towards Systems Operating System Dependencies for remote deployment/launching Requires a unified approach!

7. Motivation • Deployment of distributed applications is a complex task, e.g.: • Connect to remote hosts • Download application (and middleware) binaries • Launch middleware services and application servers • Set up environment variables • Start processes • Install application binaries into application servers/gateways • Configure and publish application services • Many other specific tasks!

8. Motivation - Related Work • Scripting is just a tool instead of a paradigm, e.g.: • How to reuse scripts? • Many work done to automatically deploy applications • Into application servers/gateways, e.g.: • J2EE, OSGi, Fractal, etc. • On distributed systems, e.g.: • OLAN, OpenCCM, ProActive, JADE, etc. • Via generic frameworks, e.g. • [DECOR 2004], etc. • Via model driven approaches, e.g. • [LMO 2005], ORYA, OMG DnC, etc.

9. Motivation - Related Work • No work cover the whole deployment process in a generic and technological independent way, e.g.: • J2EE/OSGi do not address the launching of J2EE/OSGi gateways • OSGi does not address deployment of distributed applications • ProActive only deploys ProActive-based servers and applications • OpenCCM only deploys CCM-based applications but not OpenCCM servers • JADE deploys J2EE systems but not JADE servers • Etc.

10. Motivation - Requirements • Providing a unified approach • Heterogeneity of technologies • Independent against granularity • Automation instead of manual operations • Distributed synchronizations, e.g.: • Connect to remote hosts before do anything • Set up environment variables before launch processes • Launch a transaction service before launch application servers • Launch a registry service before publish application services • Etc. • Describing instead of programming

11. Objectives • Build a generic deployment framework • Unified approach • Covering any task required to deploy • Independent against granularity • Distributed systems, applications, services, components, objects, etc. • Independent against technology • Usable with J2EE, OSGi, Web Services, Fractal, OpenCCM, etc. • Easy to use by end-users • Describing what must be deployed instead of how to do • Highly customizable and extensible by developers • Adapting the framework to specific purposes • Adding new concepts/mechanisms easily

12. Principles for a Unified Approach • A framework instead of a single tool • Build tools from the framework • Everything is reified as components • From end-user configurations to technical entities • Fine grain component-based approach • Micro components and interfaces • Intensive use of composition and sharing

13. Everything is Reified as Components remote shell service service execution platform • A hosting system is a component containing a remote access • This remote shell provides the interfaces to set variables and execute processes on the hosting system • An execution platform is a component managing applications and services, with many interfaces such as deployment, monitoring, configuration, and so on. hosting system

14. Everything is Reified as Components hosting system remote shell service service execution platform Reify and Control Network

15. Everything Reified as Components • Remote access protocols such as SSH, TELNET, etc. • File transfer protocols such as SCP, FTP, HTTP, etc. • Shells such as SH, CSH, WINBAT, etc. • Bound to a protocol • Shell variables and commands • Bound to a shared shell • Servers such as registry services and transaction managers (RMI, CORBA, WS), etc. • Composed of shell variables and commands • Application servers as JOnAS, OSGi, etc. • Composed of shell variables and commands • Application beans as J2EE jars, OSGi bundles, etc. • Bound to a shared application server • Etc.

16. Everything Reified as Components protocol: SSH shell: SH host server: JOnAS bean: EJB Reified System Real System EJB JOnAS SH SSHd

17. Technical Approach • This generic deployment framework is • Designed with the Fractal component model • Each deployment concept/mechanism is a Fractal component • Implemented with the Fraclet annotation framework • Leverage Component-Based Software Engineering • Configured with Fractal ADL • End-users’ configurations • Developers’ components • Managed with Fractal Explorer • Distributed with Fractal RMI • Could benefit from any other Fractal-based work • AOKell, FAC, etc.

18. Technical Architecture hosting system service remote shell service execution platform Fraclet End-User Configuration Components Components Deployment Components Fractal Explorer Translation Fractal ADL Manage Instantiate Fractal ADL Deploy Network

19. A Simple End-User ConfigurationThe End-User Description • MySystem = { • MyBean = BEAN { • location = URL(http://www.lifl.fr/bean.jar) ; • server = JONAS { • home = JONAS_HOME(/usr/jonas) ; • host = HOST { • hostname = HOSTNAME(host.lifl.fr) ; • protocol = SSH ; • shell = SH ; • }; }; }; }; • An easy to use end-users’ configuration language • Describing what the configuration is instead of how to deploy it

20. A Simple End-User ConfigurationThe Fractal Components location: URL protocol: SSH shell: SH MyBean: BEAN server: JONAS host: HOST home: JONAS_HOME hostname: HOSTNAME MySystem

21. A Simple End-User ConfigurationThe Fractal ADL Definition • <?xml version="1.0" encoding="ISO-8859-1" ?><!DOCTYPE definition PUBLIC "-//objectweb.org//DTD Fractal ADL 2.0//EN” "classpath://org/objectweb/fractal/adl/xml/standard.dtd"> • <definition name="MySystem"> • <component name=“MyBean" definition="BEAN"> • <component name="location" definition="URL(http://www.lifl.fr/bean.jar)"/> • <component name="server" definition="JONAS"> • <component name="home" definition="JONAS_HOME(/usr/jonas)"/> • <component name="host" definition="HOST"> • <component name="hostname" definition="HOSTNAME(host.lifl.fr)"/> • <component name="protocol" definition="SSH"/> • <component name="shell" definition="SH"/> • </component> • </component> • </component> • </definition>

22. A Simple End-User ConfigurationThe Expected Runtime Behavior • Deploying MyBean implies to automatically • Connect to host.lifl.fr via the SSH protocol • Setup the JONAS_HOME variable to /usr/jonas • Add $JONAS_HOME/bin/unix to the PATH variable • Launch the command jonas start • Download http://www.lifl.fr/bean.jar • Launch the command jonas admin –a bean.jar

23. A More Complex End-User Configuration • MySystem = ITEA_S4ALL { • hosts = { • ... • }; • servers = { • ... • }; • beans = AUTO_INSTALL { • ... • }; • };

24. A More Complex End-User Configuration • MySystem = ITEA_S4ALL { • hosts = { • M = HOST { • hostname = HOSTNAME(host.lifl.fr) ; • protocol = SSH ; • shell = SH ; • }; • … other host definitions ... • }; };

25. A More Complex End-User Configuration • MySystem = ITEA_S4ALL { • servers = { • jonas_on_M = JONAS { • home = JONAS_HOME(/usr/jonas) ; • host = hosts/M ; • }; • oscar_on_M = OSCAR { • home = OSCAR_HOME(/usr/oscar) ; • host = hosts/M ; • }; • ... Other server definitions … • }; }; • Sharing: • The M host • The SSH channel • The SH shell

26. A More Complex End-User Configuration • MySystem = ITEA_S4ALL { • beans = AUTO_INSTALL { • bean1 = BEAN {location = LOCAL_FILE(bean1.jar) ;server = servers/jonas_on_M ; • }; • bean2 = BEAN {location = LOCAL_FILE(bean2.jar) ;server = servers/jonas_on_M ; }; • bean3 = BEAN {location = LOCAL_FILE(bean3.jar) ;server = servers/oscar_on_M ; }; • bean4 = BEAN {location = LOCAL_FILE(bean4.jar) ;server = servers/oscar_on_M ; }; • ... Other bean definitions … • }; }; Ask for auto installation JOnAS on host Mshared by beans OSCAR on host Mshared by beans

27. A More Complex End-User ConfigurationThe Expected Runtime Behavior • Connect to host.lifl.fr via the SSH protocol • Setup the JONAS_HOME variable to /usr/jonas • Add $JONAS_HOME/bin/unix to the PATH variable • Launch the command jonas start • Download http://www.lifl.fr/bean1.jar • Launch the command jonas admin –a bean1.jar • Download http://www.lifl.fr/bean2.jar • Launch the command jonas admin –a bean2.jar • Setup system variables related to OSCAR • Launch OSCAR • Execute OSCAR commands to install bean3.jar and bean4.jar • Other execution sequences and/or parallelization are possible

28. Everything is Reified as Components hosting system remote shell service service execution platform Control Network

29. Everything is a Service(Not a Web Service) • Each component has a Service interface to control the life cycle of the real entity reified • interface Service { void start(); void stop();} • Semantics depends on components/implementations • Protocol.start ~ initializes the network connection • Protocol.stop ~ shutdowns the network connection • Server.start ~ launches the server process • Server.stop ~ stops the server process • Bean.start ~ launches the bean into the application server • Bean.stop ~ uninstalls the bean from its server

30. Composition of Services Composition Bean JOnAS Bean Protocol SSH Shell SH Host .inria.fr Bean OSCAR Bean

31. Orchestration of Services • Goal • Address distributed synchronizations of composition of services • Orchestration • Implements the Service interface • Schedules execution of services automatically • Implementation • Introspects the composition and bindings between components to plan the execution of services • Various execution strategies • Sequential • Parallel

32. Orchestration of Services Orchestration Bean JOnAS Bean Protocol SSH Shell SH Host .inria.fr Bean Service OSCAR Bean

33. Orchestration of Services Orchestration Bean JOnAS (6) start (5a) start Bean (6) start Protocol SSH Shell SH Host .inria.fr (1) start (4) start Bean (2) start (3) start (6) start OSCAR (5b) start Bean (6) start

34. Orchestration of Services Orchestration Bean JOnAS (2) stop (3) stop Bean (2) stop Protocol SSH Shell SH Host .inria.fr (1) stop (4) stop Bean (6) stop (5) stop (2) stop OSCAR (3) stop Bean (2) stop

35. Protocol Components • Reify and encapsulate protocols to access the real system • interface Protocol { void send(String command); } • Possible implementations • LOCAL by wrapping an external local shell • SSH by wrapping • an external local ssh process • the BSD open source JSCH library • http://www.jcraft.com/jsch/index.html • TELNET by wrapping • an external local telnet process • the open source JTelnet library • http:// Source Forge • the Apache open source Jakarta Commons Net library • http://jakarta.apache.org/commons/net Service Protocol

36. Internet Protocol Components Internet Protocol IP Host Name Basic Protocol as SSH, TELNET IP Port

37. Implementation of the IP Port Component with Fraclet • /** @fractal.itf name=“port” */interface IpPort { int getIpPort();} • class PORT implements IpPort { /** @fractal.ac */ protected int port; • public int getIpPort() { return port; }} • Fractal AttributeController interface & implementation, and Fractal ADL definitions generated automatically by Fraclet :-)

38. Implementation of the IP Host Name Component with Fraclet • /** @fractal.itf name=“hostname” */interface IpHostName { String getIpHostName();} • class HOST_NAME implements IpHostName { /** @fractal.ac */ protected String hostname; • public String getIpHostName() { return hostname; }} • Fractal AttributeController interface & implementation, and Fractal ADL definitions generated automatically by Fraclet :-)

39. Implementation of the Basic Internet Protocol Components with Fraclet • abstract class InternetProtocol implements Service, Protocol { /** @fractal.bc */ protected IpHostName hostname; /** @fractal.bc */ protected IpPort port;} • class Telnet extends InternetProtocol { public void start() { . . . hostname.getIpHostName() . . . . . . port.getIpPort() . . . } public void stop() { . . . } public void send(String command) { . . . }} • Ditto for all possible implementations, e.g.: • TELNET, SSH, etc. • Via an external process or an embedded Java library

40. Shell Components • Reify shells executed in the real system • interface Shell { void set_variable(String name, String value); void unset_variable(String name); void execute(String command); } • Each shell is bound to a single Protocol component • Behavior • Construct commands according to the shell syntax • Format path separator, i.e. / \ : ; • Possible implementations • SH for Borne Unix shells • CSH for C Unix shells • WINBAT for Windows command shells Shell Protocol

41. The HOST Component • Provide an abstract composition of (Shell, InternetProtocol, Hostname)<definition name=“HOST”> <component name=“hostname” definition=“IpHostName”/> <component name=“protocol” definition=“InternetProtocol”/> <component name=“shell” definition=“ShellComponent”/> <binding client=“shell.protocol” server=“protocol.protocol”/> <binding client=“protocol.hostname” server=“hostname.hostname”/> </definition> • Simplify the configuration for end-users<component name=“MyHost” definition=“HOST”> <component name=“hostname” definition=“HOSTNAME(host.lifl.fr)”/> <component name=“protocol” definition=“SSH”/> <component name=“shell” definition=“SH”/> </component> shell protocol hostname

42. The VARIABLE Component • Reify shell variables • Each variable is bound to a shared Shell component • Possible implementation class VARIABLE implements Service {/** @fractal.bc */protected Shell shell;/** @fractal.ac */protected String name;/** @fractal.ac */protected String value;public void start() { shell.set_variable(name, value); }public void stop() { shell.unset_variable(name); } } • Possible instantiations • JONAS_HOME(path) = VARIABLE(JONAS_HOME, path) • OSCAR_HOME(path) = VARIABLE(OSCAR_HOME, path) VARIABLE Service Shell

43. The Command Component COMMAND Command • Reify shell commands on the real system • interface Command { void execute(Map arguments); } • Each command is bound to a shared Shell component • Possible implementation class COMMAND implements Command {/** @fractal.bc */protected Shell shell;/** @fractal.ac */protected String command;private String computeCommand(Map arguments) { . . . }public void execute(Map arguments) { shell.execute(computeCommand(arguments)); }} • Possible instantiations • COMMAND(echo hello world!) • COMMAND(jonas start) • COMMAND(jonas admin –a %url%) Shell

44. Server Components • Reify a server/daemon executed in the real system • Examples • Middleware servers • Registry • Transaction Manager • Apache servers • Various implementations • Wrapping a protocol to interact with the remote server • Bound to a shell • Etc. SERVER Service

45. Implementation of the SERVER Component by Composition SERVER Server Impl * SERVER depend VARIABLE * shell COMMAND start Service COMMAND stop

46. Implementation of the SERVER Component by Composition (Java code) • class ServerImpl implements Service { • /** @fractal.bc signature=“Service” */ protected Map required-servers; • /** @fractal.bc signature=“Service” */ protected Map variables; • /** @fractal.bc */ protected Command start-command; • /** @fractal.bc */ protected Command stop-command; • public void start() { start-command.execute(null); • } • public void stop() { stop-command.execute(null);} }

47. Other Concerns to Address • Display output from remote hosts • Various GUI • Also Fractal components • Configurable by end-users • Take care of system resource management • Threads and Sockets are limited • Study work done on DREAM • Use AOKell to build new controllers for system resources • Quid for deployment of large scale systems • Distribute Fractal components into several JVM/host • To address system resource limitation • Using Fractal RMI • Transactional deployment • GoTM framework • Dynamic reconfiguration • Fractal Explorer to manage components, as done by JADE • Rules for autonomous computing

48. Work Plan • Proof of concept already done • Areski Flissi and Philippe Merle involved • Engineer recruitment from March 1st 2006 • Define Fractal interfaces and components • Develop micro-components with Fraclet • Compose components with Fractal ADL • Implement Fractal ADL translator • Implement management with Fractal Explorer • Evaluate on various ITEA S4ALL scenarios

49. Conclusion • A deployment framework • Generic • Independent against technology and granularity • Easy to use • Highly customizable and extensible • Fine grain component-based approach • Each deployment mechanism is reified as a component • Intensive use of the Fractal component model and tools • Composition and sharing • Julia, AOKell, Fraclet, Fractal ADL, Fractal Explorer, etc. • Will be developed soon • Prototyping already done • Implementation details could change but not the overall approach

50. Perspectives • Deployment on Grid’5000 • Specific components for OAR and KaDeploy services • Deployment of OpenCCM • Specific components for OpenCCM daemons • Automation on PDA prototyped • Deployment of Fractal • Specific components for Fractal RMI Registry and Server • Autonomous computing • Rules for dynamic reconfiguration – Ph.D. Jérémy Dubus