Markus Völter voelter@acm voelter.de

Using DSLs in Practice www.mdsd-buch.de Benefits and Challenges Markus Völtervoelter@acm.orgwww.voelter.de

About me Markus Völter voelter@acm.org www.voelter.de • Independent Consultant • Based out of Heidenheim, Germany • Focus on • Software Architecture • Middleware • Model-Driven SoftwareDevelopment

C O N T E N T S • Introduction • Benefits • Challenges and Experiences • Process • Technical • Adoption • More or less Open • Summary

Domain Driven Development • Domain Driven Development is about making software development more domain-related as opposed to computing related. It is also about making software development in a certain domain more efficient.

A DSL and its context several Metametamodel target subdomains software software designexpertise architecture architecture bounded area of partial knowlege/interest composable multiple knowledge viewpoint multi-step transform Domain single-step compile semantics Model Ontology interpret no precise/ Domain roundtrip executable Specific Language graphical Metamodel textual

How DSLs are usually used • Developer develops model(s)based on certain metamodel(s), expressed using a DSL. • Using code generation templates, the model is transformed to executable code. • Alternative: Interpretation • Optionally, the generated code is merged with manually written code. • One or more model-to-model transformation steps may precede code generation.

DSLs in Industry • Model-Driven Development (MDSD)pragmatic technology, process building blocks • OMG’s Model-Driven Architecture (MDA)standardization effort, technology-focus, platform independence, m2m transformations • Microsoft’s Software Factories (SF)framework for domain-specific IDE tooling, DSLs are part of this approach • Generative Programming (GP)traditional small scale, technology focused • Language-Oriented Programming (LOP)integrate DSLs into IDE with editors, debuggers, symbolic integration

Reasons for using DSLs • You want to provide a way for your domain-experts to formally specify their knowledge, and to provide a way for your technology people to define how this is implemented (using model transformations). • You might want to provide different implementations (i.e. more concrete models) for the same model, perhaps because you want to run it on different platforms (.NET, Java, CORBA). • You may want to capture knowledge about the domain, the technology, and their mapping in a clear, uncluttered format. • In general, you don’t want to bother with implementation details when specifying your functionality.

MDSD Benefits I • Models are free of implementation artifacts – they directly represent domain knowledge and are thus reusable. • Implementations for various platforms can be generated in principle – the technology change problem is adressed to some extend. • Technology freaks and domain experts can take care of „their business“ (transformations and models, respectively) and need to care of each other‘s problems only in a limited way. • Domain experts can play a direct role in development since they can more easily understand models expressed with a DSL as opposed to implementation code. Domain Experts play the central role they deserve!

MDSD Benefits II • Development becomes more efficient since repetitive implementation code can be generated automatically. • Architectural contraints/rules/patterns can more easily be enforced, since the they are embedded in the templates rather than just being documented (and ignored).This is especially important in really large teams, often in the context of Product-Line Engineering and Software System Families. • Transformer/Generator can address cross-cutting concerns (just like an aspect weaver)

Benefits for Software Quality • DSLs, if they describe software structure/architecture requires an explicit, well-defined architecture. Defining an architecture this way improves the quality of the system (indpendent of whether it is generated or not). • Transformations capture expert knowledge. The generated code reflects this expert knowledge uniformly. • An DSL-based Archtitecture defines a strict programming model for the manually developed parts – again, uniformity and constrained-ness improves quality. • Generator does not produce accidental errors – either things are always right or always wrong. This makes finding errors easier!

Benefits for Software Quality II • In general, defining a DSL forces you to take care of many good things, which you‘d like to have in any application development project: • Domain/Application Scoping • Variability Management • Well-Defined Software Architecture, Architecture Metamodelling • Trying to build a DSL/generator for a domain/target architecture enables your understanding of the domain/target architecture. This in itself is a huge benefit.

DSL/Code Generation “Financial Benefits” • „Normal“ Implementation Effort

DSL/Code Generation “Financial Benefits” II • Realistic DSL based Implementation Effort

DSL/Code Generation “Financial Benefits” III • Ideal DSL based Implementation Effort

DSLs/Generation and Agility • Agile Manifesto:We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: - Individuals and interactions over processes and tools - Working software over comprehensive documentation - Customer collaboration over contract negotiation - Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. • DSL-based Models are no „paperwork“, they are the code which is translated to executable code automatically • Agility does not oppose tools in general – compilers are ok, model transformers are a kind of compiler

MDSD and Agility II • Project automation (ant, cruisecontrol) is ok in „agile minds“, so is automation of the writing of repetitive code • Automation of the development process makes responding to change easier and faster (single source principle). • Changes in the model respond to changes in the functional requirements • Changes in the templates/transformations can be used to evolve the architecture • The customer on-site can be integrated better, if we have languages that are better related to domain concepts as opposed to 3GL code or the like. • Pair programming between developer and domain expert is more realistic.

MDSD and Agility III • Tests can still be written manually (even before generation), generators can help is building mocks or scenarios • We have done Test-Driven, Model-Driven Development • We do not recommend a waterfall that first builds metamodel/DSL/generators and then builds apps, rather, both are iteratively evolved in parallel. • Domains Architectures are based on experience, not based on „big design upfront“ • Developers can do what they can do best: • Some deal with applications and customer requirements, • Others deal with technical architecture, platforms and generators • So: There is no conflict between Agility and DSLs/Generation!

Teaming issues • Using DSLs is not very different from “normal” programming – every developer can basically do it. • Defining DSLs is, however, something completely different: • Finding the „right“ abstractions, defining metamodels, keeping the various metalevels sorted, etc. is not everybody‘s business. • Some of the tools to define metamodels, DSLs, generators and model-2-model transformations are not (yet) intuitively usable. • Therefore I recommend to keep DSL/generator development to a limited group of people in your project.

Teaming issues II - Roles • Not all of these roles are necessary in every project, of course • However, as an example, there is a fundamental difference between those who understand the domain and its abstractions (left branch) compared to those who know how to best use some platform technology (right branch)

Iterative Dual Track Development • Develop DSL/Generator and at least one application at the same time. • Establish rapid feedback from application developers to domain architecture developers. • Develop both aspects iteratively and incrementally. Use strict timeboxing. • Infrastructure develops iteration n+1 whereas application developers use iteration n. • Introduce new DSL/Generator releases only at the beginning of iterations.

Iterative Dual Track Development II - Roles • Note that the „men“ inthe diagram on the right are roles, youcan easily have someof them be handledby the same person!

Extract the Infrastructure • Before startingITERATIVE DUAL-TRACK DEVELOPMENT, Extract the transformations from manually developed application. • Either, start by developing this prototype conventionally, then build up the DSL/Generator infrastructure based on this running application, • Or extract the code from applications developed in the respective domain before doing MDSD (but only if the quality is sufficiently good!)

How do I come up with a good metamodel? • Incrementally! • Based on experience from previous projects, and by „mining“ domain experts. • A very good idea is to start with a (typically) very well known domain: the target software architecture (platform) Architecture-Centric DSLs see below, Cascading

How do I come up with a good metamodel? II • In order to continuously improve and validate the metamodel for a domain, it has to be exercised with domain experts as well as by the development team. • In order to achieve this, it is a good idea to use it during discussions with stakeholders by formulating sentences using the concepts in the meta model. • As soon as you find that you cannot express something using sentences based on the meta model, • you have to reformulate the sentence • the sentence’s statement is just wrong • you have to update the meta model. • (Based on Eric Evans’ Ubiquitous Language)

How do I come up with a good metamodel? III • Example: • A component owns any number of ports. • Each port implements exactly one interface. • There are two kinds of ports: required ports and provided ports. • A provided port provides the operations defined by its interface. • A required port provides access to operations defined by its interface.

One DSL is not enough • Most systems can be structured into various • partitions: functional subsystems • subdomains: technical aspects • It is hardly possible to describe each of these with the same DSL. • You will need to come up withseparate DSLs • … that have to be „connectable“in order to build the complete system

One DSL is not enough II • Structure your system into several technical subdomains such as persistence, GUI, deployment. • Each subdomain should have its own meta model and specifically, its own suitable DSL. • Define a small number of gateway metaclasses, i.e. meta model elements that occur in several meta models to help you join the different aspects together.

One DSL is not enough III - Example

Rich Domain-Specific Platform • Define a rich domain-specific application platform consisting of • Libraries • Frameworks • base classes • interpreters, etc. • The transformations will “generate code” for this domain-specific application platform. • As a consequence, the trans-formations become simpler. • DSLs and Frameworks are two sides of the same coin

Rich Domain-Specific Platform II - Integration • A) Generated code can call non-generated code contained in libraries • B) A non-generated framework can callgenerated parts. • C) Factories can be used to „plug-in“ the generated building blocks • D) Generated classes can also subclass non-generated classes. • E) The base class can also contain abstract methods that it calls, they are implemented by the generated subclasses(template method pattern)

Transformations as first class citizens • Transformations (and Templates) are central assets in MDSD. You should treat them accordingly. • Transformations should be versioned. • Refactor transformations to keep them current and well organized. • Modularize transformations, e.g. using object-oriented concepts such as encapsulation, polymorphism, inheritance, etc.

External Model Markings (AO-Modelling) • In order to allow the transformation of a source model into a target model (or to generate code) it is sometimes necessary to provide “support” information that is specific to the target meta model. • Example: Entity Bean vs. Type Manager • Adding these to the source model “pollutes” the source model with concepts specific to the target model. • MDA proposes to add “model markings”, but this currently supported well by only very few tools. • Instead, we recommend keeping this information outside of the model (e.g. in an XML file); the transformation engine would use this auxiliary information when executing the transformations.

Testing through Model Verification • In many cases it is possible to detect design errors already in the models. This step is called model verification. • The most „extreme“ form is to interpret and simulate the whole model; this is however, not simple to achieve. • However, it is easily possible to verify design constraints in the model before model transformation or code generation steps are done. • So, a DSL definition includes checks („constraints“) that determine if a model is correct. • Note that those constraints report errors in a language that is on the same abstraction level as the DSL – i.e. no low level compiler errors!

Testing through Model Verification II - Example • Example Metamodel

Testing through Model Verification III – Example cont‘d • Verifications in the metamodel (Implemented) public class ECInterface extends generatorframework.meta.uml.Class { public String CheckConstraints() { Checks.assertEmpty( this, Attribute(), "must not have attributes." ); } // more … } public class Component extends generatorframework.meta.Class { public String CheckConstraints() { Checks.assertEmpty( this, Operation(), "must not have attributes." ); Checks.assertEmpty( this, Generalization(), "must not have superclasses or subclasses." ); Checks.assertEmpty( this, Realization(), "must not implement any interface." ); Checks.assertUniqueNames( this, Port(), "a component's ports must have unique names." ); } // more … }

Generation vs. Interpretation I • A model that can be used to generate code, can always also be interpreted. • The usual tradeoffs apply: Performance, Changeability at runtime • There are, however, additional things to consider: • When generating code, an explicit intermediate step (i.e. the generated code) is available. Might make things more easily understandable, since the structure is often well known (although this is not always true • Debugging is often easier, since you can debug „one metalevel down“ compared to an interpreter • In many cases, the platform (J2EE, Embedded Osek, etc.) expects certain code structures. It is very hard to run an interpreter in these constraints.

Generation vs. Interpretation II • From my experience a combination of both approaches makes sense in the following way: • Code generation is used to generate structural artifacts (components, configuration files, documentation, etc.). Components are artifacts that provide interfaces, require other interfaces, etc. • Different kinds of components use different metaphors for their implementation, such as • A rule engine component uses business rules to verify business transactions • A state chart based component manages persistent, long running processes • A mapping component is used to map data to legacy data structures (legacy system integration) • This implementation is often implemented as an interpreter.

Build an IDE • DSL-based Development often includes a wealth of input, intermediate and output artifacts. • Users (application developers) can easily be confused by these many artifacts. • To improve acceptance of the approach, you should consider building an IDE for application developers which • Hides unnecessary intermediate artifacts • Provides editors/support for “programming” with the DSL • Integrates all the validation, generation, build, etc. • Interactively connects and synchronizes the various models used • Eclipse, for example, is a good toolkit for building these kinds of IDEs.

Build an IDE: Example

Adopting DSL-based Development

Cascading DSLs • You would typically start with architecture-centric DSLs where the abstractions of the DSL correspond to the core concepts of the technical platform. • This automates many aspects of the technical development/infrastructure; • Results in a wide platform/infrastructure • Many projects can be handled with the infrastructure • In later phases, functional DSLs infrastructures will be built on this technical one, resulting in cascaded DSLs.

Cascading DSLs II

Cascading DSLs III – M2M Transformations

Cascading DSLs IV – M2M Transformations II

Markus Völter voelter@acm voelter.de