Report from the trenches

1. Report from the trenches Building Statoil’s TOPS portfolio Einar Landre, 14-15 September 2005

2. Agenda • Statoil Wet Supply Chain and the TOPS portfolio • Architecture, tools & technology • Experiences & lessons learned • Development strategy - Domain Driven Design • Inversion of control & Dependency Injection • Data access – Practical use of transparent persistence • Business processes, automation and integration • User interface development – Practical use of Swing • Summary & Conclusion

3. Statoil Wet Supply Chain • Facts: • Produces 1.000.000 crude oil / day • Third largest crude oil seller in the world • Largest retailer of oil products in Scandinavia • Market two thirds of all Norwegian gas to European customers • The Wet Supply Chain is responsible for • Sales and transport of Crude oil • Sales and transport of refined oil products & LNG • Terminal operations (Mongstad, Callundborg, Kårstø ….) • The Wet Supply Chain is supported by a wide range of applications • Custom made and COTS

4. External External External External end Supplier customer Customer Supplier user Statoil Wet Supply Chain Marketing and Trading and transportation Trading and transportation Refining and Processing Find and produce distribution Exploration & Production O&S Crude/NGL O&S Products/NGL Refineries Retail O&S Processing

5. O&S Master Plan Oil • Objective • Program aimed to replace a set of legacy systems that supports the Wet Supply Chain within end of 2008 • New system portfolio called TOPS • Forces • Existing systems does not support future business needs • Existing systems built on outdated technology • New solution will be based on a combination of • COTS (packaged software) • Custom made software

6. TOPS – Trading and Operational System • TOPS is built from five functional subsystems: • Shipment & Allocation • Physical Deal & Delivery • Derivates & Risk • Accounting Interface (SAP) • Document Management

7. Architecture • SJEF • Defines naming conventions, building blocks and design concepts • Supports both build and buy sourcing strategies • Driving requirements • Role based access • From data processing to process automation (event processing) • Support integration across organisational boundaries • Decision making, autonomy and collaboration • Key tools and technologies

8. Large scale structures • Information System • Autonomous unit • Defines a context for data • Custom made or commercial package • User Context (Workbench) • Provide users role based access to services • System Integration • Data distribution • Asynchronous communication

9. Main Building Blocks • User Context • Workbench • Activity • Services • Information System • Tasks • Domain Model • Agents • Events • Integration Concepts • Message Management • Events and event processing • Distributed data model • Services

10. Clear responsibilities through layering • Presentation Layer • User interaction • Activities and Services • Process Layer • Manage and orchestrate business process flows • Events, Tasks and Agents • Domain Layer • Make business concepts explicit • Business rules • Repositories, Aggregates, Factories, Entities, Value Objects and Services • Infrastructure • Provision of infrastructure services to the other layers: GUI Components, Security, Data Storage and Transaction Support

11. Java Swing IBM WebSphere (app server) IBM WebSphere MQ (messaging) MS Biztalk (Integration broker) Spring framework Versant Open Access (JDO) Oracle DB JACK – Agent Platform Home grown & Open Source utilities JEF Session Management JEF Security Log4J JEF Process Support JEF Client Spring Rational Application Developer Clear Case Clear Quest Technology Stack

12. Domain Driven Design • Problem: • Huge and complex business domain • 1000 table databases (existing system) • Solution: • Isolate the domain from infrastructure • Domain Driven Design provide the tools • Lessons learned • Focus on what the software should do • Develop scenarios that defines what takes place • Scenarios drives the model work • Implement the scenarios in code as executable tests • Attach GUI/DB stuff when domain is sufficient distilled

13. Repository Object that provide access to aggregate root entities Manages data storage and retrieval Aggregate Cluster of related objects that are treated as one object with respect to updates Define boundaries in the model Accessible through the root object entity Factory Object responsible for creating other objects. Used to hide the complexity of building object structures Service Provides an interface to “things” that can not be modeled as an entity Entity Object Represents an object identified by its ID. The ID can not change during the lifecycle of the object. Value Object Object that is defined by its type, not its ID. Module Provides a logical namespace for a group of objects. Also known as package Domain Layer Building Blocks

14. Aggregates & Repository classes • Aggregate: • A logical group of objects that are treated as a whole during update. • Product price copied into order line to enforce consistency • Access only through the aggregates root entity • Direct access through repositories • Order Aggregate can navigate into Product Aggregate • Product Aggregate don’t know anything about orders Order Aggregate Product Aggregate

15. Inversion of control / Dependency Injection • Control Freak Object approach: • Object responsible for obtaining resources • Effect: Object dependent on resources. Makes unit testing harder if at all possible • Dependency Injection: • Context responsible for provision of resources • Simplifies testing by introducing mockups when applicable • Object become more re-usable • Key principle in SJEF Control Freak Object Get / locate MyObject Resource Dependency Injection Context Resource Set resource Use MyObject

16. Problem: How to wire a set of components into a working application Revisiting the abstract factory pattern: Provide an interface for creating families of related or dependent objects without specifying their concrete classes Decouples creation of objects from the use of objects Service locator is an implementation of abstract factory But how does the client object obtain resources? Use of factories and locators: RepositoryFactory.getRepository(); Locator.locate(Repository.class) Dependency Injection Pattern MyObject(Repository repos) { /constructor } Coding with service locators Object responsible for obtaining required resource: public class MyObject { MyRepository repository; MyObject() { repository = Locator.locate(MyRepository.class); } } Question: Where do the locator come from? Coding with Dependency Injection Resource provided from the outside: public class MyObject { Repository repos; MyObject(Repository repos) { this.repos = repos; } } Separating configuration from use

17. Wiring with Spring <bean id=“myRepository” class=“com.statoil.myRepository”/> <bean id=“myObject” class=“com.statoil.MyObject> <constructor-arg><ref bean=“myRepository”/></constructor-arg> </bean>

18. Data Access – Transparent Persistence • Facts: • Transparent persistence documented to reduce code with 25-40% • Why JDO: • JDO – The standard for transparent persistence in Java • Supported by multiple vendors (commercial & open source) • Enables migration towards OODBMS • Performance: • Reduced time from 10 minutes to 25 seconds on a complex operation • Replaced 1500 lines of SQL with 100 lines of Java

19. Defined by a simple interface: public interface OrderRepository { public final String DEEP_GROUP = “” Order save(Order o, String fetchGroup, boolean detach); Collection selectMatcing(OrderCriteria criteria, boolean detach); Order reconsitute(OrderKey key String fetchGroup, boolean ,); Collection findAll(String fetchGroup, boolean detach); } JDO Implementation: public Collection findAll(String fetchGroup, boolean detach) { Query q = getPersistenceManager().newQuery(Order.class); Collection col = (Collection)q.execute(); if (detach){ return JDOUtil.detachAsChangeTrackList(getPersistenceManager(), col, fetchGroup); } return col; } public Order save(Order order, String fetchGroup, boolean detach) { PersistenceManager pm = getPersistenceManager(); pm.makePersistent(order); order = (Order)JDOUtil.attachObject(pm, order, fetchGroup); if(detach){ order = (Deal)JDOUtil.detachObject(pm, order, fetchGroup); } return order; } Coding Repositories

20. What about object databases • Facts: • Measured to be 10-50 times faster on complex data than RDBMS • Documented simple to manage than RDBMS • Results from practical tests: • Used 2 days to move data from Oracle to Versant VDS 7 • Same object model using a JDO interface • Improved performance • Simpler to work with – No mapping issues • Conclusion: • Will have a place in our infrastructure the coming years • Improved performance (x3?)

21. Automating and Supporting Business Processes Goal: Optimize storage level • Plans: • Out of stock • No credit • Built from three construct • Process controller (agent) • Goals • Plans • Tasks (work to be done) • Events (things that happen) • Domain objects • Goal defines what the process should try to achieve • Plans defines what options exists when an event occur in a specific situation Agent Storage Manager Monitor state Order Object Events Tasks Cancel Order Reallocate product Fire events Ship Order Prepare & Ship Invoice Order Send Invoice

22. More on process support • Events & Tasks are real objects • Expresses domain specific concepts • Two categories of agents: • Simple reactive: Message Driven Beans (In production) • Goal oriented: Requires dedicated software agent platform • Have built demonstrator using software agent platform • Easy to learn • High productivity (20-40% reduced development cost) • Extends the Java language with agent specific terms • JACK from AOS • Plan: • Increase the use of goal oriented agents to solve complex event processing (integration) and optimization problems

23. Integration and distribution of data • Business process must arrange data internally to support its needs • Information Systems provides a technical context for processes • Information Systems stores data locally • Performance • Different domain models, same data. • Information Systems only allowed to change its own data. • Information Systems exchange data using events (messages) • Based on JMS / BizTalk for transport

24. User Interface Development – Java Swing • Challenges: • Competency & common understanding of client design and construction • Hard to find the balance (under/over engineering) • Few frameworks at an appropriate level of abstraction • Standalone components but not how to assemble an application • MVC pattern too hard to use efficiently • Lack of practical guidelines in a Swing environment • Lack of frameworks • Data binding • Experience: • Harder to build than initially anticipated • Developers wish to refactor when completed • Spring can be used at client. Simplifies the assembly and improves development. • Good client design depends on OO experience • Lack of patterns and best practices

25. Summary • Architecture works well • Provides us a technical vocabulary • Domain driven design has proved its value • Patterns and practice • Spring framework saved us • Separate configuration from use • Enforces Inversion of Control • Use of JDO has been successful • Improved performance • Reduced number of lines of code • Integration and process control • Requires attention • How to best combine services and events must be worked out • Client / User interface development harder than first anticipated • Revaluates tools and practice know • Netbeans looks interesting • Spring have improved the situation