Abstraction As an Organizing Principle for Agile Enterprise Architectures

1. Abstraction As an Organizing Principle for Agile Enterprise Architectures Chuck Mosher Director, Federal Architecture & Solutions 3rd SOA for E-Government Conference May 2, 2007

2. Agility is Key • Agility is the right precept • Change is a constant • Technologies • Requirements • Standards • Models • Processes • ???

3. The Business Inflexibility Trap Inflexibility is the Mother of All Business Problems If you’re flexible enough, you can solve all the other problems Ron Schmelzer, ZapThink

4. Multiple Dimensions to Address • Governance • Leadership and oversight • Policy • Rules for sharing, regulations, authority • Technology • Systems, protocols, standards, security • Culture • Potentially the most challenging issue • Economics • Incenting the correct behavior

5. Focus of this talk • Governance • Leadership and oversight • Policy • Rules for sharing, regulations, authority • Technology • Systems, protocols, standards, security • Culture • Potentially the most challenging issue • Economics • Incenting the correct behavior

6. Progressive Evolution of Solutions • IT was supposed to help! And it has: • Assembly Language • Programming Languages, Procedural • Relational Databases • Programming Languages, OO • Messaging Frameworks • CORBA, IDL, RMI, UML • Java, J2EE, .NET, Application Servers • HTTP, XML, SOA, ESB, BPEL Magic Bullets, All!

7. Magic Bullets, Current Crop • XML • Service Oriented Architecture (SOA) • Enterprise Service Bus (ESB) • Standardized Data Interchange Models The Evolution of Computer Science has been as a result of more and more sophisticated thinking about appropriate levels of abstraction in how to solve problems in information management, and the technologies that result.

8. XML • XML provides a transport-independent abstraction to interchange data between applications and across the enterprise • Self-describing data • Human and machine-readable • Easily extendable • No more proprietary data formats It does not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

9. SOA • SOA provides an effective abstraction to encapsulate business functionality & provide interoperability across the enterprise • Decouples data from applications & user views • Standardized interfaces improves info sharing • Leverage existing investments It does not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

10. Enterprise Service Buses • ESBs define an abstraction for machine and OS-independent messaging • Pub/Sub, Point-to-Point, Content-based routing • Highly fault/failure-tolerant, assured delivery • Extensive logging/auditing capability • Complex event processing They do not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

11. Standardized Data Models • Enterprise-wide or COI-specific data abstraction definitions • Simplifies information sharing between communities • Many examples: GJXDM, NIEM, C2IEDM, JC3IEDM • Concept of COI very powerful They do not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

12. Information Sharing – THE Problem • “It’s all about the data” – Hon. John Grimes, ASD(NII)/CIO • “It’s all about the data” – Mr. Dave Wennergren, Deputy CIO, DoD • “The biggest challenge is the data” – BGN Gary Salisbury, US European Command (ret) • “It’s not the chain of command, it’s the chain of information” – Gen Cartwright, US STRATCOM • The central concern of NCES • Thesis: If we can solve the data problem with agility, adaptability, and cost-effectively, we have solved the most difficult (technical) problem

13. We Need One More Abstraction! • A virtual data repository layer focused on turning the data we have into the data we want • Data as an abstract service • Accessible in multiple formats (XML, relational) • An abstraction that provides access to diverse distributed resource stores • Don’t move the data, virtualize it • Middleware handles federation, structural, and semantic reconciliation to a reusable data abstraction • Create agility precisely where it is needed, at the data layer

14. Information Virtualization Information Virtualization Layer Information Consumers Abstract Data Tier Information Resources

15. Information Virtualization Layers Information Virtualization Unification of different system concepts Unified Semantic Layer Single-query access to heterogeneous systems Data Federation Layer Data Access/Connectivity Layer Uniform, standardized access to any system Enterprise Data Sources

16. Data Service Layer in SOA Client Process & Applications App App App App App App Business Process Services Business Services Message Services (ESB) Data Service Data Service Data Service Data Service Data Service Data Services Layer Data Sources

17. Creating a Virtual Data Layer SOAP ODBC JDBC <sale/> <value/> </ sale > <WSDL> (contract) <WSDL> (contract) <WSDL> (contract) Designing data services Exposed Data Services Reusable, Integrated Data Objects Enterprise Information Sources (EIS) Information Consumers Web Services,Business Processes services warehouses EAI, Data warehouses databases Packaged Apps Intelligence spreadsheets xml Custom Apps geo-spatial Reporting, Analytics CRM, Logistics rich media …

18. Multiple layers of abstraction

19. Not Just a Notional Abstraction • Multiple vendors provide a data management layer that efficiently and reliably enables federated access to a wide range of heterogeneous data sources • Sophisticated security controls • Standards-based • Model-driven development • Model, not code, your way to agility • Rich metadata management • Data abstraction layer can concretize enterprise or COI-specific data models

20. MetaMatrix Product Architecture JDBC SOAP ODBC JMS Access Models Integrated Security Views XML Docs Services <a> Users … <b> in proc out MetaMatrix Integration Server </b> Virtual Data Bases Integration Server </a> VDB VDB Roles Query Processor Optimizer Processor Entitle ments Information Consumers MetaMatrix Server MetaMatrix Designer - Design and deploy data services MetaMatrix Connector Framework Packaged Connectors Web Svc XML RDBMS MetaMatrix Catalog CICS VSAM Siebel, SAP Oracle Apps

21. Data Abstractions and Data Models Location_ID Location_Type bldg_type bldg_id Depot_Number SITENUM Facility_ID Business Intelligence Applications Search Applications Web Services ODBC/JDBC JDBC SOAP Application views of information: • Relational, XML XML Document <a> … <b> </b> </a> T T T C2, Logistics, Intelligence, … Logical Data Model: • Agency or COI-specific • Rationalize, harmonize, mediate Semantic Data Services! T T T Authoritative Sources: • Mapped to logical Multiple Internal/External Information Sources

25. Metadata management & federation Metadata Search & Reporting Impact Analysis Metadata-aware Applications Virtual Metadata MetaMatrix Designer MetaMatrixServer MetaMatrix Repository Establish data dictionary Define integration paths Documentation Version Control XML Models RDBMS MetadataRepositories ModelingTools Metadata Sources

26. Impact Analysis

27. Information Exchange Topology Master Data Person / Facility / Vehicle Search Engine Index / Metadata Catalog Ontology Mgmt / Reasoning Mediation XSLT, Multi-source Enterprise Service Bus State/Local Orchestration Encryption High Availability Security/Authentication Operations Management Error / Exception Management Data Access Services • SQL, Web Service/XML • Staged Data (optional) MetaMatrix Dimension MetaMatrix Enterprise Stage SOA App’s Local Commands Federal Agencies

28. US Defense Logistics Solution • Global Combat Support System (GCSS) • Mission: supply the war-fighter with access to accurate and timely logistics information • Focused Logistics • Fusion of information technologies to enable forces of the future to be more mobile and versatile • Provides the joint war fighter with a single capability to manage and monitor units, personnel, and equipment • Deployed at 23 sites around the world • Networked environment allows DoD users to access shared data & applications, regardless of location

29. DISA GCSS-J IDE = Abstract Data Layer

30. GCSS Modeling Approach JOPES 4.0 Virtual Query Layer (VQL) (Exposed Views) Public Data Material_VQL Plans_VQL Facilities_VQL Virtual Mid Layer (VML) Facilities_VML Material_VML Private Data and Metadata Virtual Base Layer (VBL) CSDS_VBL Physical Layer (PL) CSDS_PL JOPESClassic GDSS GTN CSDS DMDC FLIS IDE/AV GSORTS CFDB NGA

31. Intelligence Agency: Information Portal <X> <X> <X> <X> </X> </X> Client(s) Data Host(s) Information Integration Host(s) Application Server Host(s) Browser Managed Data Presentation Layer Query Page Control ResultsPage Control Form Page Control MetaMatrix Query Engine Managed Data … Managed Data Browser MetaBase Mission Data Query Service (Business Layer) Mission Data Portal: • Entities • Components • Relationships Metadata: • Name • InformationContext • Usage • Description • Display Name • Default Value • Attribute Units • Presentation Type • Sort Order • Visible Data dictionary Integration paths Portal metadata Mission Data External Feeds Bottom-up (harmonization) Top-down (mapping) Org, Person, Image, Location T

32. Intelligence Architecture: MetaMatrix & ESB HUMINT DataService(s) ISR Data Listener Service Event Assessment Service Alert Subscription Service BC Gateway Service SIGINT Gateway Service Metadata Discovery Service Weather Effect Service Metadata Publishing Service MetaMatrix DCGS-A Services Network E-Space Services IMETS (IWEDA) MIP Blue Force Tracking HDWS (CHAMS) Handheld Infrastructure Services Google Earth Rich Client Enterprise Service Bus NCES Service Discovery Google Earth Rich Client Weather Effects Map / Coverage EW Data Force Tracking CapeClear Workflow Engine CapeClear Transformation Engine Async “Callbacks” Metadata Searches Alert Criteria Alerts / Events Metadata CapeClear Service Management Filters Metadata Catalog

33. Long Term Software Cost Comparison

34. Summary • Leverage the power of abstraction to simplify, clarify, and realize your EA vision • Abstraction is how you design for change • Be very aware of the level you are working at • Leverage the increasing power that COTS and open source tools can provide • The rubber meets the road at the data layer • That you have, not what you want • Abstract it, don’t move it! • Model, not code your way to agility! -RS • Metadata management is a must-have • For repositories too

35. Unacceptable Risk???

36. Not if you know the lay of the land