350 likes | 464 Views
Achieving Information Sharing in Federal Agencies via Rapid Data Services Enablement and SOA. MetaMatrix and the SOA CoP Demo. Chuck Mosher & Tony Vachino MetaMatrix October 31, 2006. Agenda. Data Services Rationale & Best Practices MetaMatrix Products & Capabilities
E N D
Achieving Information Sharing in Federal Agencies via Rapid Data Services Enablement and SOA MetaMatrix and the SOA CoP Demo Chuck Mosher & Tony Vachino MetaMatrix October 31, 2006
Agenda • Data Services Rationale & Best Practices • MetaMatrix Products & Capabilities • Achieving Information Sharing • Service Enabling Data Assets • Resolving Semantics • Demo • Summary/Q & A
Handling The Data Challenge Getting the right information to the right person at the right time requires: • Resolving data semantic and structural mismatches • SOA-enabling legacy data systems (i.e., Net Centricity) • Mapping data sources to vocabularies like XDM, NIEM, C2IEDM, HL7, … • Handling multi-source requests (data aggregation, mediation, fusion, federation) • Minimizing development and maintenance cost of custom code by metadata-based MDA MetaMatrix can Help
Information Challenges Communities of Interest Agency Challenges • 100’s/1000’s of data sources • 100’s/1000’s of applications • Multiple access points/modes for apps • Understanding relationships/semantics • Data consistency • Data reuse – bridging data silos • Support for Web Services & SQL • Control & manageability, compliance • Security & auditing ? Information Resources Program Challenges • Multiple sources • Different interfaces/drivers • Different physical structures • Different semantics • Single interface to data desired • Real-time access to data • Performance • Maintainability as data changes • Maintainability as apps change Mission Challenges • Time-to-deploy • Agility - Responsiveness to change • Automation – Reduce cost of new development and operations • ROI of enterprise information
Information Virtualization Information Virtualization Layer Unification of different concepts across systems Unified Semantic Layer Single-query access to heterogeneous systems Data Federation Layer Data Access/Connectivity Layer Uniform, standardized access to any system Enterprise Data Sources
Data Interoperability Through Information Services Dynamically Created COIs Weapon Systems Sensors Finance • Information on demand • High performance • Minimal replication • Manageable • Secure Information Services Personnel Logistics Intel Etc. MetaMatrix Information Integration Platform C2 Persons ODS Parts Personnel Documents Units Images Sensors Facilities
What is a Data Service? • Decouple data sources from application • Data implementation shielded from application • Semantic/Format Mediation • Standard vocabulary • Single access point • Web Service/XML • SQL • Federation • Single source or multi-source • Scalability • Security, performance XML/SOAP SQL Bridge the Gap Data Service SQL SQL API Call Master Data Agency Application Operational Data Store
Data Services: Designed for Agility • Data Services Best Practices • Provide transparency across all sources • Define known relationships today and accommodate future relationships • Support independence of mission systems • Support ownership of operational data sources at the source • Provide accelerated mechanisms for integrating new sources • Support existing security policy and add degrees of security • The value of a managed metadata abstraction layer • "Future Proofing" (future standards, exchange models, platforms) • Limited skill set requirements • Fixed long term costs for integration middleware • Building consensus • Assure data owners they will continue to have control, and … • Vocabulary of existing production systems will not be impacted • Offer an option where legacy data migration is not 'required' 1st
Data Services Approaches <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> <X> </X> </X> </X> </X> </X> </X> </X> </X> </X> </X> </X> </X> </X> </X> Data Services for Multiple Purposes: • Simplified access to value-added (tagged) data in real-time • Value-added (tagged) data materialized & staged • Phased-in migration from legacy to new • Managed archiving via classification, retention tags • Enhanced search via consistent content tags Agile Information Services Model-Driven Integration Layer Logical Data Model Logical Data Model T Org, Person, Image, Location T Organization, Customer, Imagery, Location Materialized Logical Model Materialized Logical Model Data, Content Sources Data, Content Sources Enriched Data/Content Store
Information Exchange Topology Master Data Person / Facility / Vehicle Search Engine Index / Metadata Catalog Ontology Mgmt / Reasoning Mediation XSLT, Multi-source Enterprise Service Bus / Intranet / Extranet State/Local Orchestration Encryption High Availability Security/Authentication Operations Management Error / Exception Management Data Access Services • SQL, Web Service/XML • Staged Data (optional) Distributed Data Services Enterprise Data Services Stage SOA App’s Land/Sea Federal Agencies
Agenda • Data Services Rationale & Best Practices • MetaMatrix Products & Capabilities • Achieving Information Sharing • Service Enabling Data Assets • Resolving Semantics • Demo • Summary/Q & A
MetaMatrix I.P. MetaMatrix has 2 distinct innovations that work in concert to yield significant business benefits: Model-based Extensible Sharable, reusable Standards-based Information Modeling Cost-based optimizer Read/write/transactions Uniform API, any source Battle-tested/hardened Federated Querying
MetaMatrix Approach to Data Services SOAP ODBC JDBC <sale/> <value/> </ sale > <WSDL> (contract) <WSDL> (contract) <WSDL> (contract) Designing data services Exposed Data Services Reusable, Integrated Data Objects Enterprise Information Sources (EIS) Information Consumers Web Services,Business Processes services warehouses EAI, Data warehouses databases Logistics Packaged Apps spreadsheets xml Custom Apps geo-spatial Reporting, Analytics Intelligence rich media …
Data Service Abstraction Layers • Transformations from one or more sources • Transformations defined with: • Joins/unions • Criteria • Functions • Elements mapped to dictionary • Business definitions captured
MetaMatrix Products JDBC SOAP ODBC JMS Access Models Integrated Security Views XML Docs Services <a> Users … <b> in proc out MetaMatrix Integration Server </b> Virtual Data Bases Integration Server </a> VDB VDB Roles Query Processor Optimizer Processor Entitle ments Information Consumers MetaMatrix Server MetaMatrix Designer - Design and deploy data services MetaMatrix Connector Framework Packaged Connectors Web Svc XML RDBMS MetaMatrix Catalog CICS VSAM Siebel, SAP Oracle Apps
Secure Access – Accredited Username/Password Logon • Connector connects with same ID for all queries • Optional: Integrated with existing authentication system MetaMatrix Data Source Client App Connector username password username password Membership Provider authenticates Trusted Payload Logon: • Connector uses different credentials per connection, per query • Optional: Integrated with existing authentication system source- specific MetaMatrix Data Source Client App Connector trusted payload trusted payload logon info trusted payload payload payload Optionally accesses source-specific information Authentication Service Membership Provider authenticates, generates payload authenticates, optionally modifies payload
Managing Data Service Metadata Rel Process X XML Rel Process Y XML XML XML Service A Service B Classification Schemes KeyWords B Taxonomy A Relational Transformations XML Datatypes MetaMatrix Catalog Generic Typed Relationships MetaMatrix Designer Domain [UML/ER] Models & Files [versioned] Web Services [WSDL] Processes [BPM/BPEL] Search Index Web Reporting WSDL Application/ Configuration Ontologies [OWL/RDF] Taxonomies
MetaMatrix Product Lines MetaMatrix Enterprise • Web services & SQL • Modeling enterprise data • Scalable deployment server • Metadata management • Application/legacy connectors MetaMatrix Enterprise Enterprise MetaMatrix Dimension • Web service-enablement of data sources • Expose business views as XML • Lightweight modeling – rapid integration • Standard WAR-based deployment MetaMatrix Dimension Project, Node MetaMatrix Query • Embeddable Java component • Federated query engine • Query optimization • Standard JDBC to all sources • Standard SQL to all sources MetaMatrix Query ISV / Project
Agenda • Data Services Rationale & Best Practices • MetaMatrix Products & Capabilities • Achieving Information Sharing • Service Enabling Data Assets • Resolving Semantics • Demo • Summary/Q & A
Mediation: XML From Non-XML Sources «Relational» «XML» <person> <addresses> … </addresses> <accounts> <accountID=…> … </accountID> </accounts> </person> «Application» Target: Fixed (potentially complex) XML Schema Need: Data complying to Schema Source: Data Sources containing Information to integrate «Text File» T MetaMatrix: Mapping from Data to XML
Map Data Sources to XML & Deploy MetaMatrix Designer – for XML-centric Data Services Model XML Docs, Schemas Build XML Doc. models from XML Schemas Map XML Doc. models to other data models Enable data access via XML
Dimension – Choose your approach Data Sources Source Models Business Views Web Service Operations Web Server Import Map Model Deploy <XML> <XML> to to as WAR <XML> XSD WSDL • Rapid design & deployment of Web Services • Expose integrated data as XML-based business views • Deployment of Web Services as standard Web apps • Runtime execution optimized through use of MetaMatrix Query Engine Dimension Models Start Here? Start Here?
Agenda • Data Services Rationale & Best Practices • MetaMatrix Products & Capabilities • Achieving Information Sharing • Service Enabling Data Assets • Resolving Semantics • Demo • Summary/Q & A
COI Data Dictionary Location_ID Location_Type bldg_type bldg_id Depot_Number SITENUM Facility_ID Business Intelligence Applications Search Applications Web Services ODBC/JDBC JDBC SOAP Application views of information: • Relational, XML XML Document <a> … <b> </b> </a> T T T C2, Logistics, Intelligence, … Logical Data Model: • Agency or COI-specific • Rationalize, harmonize, mediate T T T Authoritative Sources: • Mapped to logical Multiple Internal/External Information Sources
Semantic Matching - example Ontology “Sex” semantically related to “Gender” Semantic Data Services • key component of information sharing and interoperability programs • automated semantic mapping to aid domain experts in quickly reconciling disparate schemas and vocabularies • more rapid deployment of a mediation solution MatchIt • an extensible ontology-driven tool • variety of algorithms for determining semantic equivalence • discovers similarities between elements of heterogeneous data, automatically exposing potential semantic matches. • matches elements of data sources to target schemas of Data Services, such as TWPDES, GJXDM, NIEM, C2IEDM, HL7 Matched (Confidence of 90%) Gender ID Semantic Data Services Person Sex Code FBI CBP NYC NY NJ Data Sources
Automated Term Discovery (Interpret) All the available definitions found in the MatchIT knowledge-base Results of the automated tokenization All the usage instances where each term was used in any of the sources A comprehensive list of terms automatically discovered across all sources
Contextualize (Interpret) ArticleAmount Amount Article Synonym Creation Sum Type-of Assets Automated term tokenization Automated semantic linking using the default knowledge-base contained within MatchIT
Semantic Matching (Mediate) • With relationships pre-established within the knowledge-base… • Identify the Target and the Source(s) and run the match. ArticleAmount Automatically linked by a specific % distance ProductShares
Facilitate Decision Making (Mediate) Target element for matching Automatically calculated semantic distance between terms Helps facilitate rapid decision making Source candidate for matching
Support Multiple Enterprise Semantic Models J-1 Manpower / Personnel J-7 Operational Plans J-4 Logistics (GCSS) J-8 Force Structure J-5 Plans & Policy J-2 Intelligence J-3 Operations J-6 C4CS Business Intelligence Applications Portal Applications Web Services ODBC/JDBC JDBC SOAP Enterprise-wide or COI-driven Data Models • Rationalization • Harmonization • Data Catalogs (DDMS) T T T Data Sources - Authoritative • Redundant • Overlapping Multiple Internal/External Information Sources
Agenda • Data Services Rationale & Best Practices • MetaMatrix Products & Capabilities • Achieving Information Sharing • Service Enabling Data Assets • Resolving Semantics • Demo • Summary/Q & A
Agenda • Data Services Rationale & Best Practices • MetaMatrix Products & Capabilities • Achieving Information Sharing • Service Enabling Data Assets • Resolving Semantics • Demo • Summary/Q & A
MetaMatrix – Quick Facts • Middle-ware, model-driven, data management • DoD proven (DISA, NSA, TRANSCOM, etc.) • Version 5 – Mature product which is still unique and ahead of the competition • NIAP certified and NSA-credentialed • Can handle the enterprise (or COI) perspective as well as the bottom-up perspective (data service enablement of legacy systems) • Can rapidly implement data integration strategies
Major US Federal Government Customers • NSA - Multiple Programs (NES Base-lined) • In-Q-Tel/CIA • TRANSCOM – Command Metadata Management System • Air Force - Command and Control Center • DISA - Global Combat Support Systems (GCSS) • DISA – Anti Drug Network (ADNET) • DLA – Integrated Data Environment (IDE) • Mitre – Air Force ESC/DoD DDMS work • UK – NSA Equivalent, CJIT
MetaMatrix Value Proposition Highly cost-effective COTS tool for rapid enterprise information integration and exchange • On-demand information • Real time data integration • Information sharing between business units • Enabling SOA in an evolving world • Consume and produce Web services • And still provide full support for ODBC, JDBC, and legacy • Federation of disparate information • Rationalized to controlled vocabularies • Relational + XML + Web Services + Enterprise Apps + Legacy • Faster time to market • Integrated information in days, weeks • Tight coupling of design & implementation phases • Leveraging the skill-set of the data architects for integration • Costs across application lifecycle reduced • Model-driven abstraction layer eases development/maintenance • Better management of data assets across the enterprise