410 likes | 719 Views
CMR. Building a Corporate Metadata Repository. Past Present Future. Sam Highsmith Principal Researcher Statistical Research Division U.S. Census Bureau 301-457-1928 shighsmi@census.gov. Agenda. The Past The Present Requirements for 2000 Questions.
E N D
CMR Building a Corporate Metadata Repository Past Present Future Sam Highsmith Principal Researcher Statistical Research Division U.S. Census Bureau 301-457-1928 shighsmi@census.gov
Agenda • The Past • The Present • Requirements for 2000 • Questions
What is the Corporate Metadata Repository? • Electronic card catalog • Developed in conjunction with work from Sweden, Canada, Australia, and UN/ECE Metadata Workshop • Implements Business Data Model for Surveys and Censuses • Provides a process for registering data set, variables, and documents for access and reuse
Past Milestones • Built a Metadata Repository model • Built a Web enabled Prototype • Formal Agreements with other Bureau users • DADS adopted the Metadata Repository model • Developed a Transition to Production plan • Contract with Oracle to build a CMR • Economic Directorate Pilot Application
Metadata Repository Model • Business Data Model • CASRO, SRD, consultant, Reinvention Lab • Survey and Census business model • Data Element Registry • ISO 11179 • Built BDM and DER, then merged to CMR • Modeled with ERD tool
MilestonesPROTOTYPE • Interface to Multiple Statistical Information Systems - DADS, FERRET, PCDOCS • Table of Contents organization • Search capabilities • Oracle generated database for repository • Completely Web based
Milestones • Formal Memorandum of Understanding with major Bureau directorates
Milestones • DADS adopted the Metadata Repository model • Implemented by Oracle Consulting for AFF • Dissemination tool for 2000 Census data • Accessible from www.census.gov • DADS American Fact Finder is completely metadata driven • !!!!!!!!!!!!
Milestones • Developed a Transition to Production plan • Metadata support staff • metadata providers in directorates • Still waiting…...
Milestones • Built a Metadata Repository model • Built a Web enabled Prototype • Formal Agreements with other Bureau users • DADS adopted the Metadata Repository model • Developed a Transition to Production plan • Contracted with Oracle to Build CMR
Review of CMR Implementation StrategyCurrent Business Process does not include an Integrated Metadata Business Process Census 2000 ACS Demographic Surveys Econ Census Econ Surveys internally developed systems customized commercial systems CASES variety of programming languages individual tool of choice CATI CAPI Mail PAPI OCR ICR CADE CSAQ TDE PFIRS internally developed systems SAS DEVSURV COBOL FORTRAN DECForms StEPS ECON DW individual tool of choice AFF CENSAS FERRET Econ DW CD-ROM Internet Design Collect Process Share
Support One View to Internet Users Support “Virtual Statistical Agency” and sharing data/MD across web sites Support Generation of All Kinds of Products, including Reimbursables Support sophisticated analysis of BOC data (e.g. time series and cross dataset comparisons) Review of CMR Implementation StrategyBenefits of a CMR A C T I V E Corporate Metadata Repository (CMR) ------------------- Statistical MD Geographic MD Technical MD ------------------- Business Rules ------------------- Product Layouts “Faster to Market” through reuse Launch Dissemination Systems
Proposed Technical/Software ArchitectureFour Ways an Application Can Use CMR Metadata Tightly Coupled with CMR 1. Application written against CMR - uses it directly for metadata access and maintenance. 2. Application uses same extensible CMR physical model - can replicate metadata from CMR. (e.g. DADS/AFF,ECON,LEHD) 3. Application communicates with CMR through an API to exchange metadata. (e.g. FERRET) 4. Application communicates with CMR using a standard XML-based metadata interchange. (future - e.g. FEDSTATS,BLS...) Loosely Coupled with CMR
CMR Meta-Model • Meets ISO/IEC 11179 Data Element Registry standards • Meets FGDC Content Standard for Digital Geospatial Metadata (CSDGM) • Can support emerging metadata standards • Extensible • Shareable • COTS where possible
Economic Directorate Pilot • Pilot versus Prototype • Focus on two stages of the information collection life cycle of the ASM • Added functionality to two existing METADATA applications • Dissemination and Refics • Used JAD, RAD, code generation • The “workers” jumped on board! • Found multiple existing metadata applications
Metadata Reuse
Computerized Self Administered Questionnaire Metadata Reuse
Metadata Reuse
ASM IRACS Metadata Reuse
Metadata Reuse
Year 2000 PlanningWhat are we building this year?Hint: A production system!
Metadata Portal Site • Web-enabled • Access to all CMR tools • Promote knowledge management • Secure access (role-based & by individual) • Support Collaboration and Personalization: • BOC Public Access to Tools and Folders • Collaboration areas for BOC Projects and Organizations • Personalization • Customize own CMR Web Site to short-cut access and add personal work • Publish personal work, if desired
V1.0 Configuration Management V1.1 V2.0 DEPARTMENT DEPT • Object-level Versioning • Multi-path Versioning • Object-level Check-in/Check-out EMPLOYEES SALARY EMP SAL Architectural Solutions Management Meta-Model Management Repository Management Metadata Management Security • Network • System • Repository • Metadata • Fine Grain Ownership Admin Tools Bulk Load & Unload Services Metadata Exchange Services Browsing Tools • Publish on Web • Search structured & unstructured MD • Drill-through to actual data • Launch Applications • Synchronize Repositories • Master-Slave Relationships • Same Meta-Model • Meta-Model Bridges • XML-based Exchange • Create, Read, Update, Delete • Archive • Backup/Recovery • Formatted/Delimited ASCII File • XML-based File • Publish in many different standard formats
Object Layer and Open API • Object layer over the CMR physical repository tables • The underlying meta-model could change with minimal impact to the CMR tools and interfaces accessing the object layer. • Provide fine-grain, role-based metadata security for access by CMR tools • Publish an object oriented Open API for developers • Use wizards and generation tools as much as possible • Use meta-model generation tools (preferrable)
Metadata Management The CMR Metadata Management tool will enable the BOC business metadata owners and stewards to maintain their own metadata. • Web-enabled • Meta-model generated application • COTS repository administration • Supports CRUD manipulation of metadata • Supports copying and versioning metadata objects • Supports storage and retrieval of unstructured metadata • Implements business rules, audit trails, and metadata status for simple work flow
Created/Generated using Oracle Designer Hand coded Created/Generated using Rational Rose Oracle Designer Generated HTML/JS Applications Server Tier Deployment Middle Tier Deployment Client Tier Deployment Logical Models Physical Models Functional Requirements Use Cases PL/SQL generating HTML & JS Application Code Web Modules OAS Environment w/ PL/SQL Cartridge & HTTP Listeners Net8 Web Browser HTML Application HTTP Net8 TAPI (PL/SQL) UML Object Model Server Model View Layer CMR Repository
Metadata Browsing and Reporting The CMR Metadata Browsing and Reporting tools will provide read-only metadata searching, hierarchical browsing, and advanced analysis and reporting of the CMR metadata. • Web-enabled • Meta-model generated application • Supports textual and structured metadata searches • Supports nested searches • Supports hierarchical, tree-based browsing (drill up/down) • Supports search and browsing of Tier 1 documents (accessible in the CMR repository or on the Census Intranet/Internet)
Metadata Open Interchange The CMR Metadata Open Interchange provides an open, standards-based method for interchanging metadata between the CMR and external systems. • Use an open standard technology (such as XML) to interchange metadata with external systems • Support bulk metadata object level refreshes • Support metadata validation and correction in staging area prior to final load • Enforce role-based, object level security • Support web-enabled log of input/output activity • Web-enable UI as much as possible
CMR <-> base SAS CMR and SAS metadata interchange tool will enable the metadata owner to perform a bi-directional metadata exchange, and will establish a common mechanism of loading metadata into SAS system and pushing it back to CMR repository. • Web-enabled UI • Output metadata for a statistical dataset for use by base SAS • Input metadata for a statistical dataset from base SAS • Support metadata validation and correction in staging area prior to final load • Enforce role-based, object level security • Support web-enabled log of input/output activity