Authority and Data

1. Authority and Data Dr. Scott Rennersar@mitre.org 26 August 2008 DRAFT For discussion only For Limited External Release

2. Authority and Data • What could the term “authoritative data” mean in an enterprise like the DoD? • There are four distinct meanings in use today • Agreed data vocabulary • Appropriate source system • Authorized data producer • Approved data for decision-making • All four concepts are necessary in a very large enterprise • To avoid confusion, we need uniform terms of reference • Each concept clearly defined • Distinct terms for distinct concepts • This briefing presents those concepts and proposes terms

3. Producers Consumers information Preliminary Terms of Reference

4. Consumers Producers Preliminary Terms of Reference information data

5. Preliminary Terms of Reference Information is produced by people operating… desktop computers handheld devices weapon systems sensor devices

6. Preliminary Terms of Reference …with some sort of application posting information into a shared information space application shared information space application application application

7. Preliminary Terms of Reference … pulled or pushed to other applications for the benefit of other people operating various things application application shared information space application application application application application application

8. Preliminary Terms of Reference controlled by mission organizations operated by mission organizations built by acquisition organizations built by acquisition organizations operated by mission organizations application application shared information space application application application application Data Producers / Information Owners Data Sources Infospace Consuming Applications Consumers / Operators / Users

9. Preliminary Terms of Reference Common Vocabulary: All things describing semantics and syntax Data models, elements, taxonomies, ontologies, … application application Data Producers / Information Owners Data Sources Infospace Consuming Applications Consumers / Operators / Users

10. Key Characteristics of the Enterprise What could the term “authoritative data” mean in an enterprise like the DoD? • No effective central authority • Over producers, consumers, builders, and funding • Instead, we have an “enterprise of enterprises” • Common purpose • Limited autonomy • Often no single sovereign over all participants

11. UN Germany Commercial France UK US Australia China Syria NGOs Justice EPA DoD State Treasury Army Navy View in slideshow mode What Is The Enterprise? This enterprise is formed from several sub-enterprises, all having limited autonomy, and some purpose in common Air Force AETC AMC ACC AFMC AFSOC

12. Key Characteristics of the Enterprise • No effective central authority • Over producers, consumers, builders, and funding • Several enterprises, common purpose, limited autonomy • No single sovereign over all participants • Very large enterprise • Many people, large conceptual domain • Some processes and information needs are not stable • Processes vary from highly-structured to ad hoc • Flexibility is often essential • Information needs range from highly structured datato completely unstructured data What could “authoritative data” mean in an enterprise like this?

13. Authoritative = Agreed Data Vocabulary • Information / data sharing won’t happenunless people understand the data in the same way • Establish a semantic match • Cope with representation mismatch • When we write down this common understanding,we call it a “common vocabulary” • Agreement doesn’t have to be voluntary for everyone • Some people can create the agreement • Others, subject to their authority, can be told to adopt it • (There are limits on how far this will work) vocabulary When organizations agree on semantics and representation, we call the result an authoritative data vocabulary

14. Example • Each COI develops its authoritative data vocabulary • Glossaries • Taxonomies • Data models and data elements • Schemas • Ontologies • Whatever they need to record, use, and teachtheir common understanding of data • COI members are expected to understand and use it

15. Authoritative = Appropriate Data Source • A data source is an application (aka a system, or Program of Record) that provides data via services • Usually this data isstored internally data service interface application

16. Authoritative = Appropriate Data Source • A data source is an application that provides data via services • Other applications copy this data for local processing • Today, via web service (XML over HTTP) • Yesterday, in comma-separated files over FTP 4 2 5 6 1 3

17. Authoritative = Appropriate Data Source • The fateful day • Instead of getting data from the original source… • An application obtains data from a copy • Or a copy of a copy 4 1 5 6 7 2 8 9 3

18. Authoritative = Appropriate Data Source • Over time, the result is chaos • Copies of copies, latency, error “corrections” • Applications give different answers for same question • Nobody knows which applications are still needed 16 12 4 10 17 13 15 11 6 1 2 5 7 8 9 3 14

19. Authoritative = Appropriate Data Source When the appropriate source system is established, we call it the authoritative data source ADS

20. Controlled Redundancy Can Be OK • By the way, redundancy isn’t always bad • Controlled redundancy can be useful • If you can keep the sources consistent ADS ADS ADS

21. Example • Each COI has its authoritative data vocabulary • The DoD Metadata Registry (MDR) is the authoritative data source for these vocabularies • Federated registries provide redundant, consistent access runtime design time Federated Registry Federated Registry MDR

22. Authoritative = Authorized Data Producer • Data producers are mission organizationsthat create and maintain data as part of their assigned responsibilities • Systems are not producers • Systems are acquired for and operated by producers • Individual human operators are not producers • Individuals are part of a producer organization

23. Authoritative = Authorized Data Producer • Data producers are mission organizationsthat create and maintain data as part of their assigned responsibilities • Data producers are accountable tothe information owners • Information owners have rights/duties for data • What data will be collected • What level of quality to maintain • Who is allowed to access or update • How these rights are delegated/assigned • This is a mission role, not acquisitionCommander (not program manager) has authority • Delegating and assigning these rights creates an information owner hierarchy

24. Authoritative = Authorized Data Producer • Data producers are mission organizations that create and maintain data as part of their assigned responsibilities • Data producers are accountable to the information owners • Information owners have rights/duties for data,exercised within an information owner hierarchy • Many producers can be authorized/assigned to create and maintain the same kind of data • Each JFACC is authorized to create its own ATO • A single data source may support several data producers • Every JFACC uses the same application to create ATOs When a mission organization has the right & responsibility to produce some kind of data, we call it an authoritative data producer

25. Example • Each COI has its authoritative data vocabulary • The DoD Metadata Registry (MDR) is the authoritative data source for these vocabularies • Each COI is the authoritative data producerfor its own vocabulary MDR COI #1 COI #1

26. Authoritative = Approved For Decisions • Consumers use data to make decisions • Commanders responsible for those decisions have authority to determine the source of that data • They don’t let just anybody provide the data • Rules and procedures specify information needs,sources & producers • Commander as infospace authority establishes the rules • Different commands may have different rules/procedures • Sources and producers approved by onemay not be approved by another When a mission organization directs their deciders to use data from a particular source and producer, we call that the approved data, approved source, and approved producer

27. Example: Air Mobility Command (AMC) • AMC obtains certain aeronautical navigation data from the National Geospatial Agency (NGA) • NGA is the authoritative data producer • Some NGA system is the authoritative data source • AMC provides corrections of this data to NGA • NGA incorporates these, over time; however… • AMC also maintains its own corrected data • AMC is the authoritative producer of these corrections • AMC’s Table Management Distribution System (TMDS) is the authoritative source for this corrected data • AMC commander directs AMC consumers to use TMDS data • TMDS data is the approved data for AMC consumers • TMDS is the approved source Source: Tina Woodbury (AMC), Neil Custer (CSC) – March 2005

28. Summary Four concepts of “authoritative data”, four distinct terms • Authoritative data vocabulary • Semantics and representation established by agreement among “important” organizations • Authoritative data source • Application designated to provide specified data • Authoritative data producer • Organization with right & responsibility to create and maintain specified data • Approved data source and producer • Source/producer directed for use within an organization

29. OK, So What? • Let’s see how these concepts and terms help describeCPM Data Governance • CPM data governance begins with a data problem • Some consumer needs certain data, and can’t get it • Portfolio stakeholders cooperate to solve the problem • Two ways to distinguish these data problems • Does the needed data exist? • Is the solution a matter for system developers only? • This results in four categories of solution

30. HARDER Kinds of Data Problems and Solutions Does the needed data exist? Who makes the change? Yes No Net-enable the existing systems Build or change systems to collect the missing data Only Acquirers Make data sharing agreements Change TTP Users and Operators

31. Kinds of Data Problems and Solutions Infospace authority validates the data “need”, advocates for required change, and directs use of the approved data sources that are created Does the needed data exist? Who makes the change? Yes No Net-enable the existing systems Build or change systems to collect the missing data Only Acquirers Make data sharing agreements Change TTP Users and Operators

32. Kinds of Data Problems and Solutions Acquirers net-enable and/or modify the authoritative sources Does the needed data exist? Who makes the change? Yes No Net-enable the existing systems Build or change systems to collect the missing data Only Acquirers Make data sharing agreements Change TTP Users and Operators

33. Kinds of Data Problems and Solutions Authoritative data producers change their activities to begin producing the needed data Does the needed data exist? Who makes the change? Yes No Net-enable the existing systems Build or change systems to collect the missing data Only Acquirers Make data sharing agreements Change TTP Users and Operators

34. Kinds of Data Problems and Solutions Authoritative producers make sharing agreements with infospace authority Does the needed data exist? Who makes the change? Yes No Net-enable the existing systems Build or change systems to collect the missing data Only Acquirers Make data sharing agreements Change TTP Users and Operators

35. Inferences • People sometimes ask: Who owns the data element?Usually not a good question; however… • It belongs to the COI with authority over the vocabulary • Not to a data producerMany producers can create that kind of data • Not to a data sourceSeveral sources may share that kind of data • Data producer to data source is a many-many relationship • There can sometimes be several sources and producersfor a given kind of information • A single source is desirable, not always feasible • Often desirable to have many producers • Consumers may sometimes choose among several authoritative sources and/or producers

36. Conclusion • All four concepts are necessary in a very large enterprise • Without distinct terms, confusion will ensue • Within the data/info management community,this is our own “shared vocabulary” problem • Doesn’t really matter if we use the proposed terms • Does matter if we understand the conceptsand have distinct terms for each

37. References • Net-Centric Information Management (2005)http://handle.dtic.mil/100.2/ADA464442 • DoD Directive 8320.2, Data Sharing in a Net-Centric Department of Defensehttp://www.dtic.mil/whs/directives/corres/pdf/832002p.pdf • DoD 8320.2-G, Guidance for Implementing Net-Centric Data Sharing (2006)http://www.dtic.mil/whs/directives/corres/html/832002g.htm • My Two Cats Are a Community of Interest (2006)http://dodccrp.org/events/2006_CCRTS/html/papers/157.pdfhttp://www.dodccrp.org/events/2006_CCRTS/html/presentations/157.pdf • Net-Centric Information Sharing (2005) • A Style Guide for Common Vocabularies (2005)https://wwwd.my.af.mil/afknprod/DocView.asp?DocID=701841

38. BACKUP

39. Applications Post & Pull Through Services This is how most people envisioned the shared space in the early days of NCDS; however… controller post service pull service shared information space (physical) Data Source Data Consumer

40. Applications Post & Pull Through Services People now often envision the infospace as a virtual construct, which exists as the collected policy decisions of the infospace authority controller service request/response shared information space (virtual) Data Source Data Consumer