Mandates for Data Transparency in 113th Congress: DataCoalition

1. Mandates for Data Transparency in 113th Congress: DataCoalition.org Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community http://semanticommunity.info/ AOL Government Blogger http://gov.aol.com/bloggers/brand-niemann/ January 4, 2013 http://semanticommunity.info/DataTransparencyCoalition.org

2. Semantic Community • Our Mantra is: Data Science Precedes the Use of SOA, Cloud, and Semantic Technologies! We use data science to help marketing and business development efforts. • Our Mission is like Googles: Organize the world’s information and make it universally accessible and useful. • Our Method is like Be Informed 4: Architectural Diagrams and Questions and Answers are not enough, you need Dynamic Case Management! • Our Sound Byte: It is not just where you put your data (cloud), but how you put it there! • Our Work: Semantically enhancing your data and writing data science stories about it.

3. Mission Statement • 1. Personal: • Senior Data Scientist at the US EPA: • Completed Data Science Academic Training and Many EPA Data Products • Detail to Data.gov: • Built Data.gov in An Information Platform • 2. Putting Data To Work: • Data Journalist for Federal Computer Week and AOL Government: • Published Many Data Science Products and Built Own Data Journalism Handbook • Data as a First Class Citizen: Data Science and Journalism for Analytic Standards and Audit of Open Data Sites: • Working with CKAN, DoD, IC, NCOIC, NIST, OASIS, OMG, OSTP, W3C, etc. • 3. The Emergence of Data Science: • Built a Data Science Team for the Government Community: • “Killer Semantic Web Application” (Semantic MedLine on the new Cray Graph Computer) for the Federal Big Data Senior Steering Group • Challenges and Contests Using the Best High Quality Data Sets: • Heritage Provider Network Health Prize, Health Data Initiative Forums, TedMed, Department of Commerce App Challenge, etc.

4. Invitation • Your conversation with Cory Casanave over NIEM and StratML caught my eye. Our Coalition is preparing a legislative agenda on data standardization for the 113th Congress. • Chairman Issa of the Oversight Committee - for whom I wrote the DATA Act when I served as counsel to the committee - will be reintroducing the DATA Act and is interested in other standards-related legislative mandates as well. • I'd like to get your advice on how Congress could push the executive branch toward publishing more actionable data. I Legislative mandates are a powerful but very blunt tool.

5. My Response • This is what I recommended to Congressional Staff last September: • http://semanticommunity.info/AOL_Government/BIG_DATA_at_the_Hill#Story • I am working on several briefings in January and February: • http://semanticommunity.info/Emerging_Technology_SIG_Big_Data_Committee/Government_Challenges_With_Big_Data

6. DTC Response • Thanks very much for sharing these resources. I noticed that your three recommendations to Congress didn't include passing mandates for greater use of nonproprietary standards in publishing spending or regulatory data compilations that are currently one-star or zero stars. I'm eager to talk through whether some of the bills that our Coalition will be pursuing in the next Congress are consistent with your recommendations - and, if not, whether they could be made so.

7. My Response • You are welcome and excellent point. Having worked for a regulatory agency (US EPA) for 30+ years and with the new spending data recently I thought we had gotten to open standards for publishing the data and on to the problem of data quality and completeness. See for example: • http://semanticommunity.info/AOL_Government/Recovery.gov-First_Show_Us_All_the_Missing_Data • I am all for helpful legislation that focuses on the value of data analytics to government performance and accountability, more for organizational changes that foster “data work” instead of “IT projects” (e.g. Data.gov, NIEM, etc.), and government use of data scientists and data science like big data companies do to be successful. See for example: • http://gov.aol.com/2012/10/11/what-the-white-house-learned-from-linkedin-and-the-use-of-big-da/

8. DTC Response • Great! You are absolutely right that Recovery.gov is the best use case of standardized federal spending data. But it only covers stimulus spending and plus the whole platform and standard is about to be discarded - and not replaced by anything - unless we are able to pass some version of the DATA Act.

9. My Response • My work with open government data on the platform I use is not going away so we could audit every agency’s financial data like my latest example for the Bureau of Public Debt and rate them for Congress: • http://semanticommunity.info/AOL_Government/Big_Data_Part_II • I could do this: Data Transparency in Action • The technology companies joining the Data Transparency Coalition are eager to design software packages and platforms that analyze federal data in new, powerful ways. Our members' products and services will use standardized, freely-available federal data to find waste and abuse in federal spending, illuminate systemic risk in the financial markets, and help government and the private sector make better decisions. • This page will feature demonstrations of coalition members' data-driven solutions. • Source: http://datacoalition.org/data-transp...in-action.html See: http://semanticommunity.info/DataTransparencyCoalition.org#DATA_Demo_Day_on_July_10.2C_2012

10. My Example MY NOTE: This begins to illustrate the Use of My 5-Step Method To Get to 5-Stars With Open Data using the DataTransparencyCoalition.org Web site and then the 8 financial data sets. http://semanticommunity.info/DataTransparencyCoalition.org

11. My 5-Step Method • So what I like to do to illustrate (data science) and explain (data journalism) is the following (like a recipe): • Put the Best Content into a Knowledge Base (e.g. MindTouch*) • The DataTransparencyCoalition.org Web Pages • Put the Knowledge Base into a Spreadsheet (Excel*) • Linked Data to Subparts of the Knowledge Base • Put the Spreadsheet into a Dashboard (Spotfire*) • Data Integration and Interoperability Interface • Put the Dashboard into a Semantic Model (Excel*) • Data Dictionaries and Models • Put the Semantic Model into Dynamic Case Management (Be Informed*) • Structured Process for Updating Data in the Dashboard * Examples of tools used.

12. Put the Knowledge Base into a Spreadsheet http://semanticommunity.info/@api/deki/files/20342/DataTransparencyCoalition.org.xlsx

13. To Get to 5-Stars With Open Data * Examples of tools used. Source of Star and Definition: http://www.w3.org/DesignIssues/LinkedData.html

14. System of Systems Architecture Dynamic Case Management (e.g. Be Informed) Data Science Library (e.g. Spotfire) Data Science Products (e.g. Spotfire) S Semantic Index of Linked Data (e.g. Excel)

15. Key Concepts • Build a Network (instead of XBRL, NIEM, etc.): • System of Systems: Federal, Agency, and Program • My DoD SoS: Does It Follow Gall’s Law? • Big Challenges: Federal Budget, DoD Audit, Recovery.gov, Grants.gov, & Web-Services.gov • The Federal Budget Network Could Be the Killer App for the Federal Government!

16. My Question and DTC Response • I ask the following: • What are the 8 databases and a URL to a sample of their data sets and data dictionaries? • Has anyone done a demo of making those 8 databases interoperable in an interoperability interface? • If not, that would be my first pilot to document what it takes to do that on a sample of data from each. • DTC Response: • The 8 data systems are listed below (see next slide). None is comprehensive; some are not publicly accessible; and they are not interoperable.

17. Federal Financial Information Network • Data Submitted By Agencies: • 1. Federal Procurement Data System (feeds into USASpending.gov)*:https://www.fpds.gov • 2. Federal Awards Assistance Data System (feeds into USASpending.gov): http://www.census.gov/govs/www/faads.html • 3. OMB's MAX system (not publicly accessible, but the agency budget reports that go into it are accessible, piecemeal, here)*: https://max.omb.gov/maxportal/document/SF133/Budget/FACTS%20II%20-%20SF%20133%20Report%20on%20Budget%20Execution%20and%20Budgetary%20Resources.html • 4. Treasury's payments information repository (not publicly accessible) • 5. Catalog of Federal Domestic Assistance*:https://www.cfda.gov • 6. Consolidated Federal Funds Report (recently discontinued)*:http://www.census.gov/govs/cffr/ • Data Submitted By Recipients: • 7. FFATA Sub-Award Reporting System (not separately accessible but feeds into USASpending.gov): https://www.fsrs.gov/ • 8. Recovery.gov* (will be discontinued on Sept. 30, 2013): http://www.recovery.gov *: Done previous data science work.

18. 1. Federal Procurement Data System https://www.fpds.gov/fpdsng_cms/index.php/archives/18-data-archives

19. FPDS-Spotfire https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?DataTransparencyCoalition.org-Spotfire

20. 5. Catalog of Federal Domestic Assistance https://www.cfda.gov/index?s=generalinfo&mode=list&tab=list&tabmode=list&static=download

21. CFDA-Spotfire https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?DataTransparencyCoalition.org-Spotfire

22. 8. Recovery.gov MY NOTE: Used. MY NOTE: Did not use. http://www.recovery.gov/FAQ/Pages/DownloadCenter.aspx

23. Recovery.gov-Spotfire https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?DataTransparencyCoalition.org-Spotfire

24. Data Panel-Spotfire • The Data panel is used to get an overview of the columns in all data tables, in-memory as well as in-database (in-db). When working with in-database data the Data panel is the starting point for configuring both visualizations and the filters panel, since no filters are created automatically for external data. Depending on the data source, there will be different sections available for a selected data table, see below. • In-Memory or In-Database Relational Data • Data from in-memory data tables or in-database data tables based on relational databases is simply displayed as a list of the available columns in the selected data table, If data from in-db database tables have been joined with relations in the Data Tables in Connection dialog, they can be treated as a single, virtual data table within Spotfire. If no relations have been defined, each data table in the external connection will be a separate data table within Spotfire. • In-Database Cube Data • When you are working with cube data you will see more fields in the Data panel than for the other data tables.

25. Data Network-Spotfire https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?DataTransparencyCoalition.org-Spotfire

26. Add Data Connection-Spotfire

27. Data Source Tree-Spotfire

28. Some Recommendations • Standardization of financial data is good, but making data more actionable is better. • Four of the eight Federal Financial Information data sets are available to pilot a network that will show what can be done and provide data sets to DTC members to demonstrate that. • In the broader context, DTC could provide an audit and constructive feedback service to government open data and digital government strategy efforts like Semantic Community does.