Open Data Policy and Practice

1. Open Data Policy and Practice Prof. Beth Simone Noveck Dominic Mauro Institute for Information Law and Policy New York Law School V 1.0 May 23, 2011 Edits and comments to dominic.mauro@law.nyls.edu

2. Agenda • To provide guidance, informed by various models, including San Francisco, Portland and the US Federal Government, about best practices for legislating and implementing open data. • Open Data Policy – suggestions for drafting open data policy. • Open Data Practice – suggestions for crafting an implementation strategy.

3. 1) Open Data Policy

4. Open Data Policy: Suggestions for Legislation • Goals • Defining Data • Open Formats • Implementation • Setting Priorities • Deadlines and Milestones • Enforcement • Prizes and Challenges • Budget • Legislation should not be long or complicated so long as there exists a robust implementation plan. • Legislation should be addressed to the public in plain English reserving technical detail for subsequent implementation. • Topics to include:

5. Open Data Policy: The Value Proposition • By explaining why open data matters, you will ease the path to compliance. Communicate that: • open data refers to more than data about the functioning of government but includes the data that the city holds. • open data legislation enables collaboration using data to the end of solving problems, not on releasing data for its own sake. • For example: “To amend the administrative code of the city of New York, in relation to making appropriate data available to the public to the end of enabling collaboration between the public and the public sector in identifying data-driven efficiencies for government, innovative strategies for social progress, and economic opportunities for all New Yorkers.”

6. Here’s An Example of What Others Have Said • Portland • WHEREAS, the City of Portland is committed to engaging the community by working with citizens, soliciting their ideas, input, and creative energy; and • WHEREAS, the City of Portland is committed to using technology to foster open, transparent, and accessible government; and • WHEREAS, by sharing data freely, the City of Portland seeks to develop opportunities for economic development, commerce, increased investment, and civic engagement for Citizens of the Portland region; and • WHEREAS, publishing structured standardized data in machine readable formats creates new opportunities for information from different sources to be combined and visualized in new and unexpected ways, for niche markets to be identified and developed, and for Citizens to browse, interpret and draw attention to trends or issues with greater efficiency; and • WHEREAS the adoption of open standards improves transparency, access to public information, and improved coordination and efficiencies among bureaus and partner organizations across the public, non-profit and private sectors; and • WHEREAS, the City seeks to encourage the local software community to develop software applications and tools to collect, organize, and share public data in new and innovative ways;

7. Here’s An Example of What Others Have Said • San Francisco: goal of legislation is to enhance government transparency and accountability, develop new applications based on the unique data the city provides, mobilize the city’s high-tech workforce at no cost to the city, and create social and economic benefits based on innovation in how residents interact with government. • White House: open government directive “intended to direct executive departments and agencies to take specific actions to implement the principles of transparency, participation, and collaboration ... toward the goal of creating a more open government.” • Model: the stated goal is to justify city officials in directing resources toward transparency projects and have no political excuse for not complying.

8. Open Data Policy: Defining Data • Legislation should mandate that all data be inventoried. • Prioritize types of information to release. • Defining “data” or “dataset” must not preclude textual information such as the laws and ordinances. One of the most successful open gov projects to date has been the revamped Federal Register. • Find the right balance between defining data in the legislation and the implementation plan.

9. Here’s How Others Have Defined Data • Federal Government Definition: • High-value information is information that can be used to increase agency accountability and responsiveness; improve public knowledge of the agency and its operations; further the core mission of the agency; create economic opportunity; or respond to need and demand as identified through public consultation. • This general prescription to policymakers was followed by specific instructions articulated by the CIO to the CIO Council with the input of public interest and technology groups.

10. Open Data Policy: Open Formats • Any legislation needs to require that data be released in human- and machine-readable formats. • Legislation can list examples of open formats but details of domain specific metadata can be provided in subsequent guidance to CIOs, especially standards for providing an API to enable applications to continually use up-to-date information.

11. Here’s What Others Have Said • Federal Open Government Directive: • “Each agency shall take prompt steps to expand access to information by making it available online in open formats.” • “To the extent practicable and subject to valid restrictions, agencies should publish information online in an open format that can be retrieved, downloaded, indexed, and searched by commonly used web search applications. An open format is one that is platform independent, machine readable, and made available to the public without restrictions that would impede the re-use of that information.”

12. Here’s What Others Have Said • Model Open Government Initiative: • Strongly recommends the publication of multiple open formats to maximize usefulness of information. • Gives examples of open formats: HTML, XML, CSV, JSON, RDF, or XHTML. Information must be human-readable as well as machine-readable. • If information is published in PDF format, it must be configured to open with bookmarks for enhanced navigation on-screen.

13. Open Data Policy: Implementation • The legislation is only as good as the implementation plan that follows it. Legislation should mandate an open public process for developing the implementation plan. • Instead of directing one department to craft a plan to make a plan, legislation should both: 1) appoint one department as the coordinator/enforcer to craft an implementation plan; and 2) direct all agencies to develop their own open data plans to inventory their data. • There should be an aggressive timeline for progress towards a complete data inventory, regular publication of data sets starting with some number within 30-45 days of enactment, and a mandatory and iterative public consultation process. • The coordinating department should be mandated to develop technical guidelines, provide support and convene the agencies but everyone should be addressed.

14. Here is What Others Said • Federal Open Government Directive: • Individual agencies are explicitly directed to take action on their own. • Examples include publishing data sets, creating web portals, developing their own open government plans, responding to feedback from members of the public, and designating a senior official to be responsible for the quality of published data sets. • Model Open Government Directive: • City departments are directed to publish information online and develop a schedule for their release of data sets. • Further, they are to develop their own open government web page, including a mechanism to receive and respond to public feedback.

15. Open Data Policy: Setting Priorities • An interagency working group of senior officials should develop priorities together by identifying challenges (ie. reducing homelessness, improving literacy, fighting obesity) and then cataloging data each agency holds that can help solve them. • Any information provided under local open records laws (FOIL) should be published on the open data portal automatically. • Information that has been the most often requested should always be prioritized for publication.

16. Open Data Policy: Setting Priorities (cont’d) • In addition to milestones for data publication, legislation should mandate milestones for: • public consultation in setting priorities citywide • public consultation on an agency-by-agency basis in connection with the data inventory • public engagement in connection with using public data to address priorities and problems • All agencies should articulate examples in plain English of what they hope the public will do with the data to the end of helping residents. • All new regulations should be published with accompanying data on which the regulation was based.

17. Open Data Policy: Deadlines and Milestones • Release data sets early and often. At the federal level, the Open Gov Directive mandated: • Release of initial data in 45 days. • Creating inter-agency working group to develop priorities and best practices for data identification, dissemination, and public collaboration within 45 days. • Establishment of every agency’s open data page within 60 days (to feed central portal). • It is reasonable to catalog public information already online within 90 days. There are tools to automate this process. A year is reasonable for a full inventory. • Mandate that support be provided to facilitate this process (see Implementation), ie. the department should delineate the tools and resources available for agencies to publish their data sets as soon as possible. • Mandate that all data sets be released with examples of what the agency is hoping the public will do with the data set, ie. we hope that someone will develop mashups using this environmental quality data in order to help identify priority sites in need of clean-up. • Deadlines to establish deadlines don’t count!

18. Here’s What Others Have Said • Federal Open Government Directive: • requires each agency to identify and publish in an open format three high-value data sets within 45 days. Also within 45 days, each agency must designate a senior official to be accountable to for the quality of the published data sets, and the Federal OMB, CIO and CTO will establish a working group to focus on transparency. • Within 60 days, each agency must create an open government webpage; further, the OMB will release a test data set about federal spending, and later assess whether additional quality controls for released information are needed. Also within 60 days, the Federal CIO and CTO must create an open government dashboard. Within 120 days, each agency must publish an open government plan on its web site. • There is a general agreement that in a matter of months, a portal should be set up and an open data plan be published for each agency. Publishing an immediate series of high-value data sets is also a common theme in the Model language and the Federal language as a near-term goal.

19. Open Data Policy: Enforcement • Agencies should be responsible to the public not simply to another agency as that would only create a further bottleneck. • Public consultation should be a central part of any open data legislation. • For example, at the federal level each agency is required to set up a mechanism for the public to: • “Give feedback on and assessment of the quality of published information; • Provide input about which information to prioritize for publication.” • And is required to give feedback to the public on a regular basis. • In addition, there should be sticks and carrots for enforcing compliance with deadlines.

20. Open Data Policy: Prizes • If further legal authority is needed for agencies to use prizes and challenges for use in running apps contests using open data then it should be added to the open data legislation. • Prizes and challenges have had high returns on investment. • For prizes totaling $20,000, the NYC Big Apps competition generated $4.25 million in apps that used data sets from the NYC Data Mine. • There should be challenges running 365 days a year, with a different vertical focus each month and the ability to develop both big and small, publicly and privately funded challenges. • ‘The legislation should create the authority to explore use of cash and non-cash prizes as well as contracts and in-kind gifts.

21. Open Data Policy: Budget • Legislation should provide a modest budget to support: • the work of the coordinating agency to hire an open data expert to develop the implementation plan and advise the agencies on their implementation process, especially how to make data available to help achieve important goals. • the acquisition of software for automating crawling domains to “discover” data sets. With the right tools, we can drastically reduce the effort required to extract data and put it in the cloud. • Public seed fund to match private sector donations in support of prizes and challenges.

22. 2) Open Data Practice

23. Open Data: Practice • The slides that follow lay out suggestions for the plan to implement the legislation. • The basics of that plan should be in place prior to legislating to ensure a pathway to implementation. • A single agency should “own” coordination of implementation and, in addition, every agency needs a senior official with policymaking authority to take responsibility in the agency.

24. Open Data Practice: Release Early and Often • Require immediate release of a small number of datasets. • By publishing data sets immediately, this will accustom agencies to the practice and help to identify problems and opportunities early. • The federal directive required the release of three high-value datasets in the first 45 days.

25. Open Data Practice: Require a Comprehensive Inventory • Agencies must know what they have. • Each agency should create a team comprising technical, legal/FOIL, and policy to oversee the process of getting the agency to inventory its data. • Crawlers can help to automate the process of finding data on the domain.

26. Open Data Practice: Publish Data on a Rolling Schedule • Start by “doing something.” Release data! • Require a regular release of data on a schedule, ie. at least # of data sets per quarter. • Citizen enthusiasm and participation will decrease if data is only published annually.

27. Open Data Practice: Articulate Priorities • Adopt a regular schedule for articulating priorities, ie. reducing childhood obesity, improving literacy or reducing homelessness. • Every agency has to respond with whatever data it has or that the public demands that could help achieve that goal. • In addition to data in support of agreed upon priorities, prioritize: • accountability data (specify certain common data sets to start); • data requested by the public; • data designed to achieve the agency's core mission; • data to create economic opportunity.

28. Open Data Practice: Establish a Community of Practice • Convene an inter-agency community to exchange best practices on doing the data inventory. • Agencies to be represented by senior officials with policymaking authority not only CIOs. • With hundreds of thousands of data sets waiting to be published, it’s important to have help with knowing how to take stock of an agency’s information. Agencies can help one another.

29. Open Data Practice: Interdisciplinary Teams in Agencies • Doing open data right requires legal, technical, and policy input. Create teams from different disciplines within agencies. • Use this opportunity to ensure that CIOs and FOIL officers, technical and policy people, communications and new media are all talking to one another about how to use data to achieve the agency’s core mission.

30. Open Data Practice: Coordination • One central department with technical expertise should provide assistance, such as: • a blanket purchase agreement for software to inventory agency domains • Developing model data standards and formats

31. Open Data Practice: Public-Private “SWAT” Teams • Rather than have all the work fall to one Department, use large community of advocates for Implementation and Accountability. • Establish “SWAT” team with representative from coordinating department and two organizations from the transparency community. The swat team should include one open data techie and one good government advocate. • Swat team provides advice and support and helps to “market” availability of new data sets to the community interested in using them.

32. Open Data Practice: Establish Regular Mash-a-Thons • Every quarter, Administration and agencies articulate a priority, identify and release data relating to addressing that issue. • An agency is selected to coordinate. • Public-private partnerships established to fund prizes and challenges in connection with that issue. • Mash-a-thons scheduled to support development of data-driven solutions.