DataShare

1. DataShare New York City’s Enterprise Integration Platform Kate Fitta NYC Mayor’s Office of the Criminal Justice Coordinator Jonathan Natarajan NYC Department of Technology and Telecommunications (DoITT) Doug Mamroth iWay Software

2. Agenda • DataShare Overview – Progress to Date • DataShare Architecture • iWay Software the Role of Adapter Technology • DataShare - Future Phases

3. DataShare Overview – Progress to Date Kate Fitta NYC Mayor’s Office of the Criminal Justice Coordinator (CJC)

4. DataShare Overview • The Mayor’s Office of the Criminal Justice Coordinator (CJC) and the Department of Technology and Telecommunications (DoITT) partnered in early 2004 to deliver DataShare 2.0 – the replacement for a mainframe system utilized by criminal justice agencies (DataShare 1.0) and NYC’s new integration platform. • DataShare was designed for citywide integration, but deployed initially for 17 agencies (city, state, not-for-profits). • The Public Safety Portal, which is the user-facing dimension of the City’s integration solution for criminal justice agencies, includes both administrative and business user functions. • The initial phase concluded in July 2006 with the deployment of the Public Safety Portal and a set of GJXDM exchanges. • This undertaking was a Mayoral initiative, but agencies did not have a mandate to participate

5. 12 months for requirements gathering, solution analysis, systems integration procurement and solution selection 18 months to complete implementation – 2 major deployments 17 city, state and nonprofit agencies integrated via hub-and-spoke architecture 35 FTEs comprised the dedicated project team – City staff and consultants combined 60+ agency-side resources brought to bear during 1st phase $12M spent on professional services, HW, SW 100% of post prod support provided by City of New York upon Phase 1 completion Phase 1 – Facts & Figures

6. Phase 1 – Scope and Deliverables • Phase 1 Receivers • NYPD • Department of Probation • Department of Correction • District Attorneys (5) • Special Narcotics Prosecutor • CJA • Law Department • Dept. of Juvenile Justice (DJJ) • OCA • Legal Aid Society • Assigned Counsel Plan • HRA • DoITT (on behalf of CJIS) • Phase 1 Providers • NYPD • NYC DOC • Criminal Justice Agency (CJA) • OCA Agencies connected to new DataShare infrastructure and providing or receiving at least one data exchange via the hub or portal Agency Participation • Core Hub Features & Functionality • Push functionality • Pull functionality • Query application • Notification application Criminal Justice Integration Platform (Hub) The behind-the-scenes, criminal justice segment of the Citywide integration environment hosted by DoITT • Phase 1 Business Deliverables • NYPD arrest data push (legacy & GJXDM) • NYPD arrest notification (GJXDM) • CJA ROR interview push and pull (GJXDM) • OCA supreme and criminal court actions push (legacy & GJXDM) • DOC inmate information (legacy & GJXDM) • DOC release notification (GJXDM) • DOC jail status query (GJXDM) • CJA NYSID – SSN match (legacy & GJXDM) Criminal justice data that is shared between and among participating agencies via the hub and portal Data Exchanges/Services • Core Portal Features & Functionality • Functionality for agencies to subscribe to notifications • Functionality for agencies to perform queries • Functionality for agencies to pull data • System administration for agencies • Reduced sign-on (RSO) for DNA HITS application • e911 application link • webCRIMS application link from OCA • e-Justice portal link from DCJS Public Safety Portal Functionality and Links to Applications The user-facing dimension of the City’s integrated justice solution hosted by DoITT on CityShare

7. Decisions about DataShare architecture needed to consider the autonomy of agencies DataShare had to be architected and designed in such a way that it could be scaled for citywide integration in future phases DataShare was intended to serve as a pass-through system, with minimal data stored centrally The deployment of DataShare 2.0 and decommissioning of DataShare 1.0 should have minimal impact on agencies and result in no loss of functionality for agencies. The City decided to adopt GJXDM as its transport standard for criminal justice services Guiding Principles & Strategic Decisions

8. City decided to adopt GJXDM for all future criminal justice exchanges City decided to implement both legacy and GJXDM versions of existing exchanges via new platform to facilitate agency transition and minimize risk GJXDM SME consultants worked with provider and receiver agencies to develop schemas and leverage national model as appropriate. 30 representatives from 14 agencies attended GJXDM training in 2005 City-State collaboration a priority for 2007 GJXDM workgroup taking shape among stakeholder agencies to drive strategy going forward, especially collaboration with state and national initiatives; DOP, NYPD, DOC to play key leadership roles. City will post IEPDs for Phase 1 exchanges this fall and continue to stay coordinated with national efforts GJXDM Strategy & Approach

9. Public Safety Portal Services • Public Safety Portal Services • DOC Jail Status Query – This function enables users to enter a NYSID, book and case number or a combination of first name, last name and date-of-birth via the Public Safety Portal and retrieve the incarceration status and history for the individual(s) that match the criteria. Jail status data includes information about the inmate's current incarceration status, incarceration history, warrant history and DOC housing history from 1999 - present. • CJA Interview Pull – This function enables users to enter a docket number or arrest number via the Public Safety Portal and retrieve the details of the corresponding interview the Criminal Justice Agency conducted with the defendant prior to their criminal court arraignment. Interview data includes information about the defendant's employment, residence, education and criminal history and is only available for interviews conducted from 2005 - present. • Subscriptions for NYPD Arrest Notification (Basic) - This function enables business users to enter a NYSID, name information or location information via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the individual or location specified by the criteria. • Subscriptions for NYPD Arrest Notification (Advanced) - This function enables administrators to enter criteria corresponding to up to 150 arrest fields via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the specified criteria. • Subscriptions for DOC Release Notification - This function enables users to enter a NYSID or arrest number or book & case number via the Public Safety Portal and subscribe to be notified via email when the corresponding inmate is released from or admitted to City correction custody. • Links to Applications - DNA HITS, eJustice, e911, WebCRIMS

10. New GJXDM Exchanges • New Exchanges • CJA Interview Push - The CJA Interview Push is a new, GJXDM data exchange that was first deployed through DataShare 2.0 in July 2006. In conducting interviews with criminal court defendants prior to arraignment, the Criminal Justice Agency collects information about the defendant's employment, residence, education and criminal history and submits it as a report to judges to help them make their bail decisions. CJA sends this information to DataShare 2.0 post-arraignment as a single daily batch (Tuesday – Saturday at 2am EST) of all interviews. • NYPD Arrest Push - The NYPD Arrest Push is a new, GJXDM version of a legacy exchange that had available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains information about arrest incidents on a near real-time basis. • DOC Inmate Push -The DOC Inmate Tracking Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains data about DOC's entire inmate population: basic assessment information gathered at intake, information about inmate movement among DOC facilities, and a record of discharge. The DOC Inmate Tracking Push is a daily batch provided 7 days/week (2am - 6am EST). • OCA Court Actions Push - The OCA Criminal Court Action Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains criminal case information from OCA's CRIMS system including court dates, disposition, sentence information, judge name, and final information about case charges. The OCA Criminal Court Action Push is a daily batch provided 7 days/week (12am - 2am EST). • CJA NYSID/SSN Push - The CJA NYSID/SSN Mapping Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. The exchange was first deployed through DataShare 2.0 in July 2006 and contains a mapping of defendants' NYSID, arrest number, and social security number. CJA sends the NYSID/SSN Mapping Push to DataShare 2.0 as a weekly batch (Wednesdays at 7am EST).

11. DataShare – Current Landscape Providers Receivers Public Safety Portal Public Safety Portal 18B Web (Assigned Counsel) • Juvenile Case Management System • Law Department • Dept. of Probation • Dept. of Juvenile Justice (DJJ) NYPD Special Narcotics Prosecutor Dept. of Corrections DataShare Queens County DA Criminal Justice Agency Kings County DA Office Court Administrator Bronx County DA Manhattan DA Richmond County DA Legal Aid Society Richmond County DA Richmond County DA

12. Jon Natarajan NYC Department of Technology & Telecommunications (DoITT) DataShare Architecture

13. The DataShare & Public Safety Portal solution facilitates the sharing of information across city agencies providing business process integration for the participating agencies. DataShare will provide the following services related to enterprise applications integration (EAI) Business process routing (via publish/subscribe) Data standardization Web-based portal that can offer truly distributed business applications Business process auditing capabilities Field level filtering for messages Security 24/7 Monitoring Information Sharing Capabilities

14. MVS VAX/VMS LINIX UNIX Microsoft FTP SMTP CICS Transactions Flat Files Existing Systems Installed Base • DB2 • ADABASE • DATACOM • VSAM • ADMINS/VAX • RDB • CACHE • MUMPS • SQL SERVER • ACCESS • ORACLE • EXCEL

15. Standards-based (J2EE, LDAP, SQL) Enterprise Service Bus (WebSphere MQ and Broker) GJXDM (iWay Transformation) Public Safety Portal (Vingette & BEA WebLogic ) Involve the agencies that know the data Involve the agencies that know the processes Use existing City application implementation standards Meeting The Needs – New Environment

16. Security – Authentication, Authorization, Encryption, LDAP Performance – Handle 100’s of messages/second Scalability – Able to add new data exchanges easily Availability – 99.9% availability as ‘crime takes no holiday’ Extensibility – Expose functionality via different methods like Web services Standards – J2EE, LDAP, GJXDM Technical Architecture Requirements

17. DataShare: The Solution The Solution consists of 6 key components

18. Enterprise Service Bus (DataShare 2.0 integration message broker) Integration layer provides message brokerage/routing and integration business services for citywide agencies. Public Safety Portal Portal provides front-end Web applications (such as notifications and queries), user administration, and reports to the agency user community. Notification Services The DataShare 2.0 notification services provide the Public Safety Portal users and the criminal justice agency backend systems to subscribe to key transactions traversing the Enterprise Service Bus as notifications. Architecture & Components

19. Agency Adapter Platform Agency iWay adapters and MQ used as the gateway to and from the criminal justice agencies and the enterprise service bus hosted at DoITT Database DataShare 2.0 database is used to store application and transactional metadata. Novell eDirectory (DoITT’s enterprise directory server) Enterprise directory server stores citywide agency users and groups. The eDirectory server will be used to provide services such as distributed provisioning, authentication and authorization. Architecture & Components

20. GJXDM Adaptor Approaches at Agencies • Agencies opted for one of two delivery methods: • Flat File • Database • Agencies that opted for flat file delivery then chose the method for inserting data in the file: • Populate data elements in the XML format, which preserves intact the format and contents of the XML • Populate data elements following a comma separated value (CSV) format or other appropriate convention • Agencies that opted for database delivery then chose the method for inserting data into the database: • Insert the entire XML as is in the database (e.g. as a character large object or CLOB), which preserves intact the format and contents of the XML • Convert the XML elements into columns in a relational or flat table and insert each XML record as a row in the table

21. DataShare Leverages DOITT’s CityNet

22. Doug Mamroth iWay Software iWay Software & the Role of Adaptor Technology

23. Enables connectivity to the integration server from applications, application environments, and/or databases to an integration server platform Provides the flexibility to integrate with the presentation business logic database Exposes agencies systems for exchanging data in both directions Provides an architecture that enables separate applications to work together Allows agency applications to be enhanced, added or removed without affecting others The Role of Global Justice XML

24. Each Agency needed to provide the ability to: Make requests (“Pull”) – Services oriented This requires Request/Reply Exchange data (“Push”) – Data centric This requires Data send/receive Take part in process – Process centric This requires Event listener/emitter What iWay Initially Saw In DataShare The combination of the above is the EAI It demanded a new form of B2B Protocol not commonly available - GJXML

25. Talk to the adapter using a variety of services (SOAP, FTP, HTTP, . . . ) not the system Transforms information from an existing format to GJXDM and publishes as a Web Service Creates WSDL in the process Stores all services in a common directory GJXDM built around an B2B EDI model Transaction Type Standard Elements Provides “Wiggle Room” Best Implemented Using Trading Partner Agreements iWay’s Approach – Adaptive Framework

26. Every Agency Is Different Different Agencies Want Different Versions (GJXDM 3.0, 3.1, NIEM) Different Agencies Have All Of The Data (In Complete Records) Different Agencies Have Different Security Requirements Different Agencies Have Different Privacy Policies Different Agencies Have Different Backend Systems Different Errors Must Be Handled In Different Ways Thousands Of Messages Will Be Coming From Many Partners, Many Different Sources Think Of Every Agency You Exchange Information As A Trading Partner Always Consider Your Partners Capabilities Trading Partners

27. Real-Time Processing To Handle The Differences • Basic MOU • Profile • Formal names • Error handler contacts Message Manager: the history of a message Correlation Manager: expectations for a message • Names of transforms • Process flows • Pipelines • Access keys (i.e. ISA06)

28. Adapter Technology Doug’s material

29. Enables connectivity from applications, application environments, and/or databases to an integration server platform Provide the flexibility to integrate with the presentation, business logic or database tier of an enterprise application Act as an extension to an enterprise application, providing a flexible solution for exposing external GJXDM messaging interfaces which can be leveraged for exchanging data in both directions Provide an architecture that enables separate applications to work together in a de-coupled fashion such that applications can be easily added or removed without affecting others Adapter is based on iWay Software Agency Adapter Platforms

30. Phase 2 Objectives Related Integration Initiatives Technical Enhancements DataShare – Future Phases

31. NYC Objectives for Integration in P2 1) Expand List of Participating DataShare Agencies • NYS Division of Criminal Justice Services (DCJS) • NYS Office of Children & Family Services (OCFS) • NYC Department of Investigation (DOI) 2) Expand List of Services Available Via DataShare & Public Safety Portal • Integrated Juvenile Justice (IJJ) exchanges • eArraignment exchanges • Assorted other exchanges and queries 3) Enhance Existing Shared Applications: • DNA HITS • 18B Web • e911 4) Develop New Shared Application - eArraignment

32. DataShare & Related Integration Initiatives

33. Public Safety Portal • Shared Criminal Justice Applications • eArraignment • E911 • 18B Web • DNA HITS • Integrated Juvenile Justice • Law Department • Dept. of Probation • Dept. of Juvenile Justice (DJJ) New York State Agencies (e.g., DCJS) DataShare 2.0 (DoITT Hub that Brokers Data Exchanges Among Agencies and Applications) OCFS Dept. of Correction (DOC) ACS NYPD Criminal Justice Agency (CJA) Human Resource Authority (HRA) Assigned Council Plan (ACP) Office for Court Administration (OCA) Legal Aid Society (LAS) Special Narcotics Prosecutor Manhattan DA Queens County DA Kings County DA Bronx County DA Richmond County DA NYC Integration - Future Landscape

34. Phase 1 Phase 2 • Phase 1 Receivers • NYPD • Department of Probation • District Attorneys (5) • Special Narcotics Prosecutor • Law Department • Dept. of Juvenile Justice (DJJ) • OCA • Legal Aid Society • Assigned Counsel Plan • HRA • DoITT (on behalf of CJIS) • New Providers • Department of Probation • District Attorneys (5) • Special Narcotics Prosecutor • Law Department • Dept. of Juvenile Justice • Legal Aid Society • 3-1-1 • Department of Finance • NYS DCJS • Phase 1 Providers • NYPD • NYC DOC • Criminal Justice Agency (CJA) • OCA • New Receivers • NYS DCJS • NYS OCFS • NYC DOI Agencies connected to new DataShare infrastructure and providing or receiving at least one data exchange via the hub or portal Agency Participation The behind-the-scenes, criminal justice segment of the Citywide integration environment hosted by DoITT • Core Hub Features & Functionality • Push functionality • Pull functionality • Query application • Notification application Criminal Justice Integration Platform (Hub) • Platform Enhancements • Workflow pilot to support juvenile justice process • Federated query • SOA • Geo-coding service • Business Deliverables • Phase 1 exchanges sent to additional receivers • DOP’s Pre-sentencing investigation (PSI) report for NYPD, DAs • Juv. Just. exchanges between Law, DOP, DJJ, OCFS, Fam. Court • DAs’ declined to prosecute data for DCJS and NYPD • DAs’ protected parties data for OCA • Family Court UCMS data for 18B Web • Legal Aid assigned attorney data for OCA • NYS DCJS criminal history (rap) • Business Deliverables • NYPD arrest data • NYPD arrest notification • CJA ROR interview • OCA supreme and criminal court actions • DOC inmate information • DOC release notification • DOC jail status query • NYSID – SSN match Criminal justice data that is shared between and among participating agencies via the hub and portal Data Exchanges (all GJXDM) • Core Portal Features & Functionality • Functionality for agencies to subscribe to notifications • Functionality for agencies to perform queries • Functionality for agencies to pull data • System administration for agencies • DNA HITS application link • e911 application link • webCRIMS application link • e-Justice portal link from DCJS • Portal & Supported Applications • DNA HITS application enhancements and RSO • 18b Web application enhancements • NYPD Photo application link • NYS DOCS application link • Integration with e-Justice The user-facing dimension of the City’s integrated justice solution hosted by DoITT on CityShare Public Safety Portal Functionality and Links to Applications Jan ‘05 – July ’06 Aug ’06 – Dec ’07 NYC Integration – P2 Scope and Deliverables Phase 1 Phase 2 (targeted)

35. The Integration Service Offering solution has been built on flexible infrastructure, software & hardware, with expandability in mind. As technologies and user needs evolve, the Enterprise Service Bus and its supporting J2EE infrastructure will enable new levels of integration through the use of solutions such as: Web Services (centralized web services registry) Distributed / federated applications Business activity monitoring (BAM) Workflow Business process choreography (BPC) As various City agencies see the true value in this solution their needs and requirements will evolve. This constant evolution process will inform / guide DoITT in its decisions to provide the above mentioned solutions or functionality. Technology Enhancements

36. Q&A