Certification and Conformance Approaches for Canada’s EHR

1. Certification and Conformance Approaches for Canada’s EHR Elaine Sawatsky, Senior Management Consultant, Privacy & Security, Canada

2. Conformance: Canada’s EHR • Infoway Conformance Services in Canada • Supporting the Implementation of pan-Canadian Standards • Provincial/Territorial work • On the ground experiences

3. Conformance: Canada’s EHR • Sharing information requires a quality assessment of products to ensure interoperability, to reduce risk to patients, purchasers and the ‘system’. • Standards support sharing data and bringing it together • Improves comparability and harmonisation between products. • Certification: “a process of validation of products in a given context against specific validation criteria.” • Certification criteria are – even regarding the same functionality – up to a certain degree similar but also different between jurisdictions in: • The way they are formulated & validated • The importance given to each of them • Certification should be unambiguous, fair and trustworthy

4. Conformance Services Vision • To support the successful implementation of pan-Canadian Standards within projects through the provision of conformance testing artefacts and tooling. • To improve theconsistent implementation of pan-Canadian Standards • To reduce the effort required to implement pan-Canadian Standards • To reduce the cost of implementing pan-Canadian Standards • Strategic Intent of Conformance Services: • Ensure that there is a national service to support the vendor conformance and certification of iEHR, public health and EMR solutions.

5. SK EHR Blueprint and Concepts

6. The need: Bridging The Gap Vocabulary Authoring Assessment Certification Profiles Release Change Helpdesk Need Options Develop SFU The challenge is to bridge the gap that exists between the health informatician (modeler) and the clinical integration specialist (solution developer). Health Informatics Modeler Integration Specialist EHR in Operation Standards Implementation, Conformance & Maintenance Standards Development Tooling • ModelingCulture • DeliveringSpecifications • Project and Product Culture • Delivers Production Solutions 6

7. Conformance: Canada’s EHR

8. Conformance: Canada’s EHR Approach • There is recognition that there is a need for conformance testing to ensure alignment • The approach is founded on international standards • Prototype & Learn – conformance testing is not well understood yet • Build Methods & Processes with stakeholder input • Build tooling & leverage : • Standards – HL7 v3, LOINC, SNOMED, XDSI • Tooling • Component model – integration is key! • Sharing of testing artefacts – huge value to stakeholders

9. Conformance: Canada’s EHR Key Messages • There is recognition that we all need to do end to end and integrated testing • We are committed to leveraging work of others • We are making good progress in developing processes and tooling to assist our jurisdictions in implementing “Conformant” solutions

10. Conformance Continuum Conformance includes: Message content and structure Vocabulary and terminology Interoperability profiles and message sequencing Sending and receiving applications and intermediaries Conformance Testing includes: Functionality of application Basic privacy and security services related to the functionality Workflow (to an optimal level determined by conformance continuum) 10

11. Our Approach First, develop Profiles for all pan-Canadian standards Develop a process & mechanisms linking the pan-Canadian Profiles through to conformance test Develop, deploy, continuously improve “quick-hit” tools Provide business and technical support for Investment Projects to set up and execute conformance testing Document & refine requirements for long-term solution Harvest artefacts for sharing with other jurisdictions Rationalize jurisdiction artefacts to create pan-Canadian versions Establish a pan-Canadian Working Group of early participant jurisdictions and their vendor partners to provide guidance and assistance in the development of Conformance Testing artefacts

12. Scope of Part 4 Our Approach Change Management Tools and Templates Standards Selection Strategy Licenses Standards Knowledge Management Stable for Use Evaluation Liaison Final Approval Promotion & Communication Adoption Strategies

13. Conformance: Canada’s EHR Benefits: • Financialsavings through reduced duplication of efforts in test authoring • Provinces leverage other provinces’ Business Rules • Financial savings though tools based test message development • Provinces use Infoway tools and other provinces’ messages • Breadth of testing improved through increased quantity and quality of test cases • Provinces are committed to sharing test artefacts.

14. Conformance: Canada’s EHR Benefits: • Depth of testing improved through consistent test rules and tooling • Provinces’ use of tooling has improved UAT testing Too early to accrue benefits • Quality and consistency of conformance testing improved between Investment Projects • Too early to accrue benefits • Improved consistency in Investment Project implementations of pan-Canadian standards • Too early to accrue benefits

15. Benefits Providing Implementation support for the development of Test Authoring and Test Execution will enable jurisdictions to leverage and share Conformance Profiles and testing artefacts, leading to: Improved quality of testing Financial savings through reduced duplication of efforts Breadth of testing improved through increased quantity and quality of test cases Improved quality and consistency of conformance testing Financial savings though tools based test message development Depth of testing improved through consistent test rules and tooling Improved consistency in implementations of pan-Canadian standards

16. Conformance: Canada’s EHR Conformance Services Overview • Main elements influencing content of test criteria: Legal, national, cultural (business and social) • Sophistication of the industry area • Environment • Certification is the process • Conformance is the result • Must be good enough, not best or everyone will fail • Area subject matter experts – what will the medical professional need?

17. Overview of Conformance Services • Test Authoring Services • Business Rule Management • Conformance Profile Management • Vocabulary Maintenance • Test Case Authoring • Test Data Management • Message Instance Authoring • Test Execution Services • Conformance self assessment & evaluation • Conformance independent verification & validation • Conformance certification Technology • Test Authoring Tools • Repository, • remote user access • shared authentication • Test Execution Tools • Conformance Validation • Test Case Validation • Self-test, monitoring • Web presence for jurisdictions and vendors • Registration, • artifacts library, • test results • Stakeholder Engagement • Governance • Collaboration on Test Authoring components • Collaboration on supporting tools development and implementation • Registry of vendor systems & certification status • Registry of implementation projects • Ongoing liaison/communication with jurisdictions and vendors • Promotion and Education around Conformance Services 17

18. Strategic Direction Analysis Project • Project Objective: • Develop a preferred option for the future delivery of Conformance Services. • Define and identify existing conformance services models; • Develop a shared vision for the delivery of Conformance Services for the Canadian healthcare sector • Develop conformance services delivery options for the Canadian healthcare sector; and detail the preferred option. • Defined Strategic Intent of Conformance Services: • “Ensure that there is a national service to support the vendor conformance and certification of iEHR, public health and EMR solutions.” 18

19. Investment Project Support Overview • Project Objective: • Provide immediate Standards Implementation Support for Test Authoring & Test Execution to key Infoway Investment projects • Project Approach: • Develop a process & mechanisms linking the pan-Canadian Profiles through to conformance test • Develop, deploy, continuously improve “quick-hit” tools • Rollout to qualified projects (if proof of concept successful) • Harvest artefacts for sharing with other jurisdictions • Provide lessons learned and input into CS Strategic Direction Project • Provide transition and support to recommended Strategic Direction 19

20. Investment Project Support Activities • The specific services that Infoway is offering to assist projects • Assistance in developing test strategies • Sort out overlap between conformance testing vs. UAT • Education on development of business rules & test cases • Technical education on development of message instances and test data • Sharing of testing artefacts • Quality review of artefacts developed by investment projects • Full set of tools to assist getting to Conformance Testing • Assistance to jurisdiction in rationalizing and integrating their testing tools with Infoway conformance tools • Harvesting of testing artefacts for sharing with jurisdictions • Sharing of Lessons Learned • Facilitating inter-jurisdiction collaboration on conformance testing 20

21. Engagement Process to Conformance Introduction Workshop 0 Introduction Workshop 0 1 End to End Conformance Process Workshop (2-3 days) 2 Jurisdiction End-to-End Test Case Development (10%) (15-20 days) 4 Quality Review Harvest artefacts (2-3 days) Quality Review Harvest artefacts (2-3 days) 3 3 Jurisdiction End-to-End Test Case Development (20-40 days) Jurisdiction Conformance Pilot Testing, Quality Review Harvesting artefacts, Lessons learned (3-5 days) 5 End to End Test Authoring Examples Test case Examples Harvested artefacts Tools Jurisdiction Test Artefacts End to End Test Execution Examples Jurisdiction Test Authoring Artefacts Jurisdiction still need to complete, but ready to perform Conformance Testing Ongoing Harvesting 21

22. Conformance Constructs – End-to-End Process Standards Artefacts - HL7 V3 - “Quick-hit” tools - “Existing” tools Conformance Profiles Test Case Authoring Business & Technical Rules Instance Editor (Messages) Test Cases Implementation Guides Vocabulary Data HL7 Test Harness Test Data External Application Jurisdiction Input Best Practice

23. Conformance: Canada’s EHR Major Tooling Components • Business Rules Manager- Maintains Business Rules • Conformance Profile Manager - browse pan-Canadian Conformance Profiles • Implementation Conformance Profile Manager - browse profiles and create/maintain ICPs • Static Model Constraint Tool - captures changes to models, and traceability information • Instance Editor - creates Conformance Test Scenario Files, Message Fragments and Parameter Files as XML output. • Test Case Author - Maintains Test Cases • Test Authoring Manager - The main user interface • Test Execution Manager - links the Test Authoring Manager to control the test execution • Test Authoring Reports Manager - run reports

24. Conformance: Canada’s EHR Conformance Test Execution Scenarios There are different ways to perform Conformance Test Execution. This process uses the pCCT Test Execution manager and / or the HL7 Test Harness to allow the user to test both POS and iEHR systems as “real” or “simulated” applications under test. • Scenario 001 - Real POS with Simulated iEHR Application • Scenario 002 - Simulated POS with Simulated iEHR Application • Scenario 003 - Simulated POS with Real iEHR Application • Scenario 004 - Real POS with Real iEHR Application

25. Lessons Learned – Considerations for AB • Testing artefact reuse is seen to be a big win • One province has 100 business rules & 300 test cases – target is 500-700 test cases • Another target will be significantly higher (~2,500 test cases) • Others will leverage oand likely add more • Toolset are not all or nothing • One province using blend of Infoway tools and home grown • Uses Infoway tools and processes first to perform testing then conformance testing of PoS • One province using Infoway tools for go forward & PoS testing • Specific initiatives will likely use blend • Infoway is investing in tools so jurisdictions don’t have to!

26. Conformance: Canada’s EHR Tooling Vision • Create a suite of integrated Conformance Test tools based on Open Source principles and accepted pan-Canadian standards that allows jurisdictions, and vendors to perform conformance testing • Includes: • Well documented processes and procedures • A repository of structured conformance profiles, test cases, test data and business rules that can be reused • A suite of re-usable conformance testing solution components able to easily exchange structured data between each component and projects • An Enhanced Web Based Presence to allow remote administration, self registration, shared resources management, self testing / monitoring and publication of results

27. Lessons Learned – Considerations for AB • Many jurisdiction projects have left UAT & Conformance Testing until very late in their project lifecycle • There is not a clear understanding of the continuum of conformance testing • Profile  Business rules  Test cases  Messaging  Conformance test • Traceability is key to conformance test • Profile  Business rules  Test cases  Messaging  Conformance test • Using Profiles to help them define user roles • What does a physician do vs. project focus • Tools can and are providing immediate benefit to jurisdiction projects for publishing • Consolidated view of all business rules • Publication of specification for vendor 27

28. Conformance Profiles Sask Sask Nfld Nfld BC BC #47 + #84 + Library of Conformance Profiles Pan-Canadian Conformance Profile (pCCP) Extension Implementation Conformance Profile (ICP) pCCP Constraint BC Extensions Example: BC ICP #84 #84 BC Constraint Likely Initial State Desired State

29. Conformance: Canada’s EHR Tooling for Test Authoring Components (TAC) • Use prototyping methodology to conduct staged development • Test with jurisdictional review and input • Document processes, requirements and changes • Garner feedback and experience to inform the refinement of detailed specifications for long-term open source tooling development • Address immediate jurisdictional needs & provide tangible benefits • All test artefacts exportable • Work with jurisdictions to rationalize and integrate tooling • Align test cases, business rules across jurisdictions • Promote early sharing and leveraging of work by jurisdictions

30. Conformance: Canada’s EHR Tooling for Test Execution • Leverage an existing open source test harness (previous Infoway investment), adding ability for jurisdictions to easily configure to reflect local technology solutions such as transport and security • Leverage Eclipse Instance Editor for message and vocabulary creation • Use data exported from test authoring tools • Key is to ensure tools export into a format that can be consumed by HL7 Test Harness • Invest in documenting processes, requirements, integration • Rationalize and integrate tools • Develop guidelines for local Test Execution prototyping environments including a reference implementation • Develop processes to ensure data integrity between different groups of users during prototyping phase

31. Conformance: Canada’s EHR It’s not easy: • In order to get to a conformance test, you need to start with the Profile, know what the business rules are and create test data and message instances that will then test a scenario • Once you have done this for every business and technical scenario, only then can you think about performing a conformance test • By our simple calculations, this will be thousands and thousands of test cases!

32. Conformance Services Activity in Canada Panorama 0 0,1 0 0,1 0,1,2,3 0 4,5 1 0, 0 0 – Step 0 Engagement 1 – Step 1 Workshop 2 – Step 2 Activities 3 – Step 3 Workshop 4 – Step 4 Activities 5 – Step 5 Workshop 0 Planned Scheduled/Underway Completed 13-Mar-08 32

33. Conformance: Canada’s EHR • Provinces have done functional Certification for EMRs specifically. • Other certification means defining differences in function, scope, perspective (e.g. Social interaction between users, staff and patients), • Legal requirements are different between provinces, policies are different: Infoway cannot set policy but provinces must • Provinces must work in collaboration with Health Authorities and professional bodies. • Conformance criteria will be different from one business environment to another.

34. Conformance: Canada’s EHR • Work is based on existing provincial models • Establish business rules to support policy set by Ministries of Health, College of Physicians and Surgeons and College of Pharmacists • Publish compliance specifications that include all the transactions needed at the point-of-service (e.g. pharmacy claims, pharmacy prescriptions, Client Registry, Provider Registry, Common, Lab, Immunization, etc.) • Provide support to vendors during their development • Conduct compliance testing of vendor software once they pass a pre-test • Support change agents (chain stores, vendors, etc.) during their training and deployment activities

35. Conformance: Canada’s EHR POS Software Compliance Principles • POS systems • Must enhance patient safety & Support clinician workflow • Must be secure and ensure privacy of patient medical records • Must support interoperability standards • Are integral in creating and maintaining quality data for use throughout the health care system • POS vendors must be able to differentiate their products • Compliance documents • Are referred to in some provinces legislation • Communicate what rules need to be followed, not how the rules are implemented in vendor software • Provide one place to house Ministry policy, College rules and application capabilities • Provide a level playing field for all vendors regardless of size

36. Conformance: Canada’s EHR POS Conformance Specifications • Define Business rules • Based on existing business rules from compliance documents and input from Clinical Working Group • Identify items that need further clarification • Develop Compliance specifications • Based on existing compliance documents and published message specifications from Infoway • Builds on Infoway Conformance Profile definitions • Leverage, where possible, work from other jurisdictions • Materials include: • POS Software Vendor Compliance specifications; • Vendor registration and development support processes; • Compliance testing process; • Process for approving vendors with compliant software • Stakeholders included in joint review of material prior to publishing

37. Conformance: Canada’s EHR Compliance – Lessons Learned • It takes time for the vendors to develop code to the level that warrants compliance testing • Vendors may present software that was not stable enough to warrant compliance testing • Vendors can require multiple attempts at compliance testing • Thorough compliance testing takes time, especially on the very first test of new software or a major release change • Good rules are needed on what constitutes an initial test and a follow-up test for each version of the vendor software • Good rules are needed on when penalty and charges are applied to the test

38. Conformance: Canada’s EHR • Sample tests were provided for vendors • Focus needed to ensure that the vendors could prove they ran the tests and the tests worked before scheduling a compliance testing • Need mechanisms to ensure compliance team provided consistent responses to all vendors throughout the compliance support process • Good rules are needed around mandatory training that a vendor must provide e.g. training plans, training outcomes, etc. • Good test facilities are needed for vendors and for compliance testing • Multiple training environments are needed for each vendor • Good tools are needed for the compliance testers that assist them in developing the test cases and support them during the testing

39. Conformance: Canada’s EHR • Clinical guidance was critical during vendor development and compliance testing to clarify options and workflow • Requires ongoing clinical involvement • Specifications, clinical guidance and support needs to include details • Compliance team requires good communication and negotiation skills and need to be able to stand their ground under pressure • Compliance documentation needs to be targeted to specific audiences and also clearly identify elements common across audiences • A tool that enables easy linking and/or packaging of modules is highly desirable. Documents that are difficult to maintain don’t get maintained • Business context for the compliance material is just as important as the rules themselves

40. Conformance: Canada’s EHR Conformance includes: • Message content and structure • Vocabulary and terminology • Interoperability profiles and message sequencing • Sending and receiving applications and intermediaries Conformance Testing includes: • Functionality of application • Basic privacy and security services related to the functionality • Workflow (to an optimal level determined by conformance continuum)

41. Contact Standards Collaborative Information Desk Bureau d'information de l'Unité collaborative de normalisation d’Inforoute 1-877-595-3417 or 416-595-3417standards@infoway-inforoute.ca or normes@infoway-inforoute.ca Elaine.sawatsky@telus.net Privacy and Security, SME

42. Holding slide