S&I Framework LRI Validation Suite HL7 V2 Lab Results Interface Testing Overview

1. S&I Framework LRI Validation Suite HL7 V2 Lab Results Interface Testing Overview Robert Snelick National Institute of Standards and Technology October 18th, 2011 Contact: robert.snelick@nist.gov

2. Overview • LRI Validation Suite Status as of October 18th, 2011 • Testing policies and approach documented in this slide deck are proposed and still under discussion • Charter • Validation Suite Scope • LIS Test System • EHR Test System • Use of tools • Testing Flows • Test Data • Evaluation Measures • Validation Suite Artifacts • Work in-progress • Timeline Estimate

3. LRI Validation Suite Charter NIST LRI Test Tool AmbulatoryEHR ONC S&I Framework Test Data LRI ORU R01 S & I Framework EHR Pilots LIS LRI Validation Tool EHR LRI Test Harness LRI ACK R01 Vendor EHR Products • NIST LRI Test Tool • Suite scope limited to the requirements specified in the LRI IG • Develop test data • provided & verified by the S&I Framework community • LIS LRI Validation Tool • Used to validate vendor LIS systems & EHR LRI Test Harness • Develop a test plan • Capture and validate LIS LRI messages • EHR LRI Test Harness • Used to validate EHR systems • Develop a test plan • Simulation of an LIS system • Manage, Send, Receive LRI Messages • Inherently provides CLIA Testing • Targets • S & I Framework EHR Pilots • Vendor EHR Products • MU Certification • Ambulatory EHR Systems • EHR Pilots • Draft standard trial use (implementation) • Pilots are a function of the S&I Framework LRI Pilots WG • Verifies LRI IG can be implemented—provide feedback • Use LRI test harness to perform conformance testing • Second stage is to test with pilot/production LISs • Vendor Product EHRs • Can use LRI test harness to perform conformance testing • LRI Test Harness expected to be used for MU certification Part of validation suite deliverables Anticipated users of validation suite products

4. LIS LRI Validation Test Tool Capabilities • Validate Laboratory Information Systems (LIS) messages in accordance with the ONC S&I Framework Lab Results Interface (LRI) implementation guide (IG) • Including vocabulary requirements • Automated Validation • Send/Paste/Load message into tool • Obtain validation report • Testing Modes • Context free • Validate any LRI message created from LIS • Context based • Test script which includes specific lab results to be entered into the lab system • Lab creates message based on test script data sheet • Validates technical requirements and specific content based on data sheet (E.g., specific LOINC codes) • Capable of testing more of the IG than context free testing

5. LIS Validation Tool Test Flow (Context Free) Context Free Testing: Context free testing will validate a LRI message created by the LIS The context (e.g., the type and results of the lab test) is unknown to the validation tool Therefore not all conformance requirements of the LRI implementation guide can be assessed However, the validation provides a simple and convenient method for testing message structure and most vocabulary Process: An LRI message is created by the vendor’s LIS The message is sent, pasted, or loaded in the test tool A validation is performed VendorLIS HL7 V2 Lab Results Message • Load • Cut/paste • Send LIS Test Tool select test case Validation Report

6. LIS Validation Tool Test Flow (Context Based) Context-based Testing: Context-based testing will validate a LRI message created by the LIS The context (e.g., the type and results of the lab test) is known to the validation tool Therefore all conformance requirements of the LRI implementation guide can be assessed • A lab test is ordered for a patient • The specimen is collected, and is received and processed in the lab • The lab result is produced and stored in the LIS database • The lab result is transmitted to an ambulatory EHR • The lab result is viewed in the ambulatory EHR Use Case Test Case • EHR transmits Hemoglobin blood test order to the LIS for John Doe along with pertinent demographic and order request data Manual entry of test data Lab Results Data Sheet John Doe DOB: 05/23/1959 Gender: M Result: 13.7 g/dL Range 13.2 – 16.2 Status: Final Test Data Sheet Vendor LIS HL7 V2 Lab Results Message • Load • Cut/paste • Send Process: A technician enters lab results data into LIS based off the data sheet provided The message is sent, pasted, or loaded in the test tool A validation is performed LIS Test Tool select test case Validation Report

7. EHR LRI Test Harness Capabilities • Validate EHR systems that receive messages in accordance with the ONC S&I Framework Lab Results Interface (LRI) implementation guide (IG) • The LRI EHR Test Harness simulates the LIS • Create and send LRI IG conformant test messages • Develop test plans • Test procedure and methodology • Test cases • Test data (quality/quantity is dependent on community) • Juror document for inspection testing • Test cases are based on test scenarios (i.e., a context is provided) • Automated and Inspection Testing • Automated testing of acknowledgement messages • Inspection testing of EHR • Message configuration • Allows for local requirements (e.g., Patient IDs)

8. EHR Test Harness Test Flow—Testing the EHR (Model 1) Use Case Inspection Testing can be performed by: • On-site inspection • Over a webex like technology • Screen-scraper or screen-capture (include clock) • Printed Reports LRI EHR Test Harness Test Case Inspection Testing Techniques • EHR display screens • Database access • Configuration files Test Data Juror Document LRI Test Message Lab Message EHR Communication ACK ACK Automated Testing • Acknowledgement Message • Limited Utility Validation Report Validation

9. Can we Leverage the MU Stage-1 Requirements? S&I Framework LRI HL7 ELR IG 2010 LAB (or any sending system or tool) EHR (System under Test) Public Health Meaningful Use Scope Stage 2 Meaningful Use Scope Stage 1 ORU Message ORU Message ORU Elements DirectData Entry NIST Message Validation Tool ORU Message Elements Meaningful Use Stage-1 requirements includes the EHRs to send reportable lab results to public health using the HL7 Version 2.5.1 Implementation Guide: ELR to Public Health (Release 1: February 2010) NIST test tool already exists and used for Meaningful Use stage-1 certification Identify which test cases fall into the category of reportable lab results Identify the differences in the implementation guides (data elements, etc.) Provide data sheet for additional information that needs to be entered into the EHR system Correlate content of public health message to original LRI test case message This process provides additional automated testing Test EHR workflow

10. EHR Test Harness Test Flow—Testing the EHR (Model 2) Use Case Inspection Testing can be performed by: • On-site inspection • Over a webex like technology • Screen-scraper or screen-capture (include clock) • Printed Reports LRI EHR Test Harness Test Case Inspection Testing Techniques • EHR display screens • Database access • Configuration files Test data sheet for additional requirements Test Data Juror Document LRI Test Message Lab Message EHR MU-1 Public Health Message Communication ACK ACK Automated Testing • Acknowledgement Message • Limited Utility LRI ACK Validation Validation Report Public Health Lab Message Validation Tool Automated Testing HL7 ORU Message High Utility

11. Test Data • Selection of LRI conformant and non-conformant messages • Provides example and test messages • In-scope messages include the recommended lab results list from the vocabulary WG (~144 lab results) • Validation will not be limited to this set; although users will need to provide the test messages • Goal is to provide at least one test case for each of the recommended lab tests (the validation suite WG is dependent on the community to provide) • Recommended lab tests are categorized and associated LOINC code(s) are given for each lab test—will be used in testing • Initial validation suite WG will provide a core set of test messages • Core set will include a message from each of the major categories • Currently in the process obtaining test messages (Can you help?) • Variations on the core set of messages are designed to test the technical requirements of the LRI IG

12. Test Messages • Test Categories • Common Hematology • Urinalysis • Common Chemistry • Special Chemistry (e.g., Calcidiol, Thyroxine (T4)) • Microbiology • Common ATB Susceptibility • Panels (CBC, CMP, etc.) • Other common (e.g., surgical pathology) • Initial core set of test messages • Provides at least one message for each of the categories we identified • We will use these to finish test plan skeletons, to develop data management spreadsheet, create test case variations, etc. • We have example messages for each—not yet mapped to LRI IG requirements (Work in progress)

13. Validation Suite Artifacts • Test Plans • Test Procedure • Test Cases • Validation Criteria • Collection of test cases/test data • Maintained in a multi-dimensional spreadsheet • Test messages • MWB message profiles • ORU (4 versions) and ACK • Formats: MWB, HL7 V2 XML, HTML (integrated into tool) • Collection of value sets • Spreadsheet • NIST XML format • PHIN VADS (goal is to have PHIN VADS as the source of truth) • HTML (integrated into tool)

14. Test Dimensions (What are we testing?) • Structure (segment/field/datatype) requirements • Message Structure • Usage, Cardinality, Data Type, Length, Conformance Length • Truncation • Data Format (Pattern Matching) • Element Relationships • Constant Values • Content • Terminology • Test case specific content • Invalid content • Use Case Scenarios • E.g., Preliminary, Final, and Corrected Results • Profiles (GU, NG, RU, RN) • Negative/error testing • Minimal, Maximum (usage), Maximum (Cardinality) • Large message (large text strings) • Special Cases • Different uses of OXB.2/OBX.5 • Snapshot processing • Terminology breath (scope of LOINC) • PID.5 for unknown patient name • And more…

15. Action Item List I • Select message to handle core lab results • Identify 20 or so common lab results (In progress) • Obtain/Adapt/Create test messages to cover the core set of lab results (In progress) • Identify/List all pertinent data elements (In progress) • Create spreadsheet of all data elements with usage of R, RE, and C (rows) • Columns will identify: • Juror Document (How to assess the element) • Identify the elements required for CLIA testing • Identify static, configurable, or indifference data elements • Identify/create/verify value sets (In progress) • Create Spreadsheet; convert to NIST XML tool format • Incorporate the value sets in PHINVADS • Develop download mechanisms and transformation of values to support the NIST tooling format

16. Action Item List II • Review LRI implementation Guide and create a list of all conformance requirements (Not Started) • Create matrix based on data elements • Link all conformance requirements to data elements when possible • Create “higher” level list of conformance requirements • Determine the policy for assessing receiver side terminology (Done: need to write policy statement) • Inspection test requirements and procedure • Automated test requirements and procedure • Complete development of LIS Test Plan Skeleton • Complete development of EHR Test Plan Skeleton

17. Action Item List III • Identify and document the test dimensions (Not Started) • Coverage of Lab Results • Scenarios (e.g., Preliminary, Final, Corrected) • Reporting formats • Negative testing • Minimally and maximally populated • Contact CLIA and CAP inspectors to get their lab inspection process (Need contacts) • Determine a process for verifying test cases (Not Started) • Implement process for verifying test cases (Not Started) • Research ELINCS Test Tool (DONE) • Determine what we can leverage • Process flow, source code, test messages

18. Action Item List IV • Identify all the public health reportable lab results (Completed) • Identify the data elements that differ from the public health IG and the S&I Framework LRI IG (Not Started) • Determine a policy for validating LRI messages using EHR PH lab results messages (Not Started) • Develop spreadsheets for managing test cases/data (In progress) • Adapt tooling to process and incorporate data • Phase 1 nearly complete • Phase 2 will include the multiple dimensions (Data, Profile, Juror) • Create the HL7 standard message profiles (Starting soon) • MWB (then produce XML message template) • Need to make updates to the message profile based on changes made in version 2.7 and 2.7.1 • Write XSLT to modify XML message profile

19. Action Item List V • Identify the CLIA conformance requirements and compare to the requirements in the implementation guide (In progress) • Mark in spreadsheet – (DONE) • Make sure conformance requirements and IG match • Write conformance requirements in IG where necessary to match CLIA requirements • Prototype tool (In progress) • Requirements and design (In progress) • Development (In progress) • Incorporate test cases (Not Started) • Testing (Not Started)

20. Anticipated Timeline (Estimate) • Earlyprototype—December 2011 • Work-in-progress; available for pilots; seeking feedback • Based on draft implementation guide • Support for 1 profile variation (e.g., Profile GU-RU) • A limited number of test cases (3-5 messages with variations) • Note: each test case/message can have many variations to test the technical requirements of the IG • Draft LIS and EHR test plans and testing procedures • Version 1—3/4 Months after publish IG • Based on final published IG • Support all 4 profile variations • 7-10 core test cases/messages (with variations covering all conformance requirements)** • Finalize LIS and EHR test procedures (serve as a template for adding test cases/messages) • Version 2—October 2012 • Full feature tooling and test plans • Meaningful Use test procedures • 15-20 test cases (with variations)** • Longer Term Goals • Goal is to provide test cases for the recommended 144 most common lab results • Validation Suite WG will provide mechanism (template) for adding test cases • S&I Framework community needs to provide test cases/messages** ** NIST is relying on help from the community to provide and verify test messages—test messages will be limited otherwise