1 / 44

Medical Device CommunicationTest Effort NIST Team Members

IHE-PCD and NIST IHE-PCD Cycle 4 (…any beyond) Test Overview and Strategy IHE-PCD Face-to-Face WG Meetings (@ Carrollton, TX) 7 October 2009. Medical Device CommunicationTest Effort NIST Team Members. John Garguilo ( john.garguilo@nist.gov , 301-975-5248)

kobe
Download Presentation

Medical Device CommunicationTest Effort NIST Team Members

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IHE-PCD andNISTIHE-PCD Cycle 4 (…any beyond)Test Overview and StrategyIHE-PCD Face-to-Face WG Meetings (@ Carrollton, TX)7 October 2009

  2. Medical Device CommunicationTest EffortNIST Team Members • John Garguilo (john.garguilo@nist.gov, 301-975-5248) • Sandra Martinez (sandra.martinez@nist.gov, 301-975-3579) • Maria Cherkaoui (maria.cherkaoui@nist.gov Guest Researcher) • *Jing Gao (jing.gao@nist.gov Guest Researcher) * Jing joined our project in May ’09 Project Web site: hl7v2tools.nist.gov Project Web site: www.nist.gov/medicaldevices

  3. IHE-PCD Testing IHE-PCD Testing – Key Objectives • Increase test comprehensiveness & quality • Support both conformance & interoperability testing • Support for pre- & virtual connectathons, actual connectathon & enable year round testing • Remain in alignment with IHE-PCD profile development road map • Establish single framework for PCD covering increasing complexity and technologies over next 5 years • Coordinate with IHE “Gazelle Project” • Generate work products that companies can use in their regulatory submissions

  4. IHE-PCD Testing IHE-PCD Testing – Key Ingredients • Well Defined Integration Profiles • Technical Framework • Supplements • Test Guidelines • Test Plans • Test Scenarios, Actors, Transactions, Validation Criteria • Test Artifacts, including: • HL7 Profile(s) (and eventually x73 Device Specializations?) • Repositories • Nomenclature (e.g., RTM) • Value Tables (e.g., HL7, units, local, etc.), Default and Sample Values • Test Cases

  5. NIST Testing StrategyOverview (for Cycle 4) • IHE-PCD Test Strategy • NIST Framework for Testing Distributed Healthcare Applications • Documentation: Test Guideline and Test Plans • IHE-PCD “NIST HL7 V2 Testing Framework” Instance • Web Site: hl7v2tools.nist.gov • Cycle 4 Pre-connectathon (and virtual-connectathon) • …and beyond…. • Support both Conformance & Interoperability • RTM Terminology (hRTM) • Integrated via context validation files for this year’s cycle • Future, likely a web service (call to repository) • Going forward – RTMMS (and tied to ISO/IEEE x73)… • Integrated as part of NIST Test Framework

  6. NIST IHE-PCD Test Guidelines (’09-’10 Pre-connectathon) • Quick, executive summary level view • Defines testing requirements for those performing testing • HL7 Messaging version (e.g., V2.6) • HL7 profile • Identifies test environment • See ‘Test Environment’ slides below… • Identifies what types of testing shall occur • Links test artifact(s) (needed) to validate specific test case(s) • Syntax Validation • HL7 Syntax/structure – basic syntax via HL7 V2.6 standard • HL7 IHE-PCD Profile Validation – required fields, order, data types, cardinality via message profile • Semantic Validation(via context validation file) • Integration Profile Validation – (semantic) via message profile file (XML) • Rosetta Validation – (semantic) terminology w/ co-constraints from RTM Integration Profile • Test case validation – (semantic) specific field values

  7. NIST IHE-PCD Test Plans (’09-’10 Pre- and Virtual- Connectathon) • Clear and concise documentation • Mirror (point to) and/or include test cases • Already defined and new test cases as developed/introduced • Necessary for rigorous HL7 profile and message validation • Defines “validation context” specifications used by tooling • Recommended document template as we go forward… • Used by IHE-PCD Integration Profile working groups to establish exactly what’s to be tested • Communicates specific validation criteria to participating vendors • I.e., no surprises • …Supporting documentation for work products • …that perhaps companies can use in their regulatory submissions • Start w/ static messages and move into scenarios • Way to identify appropriate Rosetta Terminology according to IHE use case • …Perhaps be adopted by other domains as “conformance” documentation model…

  8. IHE-PCD Cycle 4 TestingDocumentation Test Guideline Documents • NIST DEC Pre-Connectathon Test Guidelines.docx • NIST ACM Pre-Connectathon Test Guidelines.docx • NIST IDCO Pre-Connectathon Test Guidelines.docx • NIST PIV Pre-Connectathon Test Guidelines.docx Test Plan Documents • DEC_TEST PLAN.docx • DEC_SPD_TEST PLAN.docx • ACM_TEST PLAN.docx • IDCO_TEST PLAN.docx • PIV_TEST PLAN.docx

  9. NIST HL7 Profiles (’09-’10 Pre-connectathon) • NIST HL7 Message Profiles (XML format) • To be used with NIST V2 Validation Tooling… • Website:hl7v2tools.nist.gov • Currently vetting w/ Messaging Workbench (MWB) • Naming Convention Used: • *NIST_[w...w]_xxx_yyy-NN_zzz^TTT.xml • Example 1: NIST_IDCO_DOR_PCD-09_ORU^R01.xml • Example 2: NIST_SPD_DOC_PCD-02_QSB^Q16.xml • [w...w]: Integration Profile Group/Option (e.g., “PIV”) • xxx: Actor (e.g., “IOP”) • yyy-NN: IHE-PCD transaction-Number (e.g., “PCD-02”) • zzz^TTT: Message Type^Trigger Event (e.gs., “ROU^R01”, “RGV^O15”) *MWB is primary tool this cycle used by vendors, so keep “NIST” label to avoid confusion.

  10. NIST IHE-PCD HL7 Profiles (cycle 4) • NIST_DEC_DOR_PCD-01_ORU^R01.xml • NIST_SPD_DOC_PCD-02_QSB^Q16.xml • NIST_PIV_IOP_PCD-03_RGV^O15.xml • NIST_ACM_AR_PCD-04_ORU^R01.xml • NIST_IDCO_IDCR_PCD-09_ORU^R01.xml

  11. Profile ValidationFeedback to TF/Supplements… • NIST_DEC_DOR_PCD-01_ORU^R01.xml • Remaining errors concern issues related to contradictions with HL7 v2.5 versus v2.6 • (captured using HL7Web application) • Errors • Error1: segment PV1 is ‘R’ Required by the standard but Optional in DEC • Error2: segment TQ1 is ‘R’ Required by the standard but Optional in DEC • Error3: segment SPM is ‘R’ Required by the standard but Not Supported in DEC • Error4: OBX-20 is not defined in the v2.5 standard (it is in v2.6)

  12. Message Validation ContextTechnical Framework/Supplement Validation • Naming Convention: w…w-yyy-nn-ValidationContext.xml • [w...w]: Integration Profile Group/Option (e.g., “PIV”) • yyy-nn: IHE-PCD transaction-Number (e.g., “PCD-03”) • DEC-PCD-01-ValidationContext.xml • SPD-PCD-02-ValidationContext.xml • PIV-PCD-03-ValidationContext.xml • ACM-PCD-04-ValidationContext.xml • IDCO-PCD-09-ValidationContext.xml

  13. Message Validation ContexthRTM Validation • hRTM • Integrated via context validation files for this year’s cycle • Future, likely a web service (call to repository) • hDEC-PCD-01-ValidationContext.xml • hPIV-PCD-03-ValidationContext.xml • hACM-PCD-04-ValidationContext.xml • Needed: IDCO-PCD-09ValidationContext.xml

  14. NIST Test CaseMessage Validation Context (’09-’10 Pre-connectathon) • NIST validation of specific values (derived from the test cases for each Integration Profile) • Based on Use Case • Instance of Use Case ultimately demonstrated at Showcase • PIV(for PCD-03) • 60101-ValidationContext.xml • 60102-ValidationContext.xml • 60103-ValidationContext.xml • 60104-ValidationContext.xml • 60105-ValidationContext.xml • 60106-ValidationContext.xml • 60107-ValidationContext.xml • 60108-ValidationContext.xml • 60109-1-ValidationContext.xml • 60109-2-ValidationContext.xml • 60109-3-ValidationContext.xml • 60109-4-ValidationContext.xml • 60110-ValidationContext.xml

  15. Show NIST V2 Tools (via Web) Demonstration NIST HL7 Web hl7v2tools.nist.gov (subject to change) • Upload HL7 IHE-PCD ORU^RO1 profile • Upload HL7 IHE-PCD context validation file (XML) • Upload HL7 IHE-PCD message (ER7 format) • Validate • Generate and review resulting ‘Report’ • Review errors

  16. hl7v2tools.nist.gov

  17. Example Message Validation Report

  18. Example Message Validation Report

  19. View errors in HL7 Message

  20. Example RTM Error Report

  21. Example Profile Validation Report

  22. Example Profile Validation Report

  23. HL7 Message Validation StudyExample PIV (PCD-03)

  24. PIV-PCD-03 Test Case 60101Validation using the IHE TF/Supplements • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  25. PIV-PCD-03 Test Case 60101Validation Using HL7 Profile Tables • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  26. PIV-PCD-03 Test Case 60101hRTM Validation • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI64|BBRAUN|200901120909380600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  27. PIV-PCD-03 Test Case 60101Use/Test Case Specific Validation • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  28. PIV-PCD-03 Test Case 60101Overall Validation • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  29. NIST V2 IHE-PCD Validation Coverage Example Coverage % = Number of Values Checked / Number of Values in Message *May include duplicate checks from other Validation Items across validation type

  30. NIST Testing StrategyLarger Scope and moving forward…(Beyond Cycle 4) • IHE-PCD Test Strategy • NIST Framework for Testing Distributed Healthcare Applications • IHE-PCD “NIST HL7 V2 Testing Framework” Instance

  31. Test Environments • Instance Testing • Conformance (e.g., against HL7 2.x or CDA) • Implementation conforms to Spec. on which it is based • IHE Model: ~Virtual and Pre-Connectathon • Isolated System Testing • Includes Instance Testing Activities • Protocol Conformance • Functional Behavior Conformance • Features and Operational behavior correspond to Specs. • IHE Model: ~Virtual and Pre-Connectathon • Peer-to-Peer System Testing • Includes Isolated System Testing Activities • Interoperability Testing • Testing complete application environment • May include interacting w/ Database, using Network Communications, or interacting w/ other hardware, apps, or systems if appropriate • IHE Model: ~Connectathon

  32. Test Environment Message ValidationNIST V2 Testing Tools: IHE-PCD Validation of IHE-PCD message(s) and corresponding HL7 Profile(s) Syntax and Semantic Content Validation Against HL7 V2 message (e.g., PCD-01) Message structure (e.g., MSH,PID,PV1,OBR,NTE,{{OBX},OBX,OBX,OBX,…}) Against HL7 profile (Msg_type^Event_type^ e.g., ORU^R01^…) Against HL7 and/or user provided tables Example of user provided table is RTM for Ref_IDs, Units, etc. Against ‘validation context’, including specific values Defined in XML (e.g., specific test case values)

  33. NIST V2 Testing Tools and Services Testing Validation Types Validation against ‘failure types’: VERSION*: The version in the message and in the profile should match. MESSAGE_STRUCTURE_ID*: The message type (MSH.9 element) in the profile and in the message should match. MESSAGE_STRUCTURE: The message should have a valid message structure (correct usage, correct cardinality, and correct element name). USAGE: R elements should be present; X elements should not be present in the message. CARDINALITY: Elements should be present at least the minimum times and at most the maximum times specified in the profile. It should also take into account the usage of the element (X element with a minimum of 4 should not be present in the message). LENGTH: The value of the element should have a length equal or less than the value specified in the profile. DATATYPE: For the datatype NM, DT, DTM, SI and TM, the value of the element should match the regular expression defined in the standard. DATA: The value of the element should match a constant specified in the profile, a value set specified in a table, a value or a regular expression specified in the message validation context. MESSAGE_VALIDATION_CONTEXT*: This is a user input error when the location specified in the message validation context can't be found in the message. TABLE_NOT_FOUND*: This is a user input when a table can't be found in the table files (TableProfileDocument). AMBIGUOUS_PROFILE*: The profile should not be ambiguous.

  34. Instance Testing HL7 V2Message Validation User Test Management Services Web Application Client HL7 V2 Message Validation Criteria HL7 V2 Message Validation Report HL7 V2 Message Results HL7 V2 Message Validation Report • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Test Harness (Java Code) Test Execution Conformance Testing: Showing that messages are in compliance w/ HL7 V2.x and constraints specified in IHE-PCD framework documents.

  35. Instance System Testing IHE-PCD RTM Validation User Test Management Services HL7 V2 Message RTM Validation Criteria Web Application Client Report HL7 V2 Message Validation Results RTM Message Validation Report HL7 V2 Message IHE PCD DOC Test Agent RTM Message Validation Test Harness (Java Code) Test Execution • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Conformance Testing: Showing that messages are in compliance w/ Harmonized-Rosetta terminology Time Security

  36. Future: Isolated System Testing IHE-PCD Application Functional Behavior Testing Vendor IHE-PCD DOC System Test Management Services IHE-PCD DEC DOC Test Scenario Web Application Client HL7 V2 Message Generation HL7 V2 Message Validation Results Message Validation Reports IHE-PCD DOR Test Agent IHE-PCD DOC Test Agent Test Harness (Java Code) Test Execution Report RTM Message Validation • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Conformance Testing: May include HL7 message syntax validation, RTM, and/or application functional behavior testing Time Security Router

  37. Future: Isolated System Testing IHE-PCD RTM Validation User IHE-PCD DOR System Test Management Services HL7 V2 Message HL7 V2 Message RTM Validation Criteria Web Application Client Report HL7 V2 Message Validation Results RTM Message Validation Report IHE PCD DOC Test Agent RTM Message Validation Test Harness (Java Code) HL7 V2 Message Test Execution • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Conformance Testing: Showing that messages are in compliance w/ Harmonized-Rosetta terminology Time Security

  38. Future: Peer-to-Peer System Testing IHE-PCD Application Functional Behavior Testing Vendor IHE-PCD DOC System IHE-PCD DOC System Test Management Services IHE-PCD DEC DOC Test Scenario Web Application Client HL7 V2 Message Generation HL7 V2 Message Validation Results Message Validation Reports IHE-PCD DOR Test Agent IHE-PCD DOC Test Agent Test Harness (Java Code) Test Execution Report RTM Message Validation Conformance Testing: May include HL7 message syntax validation, RTM, and/or application functional behavior testingInteroperability Testing: system(s) (functional) behavior • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Time Security Router/Logger/Proxy

  39. Next Steps… • Update NIST V2 Tool Web interface • Include pick lists for IHE-PCD files • IHE-PCD HL7 Profiles • RTM Context Validation Files • Test Case Context Validation Files • Allow uploading of multiple files at once • ER7 messages • Context Validation Files • Complete User’s Guide and distribute • Distribute NIST-produced files to Integration Profile WGs • Discuss/Review Test Guideline/Plans w/ IP WGs • Vet all documentation through IHE-PCD WGs • Tool Tutorial for IHE-PCD Members – Monday, October 19th

  40. Thank YOU! Discussion Recommend Test Plan approach to define validation criteria across Integration Profiles Recommend HL7 Profile approach To be used as baseline in each cycle going forward… Easy for new companies to get up and running… IHE-PCD profiles already integrated into NIST V2 tooling as we go forward… Decide on level of requirement by vendors (MWB and NIST V2 Tooling) Questions? Note: Tool tutorial scheduled 10/19/09 Thanks for your attention  Please visit us at: hl7v2tools.nist.gov and http://www.nist.gov/medicaldevices

  41. Extra Slides

  42. NIST Framework ImplementationTesting Flow Proposal NIST proposes developing a process flow to support HIT conformance and interoperability testing This process will help in the development of test tools providing: Test Scenarios Transactions and actors involved in the test scenarios Pre-conditions, Action/Events and post-conditions Business rules Message profile, value sets, and data Validation criteria for each transaction

  43. NIST Framework ImplementationTesting Flow ProposalHow to Detail Work? NIST proposes using templates to develop test scenarios across use cases slated for pre-, virtual- and connectathon testing NIST is working/completed examples (Pre-connectathon) and will share shortly (Tcon? And FTP site)… Have profile/use case sub-groups work in parallel on test scenarios (with Manny and NIST folks) Much like framework docs w/ actors, transactions, and details of messages (including data/value sets) Continue to develop HL7 profiles in MWB and Where possible test against MWB tool (prior to) pre- and virtual-Connectathons Use NIST test services (IHE-PCD Test Agents) for pre-, virtual- and Connectathons as developed

  44. NIST V2 Testing Tools Validating IHE-PCD Transactions (messages)What’s Needed? • Test Scenario for IHE-PCD Profiles (e.g. DEC, ACM, PIV, etc.) • HL7 Profile (typically from MWB) • MWB profile: .mwb file • MWB table: .mwt file(s) • A Validation Context (for NIST V2 Validation Tools) • Context must be provided where the user specifies the elements of the message to be validated. • Referred to as “failure types”: User-provided (XML) ‘validation context’ file in a specific format • User provided tables or databases • E.g., Medical Device Semantic Database or • By providing the values in the ‘validation context’ file… • HL7 Transactions (Messages)

More Related