1 / 35

Medical Device Test Effort NIST Team Members

IHE-PCD , HL7 HC Dev WG, ISO/IEEE 11073, and NIST Medical Device Communication and IHE-PCD Cycle 4 Test Strategy IHE-PCD, HL7, ISO/IEEE Joint WG Meetings (@ Atlanta, GA) 23 September 2009. Medical Device Test Effort NIST Team Members. John Garguilo ( john.garguilo@nist.gov ,

hoshi
Download Presentation

Medical Device Test Effort NIST Team Members

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IHE-PCD , HL7 HC Dev WG,ISO/IEEE 11073, andNISTMedical Device Communication andIHE-PCD Cycle 4 Test StrategyIHE-PCD, HL7, ISO/IEEE Joint WG Meetings (@ Atlanta, GA)23 September 2009

  2. Medical Device Test EffortNIST Team Members • John Garguilo (john.garguilo@nist.gov, 301-975-5248) • Sandra Martinez (sandra.martinez@nist.gov, 301-975-3579) • Maria Cherkaoui (maria.cherkaoui@nist.gov Guest Researcher) • *Jing Gao(jing.gao@nist.gov Guest Researcher) * Jing joined our project in May ’09 Project Web site: www.nist.gov/medicaldevices

  3. NIST MDC Testing Topics • IHE-PCD Test Strategy • NIST Framework for Testing Distributed Healthcare Applications • IHE-PCD Testing Framework Instance • Cycle 4 Pre-connectathon (and virtual-connectathon) • Test Plan • Support both Conformance & Interoperability • RTM Terminology Management System Update • NIST ISO/IEEE 11073 Tooling • ICSGenerator, ValidatePDU • NIST ISO/IEEE 11073 Standards Development Work • MDC X73 XML Schema (X73-P10202)

  4. Test Environments • Instance Testing • Conformance (e.g., against HL7 2.x or CDA) • Implementation conforms to Spec. on which it is based • IHE Model: ~Virtual and Pre-Connectathon • Isolated System Testing • Includes Instance Testing Activities • Protocol Conformance • Functional Behavior Conformance • Features and Operational behavior correspond to Specs. • IHE Model: ~Virtual and Pre-Connectathon • Peer-to-Peer System Testing • Includes Isolated System Testing Activities • Interoperability Testing • Testing complete application environment • May include interacting w/ Database, using Network Communications, or interacting w/ other hardware, apps, or systems if appropriate • IHE Model: ~Connectathon

  5. Test Environment Message ValidationNIST V2 Testing Tools: IHE-PCD Validation of IHE-PCD message(s) and corresponding HL7 Profile(s) Syntax and Semantic Content Validation Against HL7 V2 message (e.g., PCD-01) Message structure (e.g., MSH,PID,PV1,OBR,NTE,{{OBX},OBX,OBX,OBX,…}) Against HL7 profile (Msg_type^Event_type^ e.g., ORU^R01^…) Against HL7 and/or user provided tables Example of user provided table is RTM for Ref_IDs, Units, etc. Against ‘validation context’, including specific values Defined in XML (e.g., specific test case values)

  6. NIST V2 Testing Tools and Services Testing Validation Types Validation against ‘failure types’: VERSION*: The version in the message and in the profile should match. MESSAGE_STRUCTURE_ID*: The message type (MSH.9 element) in the profile and in the message should match. MESSAGE_STRUCTURE: The message should have a valid message structure (correct usage, correct cardinality, and correct element name). USAGE: R elements should be present; X elements should not be present in the message. CARDINALITY: Elements should be present at least the minimum times and at most the maximum times specified in the profile. It should also take into account the usage of the element (X element with a minimum of 4 should not be present in the message). LENGTH: The value of the element should have a length equal or less than the value specified in the profile. DATATYPE: For the datatype NM, DT, DTM, SI and TM, the value of the element should match the regular expression defined in the standard. DATA: The value of the element should match a constant specified in the profile, a value set specified in a table, a value or a regular expression specified in the message validation context. MESSAGE_VALIDATION_CONTEXT*: This is a user input error when the location specified in the message validation context can't be found in the message. TABLE_NOT_FOUND*: This is a user input when a table can't be found in the table files (TableProfileDocument). AMBIGUOUS_PROFILE*: The profile should not be ambiguous.

  7. Instance Testing HL7 V2Message Validation User Test Management Services Web Application Client HL7 V2 Message Validation Criteria HL7 V2 Message Validation Report HL7 V2 Message Results HL7 V2 Message Validation Report • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Test Harness (Java Code) Test Execution Conformance Testing: Showing that messages are in compliance w/ HL7 V2.x and constraints specified in IHE-PCD framework documents.

  8. Instance System Testing IHE-PCD RTM Validation User Test Management Services HL7 V2 Message RTM Validation Criteria Web Application Client Report HL7 V2 Message Validation Results RTM Message Validation Report HL7 V2 Message IHE PCD DOC Test Agent RTM Message Validation Test Harness (Java Code) Test Execution • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Conformance Testing: Showing that messages are in compliance w/ Harmonized-Rosetta terminology Time Security

  9. Future: Isolated System Testing IHE-PCD Application Functional Behavior Testing Vendor IHE-PCD DOC System Test Management Services IHE-PCD DEC DOC Test Scenario Web Application Client HL7 V2 Message Generation HL7 V2 Message Validation Results Message Validation Reports IHE-PCD DOR Test Agent IHE-PCD DOC Test Agent Test Harness (Java Code) Test Execution Report RTM Message Validation • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Conformance Testing: May include HL7 message syntax validation, RTM, and/or application functional behavior testing Time Security Router

  10. Future: Isolated System Testing IHE-PCD RTM Validation User IHE-PCD DOR System Test Management Services HL7 V2 Message HL7 V2 Message RTM Validation Criteria Web Application Client Report HL7 V2 Message Validation Results RTM Message Validation Report IHE PCD DOC Test Agent RTM Message Validation Test Harness (Java Code) HL7 V2 Message Test Execution • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Conformance Testing: Showing that messages are in compliance w/ Harmonized-Rosetta terminology Time Security

  11. Future: Peer-to-Peer System Testing IHE-PCD Application Functional Behavior Testing Vendor IHE-PCD DOC System IHE-PCD DOC System Test Management Services IHE-PCD DEC DOC Test Scenario Web Application Client HL7 V2 Message Generation HL7 V2 Message Validation Results Message Validation Reports IHE-PCD DOR Test Agent IHE-PCD DOC Test Agent Test Harness (Java Code) Test Execution Report RTM Message Validation Conformance Testing: May include HL7 message syntax validation, RTM, and/or application functional behavior testingInteroperability Testing: system(s) (functional) behavior • Test Artifacts • Conformance Profiles • HL7 Tables • Validation Context Files • Generation Context Files Time Security Router/Logger/Proxy

  12. HL7 Message Validation StudyExample PIV (PCD-03)

  13. PIV-PCD-03 Test Case 60101Validation using the IHE TF/Supplements • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  14. PIV-PCD-03 Test Case 60101Validation Using HL7 Profile Tables • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  15. PIV-PCD-03 Test Case 60101hRTM Validation • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  16. PIV-PCD-03 Test Case 60101Use/Test Case Specific Validation • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  17. PIV-PCD-03 Test Case 60101Overall Validation • MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 • PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L • ORC|RE|12345|||||||||||||||||N60101 • RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC • RXR|IV||IVP • OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

  18. PIV-PCD-03 Test Case 60101Validation Coverage • Coverage % = Number of Values in Message / Number of Values Checked • IHE Supplement Validation43% • Validation using Profile Tables 24% • Test Case Specific Validation 19% • Overall Validation 74% • Overall Validation without profile 62% • Note: remaining 24% (from overall validation percentage) is validated at the message structure level via HL7; these include message segments that are not of primary interest to the IHE-PCD. If this level of validation is considered, then nearly 100% validation coverage can be expected.

  19. Message Validation ContextTF/Supplement Validation • DEC-PCD-01-ValidationContext.xml • SPD-PCD-02-ValidationContext.xml • PIV-PCD-03-ValidationContext.xml • ACM-PCD-04-ValidationContext.xml • IDCO-PCD-09-ValidationContext.xml

  20. Message Validation ContexthRTM Validation • DEC-PCD-01-ValidationContext.xml • PIV-PCD-03-ValidationContext.xml • ACM-PCD-04-ValidationContext.xml • Needed: IDCO-PCD-09ValidationContext.xml

  21. Message Validation ContextTest Case Specific Validation • PIV(for PCD-03) • 60101-ValidationContext.xml • 60102-ValidationContext.xml • 60103-ValidationContext.xml • 60104-ValidationContext.xml • 60105-ValidationContext.xml • 60106-ValidationContext.xml • 60107-ValidationContext.xml • 60108-ValidationContext.xml • 60109-1-ValidationContext.xml • 60109-2-ValidationContext.xml • 60109-3-ValidationContext.xml • 60109-4-ValidationContext.xml • 60110-ValidationContext.xml

  22. IHE-PCD ’09-’10 Pre-Connectathon NIST Support (see recommendation document: "NIST Support for IHE-2009v1.docx") • Primary Goal:Provide tools to test conformance to HL7 and IHE-PCD Integration Profiles such as DEC, DEC-SPD, ACM, PIV, IDCO, and RTM • Instance Testing • System Under Test (SUT)  Test Agent (TA) • HL7 syntax validation (basic validation) • IHE-PCD profile syntax • IHE-PCD profile semantic validation • RTM validation • Test case specific validation • Develop Test Plan across IHE-PCD Integration Profiles • Test Scenarios • Interaction diagrams • Pre-conditions • Validation criteria – all test-able requirements (assertions) • Test cases • Validation context file(s)

  23. IHE-PCD ’09-’10 Pre-ConnectathonContinued Proposed conformance testing for ’09-’10 Pre-Connectathon • All DOR messages (PCD-01) sent during DOR testing. • All IOP messages (PCD-03) sent during IOP testing. • All IDCO-DOR messages (PCD-09) sent during IDCO testing. • All DOF messages (PCD-01) sent during DOF testing. • All DOC messages (PCD-02) sent during DOF testing. • All AR messages (PCD-04) sent during AR testing.

  24. Show NIST V2 Tools (via Web) Demonstration NIST HL7 Web • Upload HL7 IHE-PCD ORU^RO1 profile • Upload HL7 IHE-PCD context validation file (XML) • Upload HL7 IHE-PCD message (ER7 format) • Validate • Generate and review resulting ‘Report’ • Review errors

  25. RTM Terminology Management System Update

  26. RTM Mgmt System Changes(from last WG mtgs in May) Features Completed • Enhanced lookup of REFIDs (description of REFID’s) and provided an interface to propose new terms • Implemented an interface for browsing terms in X73 Nomenclature

  27. RTMMS In progress and next steps… Features based on the Rosetta Supplement • Incorporate “Enumerations” management capability(ies) • Add ranking capabilities of implementation priority of valid terms in the Rosetta table • Implement Rosetta validation against hRTM • Add ability to edit hRTM table and save changes • Implement hRTM version control and management system 

  28. RTMMSFuture New Enhancements Features based on the new/recent enhancements to HRTM • Assign dimensions to the units • Implement Rosetta rules • Implement generation of hRTM based on Rosetta rules • Generate comment table based on the Recommended Rosetta Comment Submittal Format • Implement RCH table User oriented features • Regular Expression based column filtering • Implement “change trailing” capabilities • To identify occurred changes, time they were made, users who made them… • Enhance registration process • Incorporate enhanced X73 Nomenclature database 

  29. NIST ISO/IEEE 11073 Tooling Update

  30. X73 NIST Tooling… • ICSGenerator • User Interface Work • Met w/ Todd Cooper, captured issues, made changes • Updated issues list – used as basis for task list • Integrating database into ICSGenerator… • Lite-weight database identified • To include hRTM… • Initial discussions • X73-compliant Device Containment • Generation of partial OBX segments • On horizon: MDC Semantic Database • Terminology • Nomenclature • Medical Device Profiles and other Test Artifacts

  31. ISO/IEEE 11073 Standards Work Update

  32. ISO/IEEE 11073 Standards Work • MDC X73 XML Schema (X73-P10202) • Verification and validation of XML Schema continues via tool usage • Support for Personal Health Device Working Group • X73-20601 • 104xx device specializations • Continue to validate via tooling prior to ballot • Report errors for correction prior to balloting

  33. Thank YOU! Discussion Propose “ORU^R01^ORU^Rxx” naming convention for IHE-PCD HL7 profiles xx denotes profile specific for Integration profile/device class (use of DEC/PCD-01) Questions? Thanks for your attention  Please visit us at: http://www.nist.gov/medicaldevices

  34. Extra Slides

  35. User Facilitator Monitor IHE-PCD DOC System NIST HIT Testing InfrastructureA Framework for Building Test Systems Test Management Services Test Description Validation Generation Results Test Data Test Agent Test Harness Test Execution Evaluation Agent Aggregated Service Test Artifacts Time Report Security Other Services Specialized Services Router/Logger/Proxy

More Related