500 likes | 609 Views
Implementing Survey Instruments in LOINC. Thomas White, MD, MS. Outline. History and Vision Why are Survey Instruments Challenging? Proposed Migration to an Implementation Model (“the question is the concept”) Validation Nursing LOINC => Dialogix Dialogix Instruments => LOINC
E N D
Implementing Survey Instruments in LOINC Thomas White, MD, MS Thomas White, MD, MS – NYS Office of Mental Health
Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health
History • Motivation • Web-based instruments so can track symptoms (diagnoses), clinical outcomes over time • Aggregate across instruments / variants (e.g. severity of depression) • Challenges • 100s of instruments, >10K variables; no standards • Variants may not be comparable Thomas White, MD, MS – NYS Office of Mental Health
Vision • National infrastructure for assessment instruments • Support all common instruments • Use a standardized coding system • Support multiple languages • Psychometric re-validation • Foster comparison of different ways of measuring construct Thomas White, MD, MS – NYS Office of Mental Health
Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health
Variants May Not Be Comparable Original CNYPC Thomas White, MD, MS – NYS Office of Mental Health
How Should Variants Be Coded? • Current approach • One code for all variants • Risks type I and II errors if scales differ • Safer approach • Unique code per minor variant • Reduces data sharing • Best approach • Fewest codes needed to disambiguate versions with truly different meanings or reference ranges • Catch-22 – research to compare versions requires separate codes Thomas White, MD, MS – NYS Office of Mental Health
How Are Survey Variables Defined? • Conceptual definition • Construct being assessed • Operational definition • Wording of the question • Variable definition • Available answer options • Validation criteria • Associated internal coding values Thomas White, MD, MS – NYS Office of Mental Health
What Can Modify a Variable’s Meaning or Reference Range? • How questions are posed • Wording and language • Context (order in which asked) • Presentation attributes • The method used to ask them • Likert scale, standard gamble, time trade-off, … • Who does the asking • Interviewer, self, computer, … • Who does the answering • Capability and willingness to answer truthfully Thomas White, MD, MS – NYS Office of Mental Health
Wording • Section • Headers, instructions, labels • Question • Labels, instructions • Answer option • Labels, anchors, codes Thomas White, MD, MS – NYS Office of Mental Health
Context / Ordering • Length • Waning attention or motivation • Proximity / Grouping • Neighboring questions affect expectations Thomas White, MD, MS – NYS Office of Mental Health
Presentation • Look and feel of instructions / questions • Mechanism for asking / responding • Likert scales • Lengthy lists • Standard gamble, … Thomas White, MD, MS – NYS Office of Mental Health
DISC: DiagnosticInterview Schedulefor Children More Complex Assessment Instruments ClinicalGuideline Scored Questionnaire StructuredDiagnosticInterview Thomas White, MD, MS – NYS Office of Mental Health
Sources of Instrument Complexity • Branching (order / context) • Tailoring (wording) Thomas White, MD, MS – NYS Office of Mental Health
A1 goto Q2 Q1 A2 goto Q3 A3 (implicit goto Q2) Q2 Q3 Branching: Traditional Solutions Rely on Post-conditioned Gotos Response-Level Branching (e.g. Perseus SurveySolutions) Hidden-Branching Nodes(e.g. NovaResearch’s QDS, Sawtooth Ci3) Q1 Hidden branching node if (exp) goto Q3 Q2 Thomas White, MD, MS – NYS Office of Mental Health
Relevance–Based Branching (e.g. Dialogix, ?XForms?) no Q1: Q1.relevant? yes Ask Q1 Store N/A Q2: Q2.relevant? Ask Q2 Example: name: 1 What is the subject’s name? gender: 1 What is [name]’s gender? age: 1 How old is [name]? menarche: age > 9 && gender == ‘female’ Has [name] had her first period? LMP: menarche == ‘true’ When was [name]’s last period? sexual: 1 Is [name] sexually active? Relevance-based Branching Logic Thomas White, MD, MS – NYS Office of Mental Health
A. SI DICE SI, ¿Hubo varias semanas seguidas en las que se sintió nervioso(a) o asustado(a) por estar separado(a) de [useted/su [FIGURA DE APEGO]] o fuera de su casa? Tailoring:Dynamically Composed Questions Some [children/young people] get very upset or nervous when they are not with their parents or with the grown-ups who usually look after them. I am going to ask you some questions about that. 1. In the last year – that is, since [[NAME EVENT]/[NAME CURRENT MONTH] of last year] – was there a time when [CHILD’S NAME] often wanted to stay at home and not go (to [school/work] or other) places without [you/[ATTACHMENT FIGURE]]? • A. IF YES, Where there several weeks in a row when [he/she] seemed nervous or afraid about being away from [you/[ATTACHMENT FIGURE]] or away from home? Thomas White, MD, MS – NYS Office of Mental Health
Static vs. Dynamic Features • Instrument Definition (Static) • Wording & Presentation – untailored text • Order – branching logic • Instrument Use (Dynamic) • Tailored content • Actual order of questions seen • Time spent reading and answering • Who does the asking (face-to-face, self, …) • Modality used to ask it (Likert, standard gamble, …) Thomas White, MD, MS – NYS Office of Mental Health
Implications for Coding Assessment Instruments • Meaning and reference range are: • Determined by operational and variable definitions • Altered by wording, order, presentation, assessment method • Further altered by dynamic tailoring, language, and actual order asked • Without separate codes, risk of type I and II errors • Need explicit modeling to compare variants Thomas White, MD, MS – NYS Office of Mental Health
Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health
Current LOINC Schema for Nursing Instruments (1) Thomas White, MD, MS – NYS Office of Mental Health
Current LOINC Schema for Nursing Instruments (2) Thomas White, MD, MS – NYS Office of Mental Health
Current LOINC Schema – Limitations and Recommendations • Assumes one version of each instrument • Alter Method to create hierarchy of variants • Inadequate modeling of operational and variable definitions • Create formal XML model for these • No modeling of instrument branching logic • Extend LOINC Panel to codify instruments • No support for dynamic features • Can annotate answers/values with tailoring, branching, and timing information Thomas White, MD, MS – NYS Office of Mental Health
Dialogix As Possible Solution • Syntax and schema for defining instruments • Each node represents a single measurable entity or communicates an important message • Engine for deploying them • Data Schema for storing results • Records order, logs backtracking and changed answers, stores fully tailored text and time spent / method used to answer questions. Thomas White, MD, MS – NYS Office of Mental Health
Sample Dialogix Instrument Thomas White, MD, MS – NYS Office of Mental Health
Dialogix Instrument Syntax Thomas White, MD, MS – NYS Office of Mental Health
Dialogix Data Logs • Data (per answer given) • localName, answer, timestamp, sequence, tailored content, language used • Events (for each screen-full of questions) • Server, network, and browser processing time • Per-question keystroke and mouse events and timing Thomas White, MD, MS – NYS Office of Mental Health
Four Recommendations for LOINC • Alter Method naming schema to disambiguate variants • Add Assessment_Instruments table • Migrate existing nursing instruments • Store instrument branching logic in LOINC Panel • Post-coordinate dynamic features with value portion of name-value pairs Thomas White, MD, MS – NYS Office of Mental Health
(1) Alter Method Schema • [Observed|Reported].[Inst name].[Variant identifier] • Variant Identifier refers to entry in Assessment_Instruments table • Hierarchies • Root level has unspecified (or unknown method) • Levels below root specify details – initially one level deep (no known relationship between versions) • Re-arrange hierarchy as psychometric studies show similarity / differences among variants Thomas White, MD, MS – NYS Office of Mental Health
(2) Assessment_Instruments Table • One row per measurable entity • Foreign Keys pointing to LOINC entry to text and LOINC Panel which codifies instrument • New Dialogix fields to fully codify instrument • Re-purpose Existing LOINC Columns • SurveyQuestionText => ActionPhase • AnswerList => AnswerOptions Thomas White, MD, MS – NYS Office of Mental Health
Sample Concept Hierarchy Thomas White, MD, MS – NYS Office of Mental Health
Sample Question – BPRS Thomas White, MD, MS – NYS Office of Mental Health
Specifying Validation Criteria Thomas White, MD, MS – NYS Office of Mental Health
Simple Calculations – Anxiety and Depression Subscale of BPRS Thomas White, MD, MS – NYS Office of Mental Health
Complex Calculations: List of DSM-IV Diagnoses Thomas White, MD, MS – NYS Office of Mental Health
Multilingual Support – XML Model Thomas White, MD, MS – NYS Office of Mental Health
Multilingual – Translator’s View Thomas White, MD, MS – NYS Office of Mental Health
(3) Store Instrument Sequence Information in LOINC Panel • Give each instrument variant its own Panel • Could use existing Dialogix XML schema • Issues • Consider using GLIF and Arden for metadata • W3C XForms, VoiceXML, and P3P schemas • Possible many-to-many relationship between questions and instruments • What about dummy variables? • What about module-level presentation features? Thomas White, MD, MS – NYS Office of Mental Health
Dialogix Schema – Prototype for LOINC Panel Thomas White, MD, MS – NYS Office of Mental Health
(4) Post-Coordinate Dynamic Features • Use ANO field within OBX of HL7 • Might use Dialogix data model • Should consider using a standard schema, like those in click-stream analysis. • Consider normalizing to XML-Event model Thomas White, MD, MS – NYS Office of Mental Health
Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health
Columns in Assessment Instruments Table • One row per measurable entity • Instrument • Sequence • Concept • LocalName • SourceName • Relevance • ActionType • ActionPhrase • AnswerOptions Thomas White, MD, MS – NYS Office of Mental Health
Validation: Existing Instruments LOINC => Dialogix • Concept = Component; Relevance = 1; ActionType = q • SourceName = Survey_Quest_Src • LocalName ~= Source Name or “q” + Sequence • ActionPhrase = Best of: • Survey_Quest_Text (e.g. LIV-HIV) • Comments (HHCC) • Component (Omaha, QAM, HIV-SSC) • Component: System (Omaha, reflexes in HHCC) • AnswerOptions = Best of AnswerList and Exmpl_Answers, using Dialogix’s syntax Thomas White, MD, MS – NYS Office of Mental Health
Review Nursing Instruments in LOINC + Dialogix Thomas White, MD, MS – NYS Office of Mental Health
Validation: New InstrumentsDialogix => LOINC (1) • Diagnostic Instruments, with Scoring • AUDIT, BPRS, BASIS-32, SF-36 • Structured Interviews • DISC, SCID-II • Decision Trees • GAF-Tree • Clinical Guidelines • NYC DOH TB Testing & Treatment Guideline • Healthcare Education • AutoMEQ Thomas White, MD, MS – NYS Office of Mental Health
Validation: New InstrumentsDialogix => LOINC (2) • Most Axes Can Be Derived From Dialogix Schema • Property = FIND for questions; IMP for eqns • System = ^Patient (some exceptions) • Scale_Type = AnswerOptions and DataType • Method_Typ = [Observed|Reported].Instrument • Class = Survey.[Domain].Instrument • Some Axes Require Manual Review • Concept • Time_Aspect • Sometimes System, Method_Typ Thomas White, MD, MS – NYS Office of Mental Health
Review Dialogix Instruments Newly Added to LOINC Thomas White, MD, MS – NYS Office of Mental Health
Some Issues From Dialogix => LOINC • Modeling global and modular instruction? • Should any them be named? • Modeling equations – which should be named? • Add Date, Month, and Day to Scale? • Time_Aspect – add Lifetime when asking whether a diagnosis / symptom ever occurred? • Mapping local values to standard ones for dates, demographics, diseases, etc. • Supporting multiple languages Thomas White, MD, MS – NYS Office of Mental Health
Conclusions • Instrument variants may need different codes. • Proposed LOINC extensions can • Model complex instruments • Disambiguate versions • Facilitate meta- and psychometric analysis • Operationalize instruments • Facilitate translation and ongoing QA • Additional modeling work needed for LOINC Panel and HL7 ANO attachments. Thomas White, MD, MS – NYS Office of Mental Health
Thank You Thomas White, MD, MS – NYS Office of Mental Health