Implementing Survey Instruments in LOINC

1. Implementing Survey Instruments in LOINC Thomas White, MD, MS Thomas White, MD, MS – NYS Office of Mental Health

2. Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health

3. History • Motivation • Web-based instruments so can track symptoms (diagnoses), clinical outcomes over time • Aggregate across instruments / variants (e.g. severity of depression) • Challenges • 100s of instruments, >10K variables; no standards • Variants may not be comparable Thomas White, MD, MS – NYS Office of Mental Health

4. Vision • National infrastructure for assessment instruments • Support all common instruments • Use a standardized coding system • Support multiple languages • Psychometric re-validation • Foster comparison of different ways of measuring construct Thomas White, MD, MS – NYS Office of Mental Health

5. Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health

6. Variants May Not Be Comparable Original CNYPC Thomas White, MD, MS – NYS Office of Mental Health

7. How Should Variants Be Coded? • Current approach • One code for all variants • Risks type I and II errors if scales differ • Safer approach • Unique code per minor variant • Reduces data sharing • Best approach • Fewest codes needed to disambiguate versions with truly different meanings or reference ranges • Catch-22 – research to compare versions requires separate codes Thomas White, MD, MS – NYS Office of Mental Health

8. How Are Survey Variables Defined? • Conceptual definition • Construct being assessed • Operational definition • Wording of the question • Variable definition • Available answer options • Validation criteria • Associated internal coding values Thomas White, MD, MS – NYS Office of Mental Health

9. What Can Modify a Variable’s Meaning or Reference Range? • How questions are posed • Wording and language • Context (order in which asked) • Presentation attributes • The method used to ask them • Likert scale, standard gamble, time trade-off, … • Who does the asking • Interviewer, self, computer, … • Who does the answering • Capability and willingness to answer truthfully Thomas White, MD, MS – NYS Office of Mental Health

10. Wording • Section • Headers, instructions, labels • Question • Labels, instructions • Answer option • Labels, anchors, codes Thomas White, MD, MS – NYS Office of Mental Health

11. Context / Ordering • Length • Waning attention or motivation • Proximity / Grouping • Neighboring questions affect expectations Thomas White, MD, MS – NYS Office of Mental Health

12. Presentation • Look and feel of instructions / questions • Mechanism for asking / responding • Likert scales • Lengthy lists • Standard gamble, … Thomas White, MD, MS – NYS Office of Mental Health

13. DISC: DiagnosticInterview Schedulefor Children More Complex Assessment Instruments ClinicalGuideline Scored Questionnaire StructuredDiagnosticInterview Thomas White, MD, MS – NYS Office of Mental Health

14. Sources of Instrument Complexity • Branching (order / context) • Tailoring (wording) Thomas White, MD, MS – NYS Office of Mental Health

15. A1 goto Q2 Q1 A2 goto Q3 A3 (implicit goto Q2) Q2 Q3 Branching: Traditional Solutions Rely on Post-conditioned Gotos Response-Level Branching (e.g. Perseus SurveySolutions) Hidden-Branching Nodes(e.g. NovaResearch’s QDS, Sawtooth Ci3) Q1 Hidden branching node if (exp) goto Q3 Q2 Thomas White, MD, MS – NYS Office of Mental Health

16. Relevance–Based Branching (e.g. Dialogix, ?XForms?) no Q1: Q1.relevant? yes Ask Q1 Store N/A Q2: Q2.relevant? Ask Q2 Example: name: 1 What is the subject’s name? gender: 1 What is [name]’s gender? age: 1 How old is [name]? menarche: age > 9 && gender == ‘female’ Has [name] had her first period? LMP: menarche == ‘true’ When was [name]’s last period? sexual: 1 Is [name] sexually active? Relevance-based Branching Logic Thomas White, MD, MS – NYS Office of Mental Health

17. A. SI DICE SI, ¿Hubo varias semanas seguidas en las que se sintió nervioso(a) o asustado(a) por estar separado(a) de [useted/su [FIGURA DE APEGO]] o fuera de su casa? Tailoring:Dynamically Composed Questions Some [children/young people] get very upset or nervous when they are not with their parents or with the grown-ups who usually look after them. I am going to ask you some questions about that. 1. In the last year – that is, since [[NAME EVENT]/[NAME CURRENT MONTH] of last year] – was there a time when [CHILD’S NAME] often wanted to stay at home and not go (to [school/work] or other) places without [you/[ATTACHMENT FIGURE]]? • A. IF YES, Where there several weeks in a row when [he/she] seemed nervous or afraid about being away from [you/[ATTACHMENT FIGURE]] or away from home? Thomas White, MD, MS – NYS Office of Mental Health

18. Static vs. Dynamic Features • Instrument Definition (Static) • Wording & Presentation – untailored text • Order – branching logic • Instrument Use (Dynamic) • Tailored content • Actual order of questions seen • Time spent reading and answering • Who does the asking (face-to-face, self, …) • Modality used to ask it (Likert, standard gamble, …) Thomas White, MD, MS – NYS Office of Mental Health

19. Implications for Coding Assessment Instruments • Meaning and reference range are: • Determined by operational and variable definitions • Altered by wording, order, presentation, assessment method • Further altered by dynamic tailoring, language, and actual order asked • Without separate codes, risk of type I and II errors • Need explicit modeling to compare variants Thomas White, MD, MS – NYS Office of Mental Health

20. Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health

21. Current LOINC Schema for Nursing Instruments (1) Thomas White, MD, MS – NYS Office of Mental Health

22. Current LOINC Schema for Nursing Instruments (2) Thomas White, MD, MS – NYS Office of Mental Health

23. Current LOINC Schema – Limitations and Recommendations • Assumes one version of each instrument • Alter Method to create hierarchy of variants • Inadequate modeling of operational and variable definitions • Create formal XML model for these • No modeling of instrument branching logic • Extend LOINC Panel to codify instruments • No support for dynamic features • Can annotate answers/values with tailoring, branching, and timing information Thomas White, MD, MS – NYS Office of Mental Health

24. Dialogix As Possible Solution • Syntax and schema for defining instruments • Each node represents a single measurable entity or communicates an important message • Engine for deploying them • Data Schema for storing results • Records order, logs backtracking and changed answers, stores fully tailored text and time spent / method used to answer questions. Thomas White, MD, MS – NYS Office of Mental Health

25. Sample Dialogix Instrument Thomas White, MD, MS – NYS Office of Mental Health

26. Dialogix Instrument Syntax Thomas White, MD, MS – NYS Office of Mental Health

27. Dialogix Data Logs • Data (per answer given) • localName, answer, timestamp, sequence, tailored content, language used • Events (for each screen-full of questions) • Server, network, and browser processing time • Per-question keystroke and mouse events and timing Thomas White, MD, MS – NYS Office of Mental Health

28. Four Recommendations for LOINC • Alter Method naming schema to disambiguate variants • Add Assessment_Instruments table • Migrate existing nursing instruments • Store instrument branching logic in LOINC Panel • Post-coordinate dynamic features with value portion of name-value pairs Thomas White, MD, MS – NYS Office of Mental Health

29. (1) Alter Method Schema • [Observed|Reported].[Inst name].[Variant identifier] • Variant Identifier refers to entry in Assessment_Instruments table • Hierarchies • Root level has unspecified (or unknown method) • Levels below root specify details – initially one level deep (no known relationship between versions) • Re-arrange hierarchy as psychometric studies show similarity / differences among variants Thomas White, MD, MS – NYS Office of Mental Health

30. (2) Assessment_Instruments Table • One row per measurable entity • Foreign Keys pointing to LOINC entry to text and LOINC Panel which codifies instrument • New Dialogix fields to fully codify instrument • Re-purpose Existing LOINC Columns • SurveyQuestionText => ActionPhase • AnswerList => AnswerOptions Thomas White, MD, MS – NYS Office of Mental Health

31. Sample Concept Hierarchy Thomas White, MD, MS – NYS Office of Mental Health

32. Sample Question – BPRS Thomas White, MD, MS – NYS Office of Mental Health

33. Specifying Validation Criteria Thomas White, MD, MS – NYS Office of Mental Health

34. Simple Calculations – Anxiety and Depression Subscale of BPRS Thomas White, MD, MS – NYS Office of Mental Health

35. Complex Calculations: List of DSM-IV Diagnoses Thomas White, MD, MS – NYS Office of Mental Health

36. Multilingual Support – XML Model Thomas White, MD, MS – NYS Office of Mental Health

37. Multilingual – Translator’s View Thomas White, MD, MS – NYS Office of Mental Health

38. (3) Store Instrument Sequence Information in LOINC Panel • Give each instrument variant its own Panel • Could use existing Dialogix XML schema • Issues • Consider using GLIF and Arden for metadata • W3C XForms, VoiceXML, and P3P schemas • Possible many-to-many relationship between questions and instruments • What about dummy variables? • What about module-level presentation features? Thomas White, MD, MS – NYS Office of Mental Health

39. Dialogix Schema – Prototype for LOINC Panel Thomas White, MD, MS – NYS Office of Mental Health

40. (4) Post-Coordinate Dynamic Features • Use ANO field within OBX of HL7 • Might use Dialogix data model • Should consider using a standard schema, like those in click-stream analysis. • Consider normalizing to XML-Event model Thomas White, MD, MS – NYS Office of Mental Health

41. Outline • History and Vision • Why are Survey Instruments Challenging? • Proposed Migration to an Implementation Model (“the question is the concept”) • Validation • Nursing LOINC => Dialogix • Dialogix Instruments => LOINC • Lessons Learned • Next Steps Thomas White, MD, MS – NYS Office of Mental Health

42. Columns in Assessment Instruments Table • One row per measurable entity • Instrument • Sequence • Concept • LocalName • SourceName • Relevance • ActionType • ActionPhrase • AnswerOptions Thomas White, MD, MS – NYS Office of Mental Health

43. Validation: Existing Instruments LOINC => Dialogix • Concept = Component; Relevance = 1; ActionType = q • SourceName = Survey_Quest_Src • LocalName ~= Source Name or “q” + Sequence • ActionPhrase = Best of: • Survey_Quest_Text (e.g. LIV-HIV) • Comments (HHCC) • Component (Omaha, QAM, HIV-SSC) • Component: System (Omaha, reflexes in HHCC) • AnswerOptions = Best of AnswerList and Exmpl_Answers, using Dialogix’s syntax Thomas White, MD, MS – NYS Office of Mental Health

44. Review Nursing Instruments in LOINC + Dialogix Thomas White, MD, MS – NYS Office of Mental Health

45. Validation: New InstrumentsDialogix => LOINC (1) • Diagnostic Instruments, with Scoring • AUDIT, BPRS, BASIS-32, SF-36 • Structured Interviews • DISC, SCID-II • Decision Trees • GAF-Tree • Clinical Guidelines • NYC DOH TB Testing & Treatment Guideline • Healthcare Education • AutoMEQ Thomas White, MD, MS – NYS Office of Mental Health

46. Validation: New InstrumentsDialogix => LOINC (2) • Most Axes Can Be Derived From Dialogix Schema • Property = FIND for questions; IMP for eqns • System = ^Patient (some exceptions) • Scale_Type = AnswerOptions and DataType • Method_Typ = [Observed|Reported].Instrument • Class = Survey.[Domain].Instrument • Some Axes Require Manual Review • Concept • Time_Aspect • Sometimes System, Method_Typ Thomas White, MD, MS – NYS Office of Mental Health

47. Review Dialogix Instruments Newly Added to LOINC Thomas White, MD, MS – NYS Office of Mental Health

48. Some Issues From Dialogix => LOINC • Modeling global and modular instruction? • Should any them be named? • Modeling equations – which should be named? • Add Date, Month, and Day to Scale? • Time_Aspect – add Lifetime when asking whether a diagnosis / symptom ever occurred? • Mapping local values to standard ones for dates, demographics, diseases, etc. • Supporting multiple languages Thomas White, MD, MS – NYS Office of Mental Health

49. Conclusions • Instrument variants may need different codes. • Proposed LOINC extensions can • Model complex instruments • Disambiguate versions • Facilitate meta- and psychometric analysis • Operationalize instruments • Facilitate translation and ongoing QA • Additional modeling work needed for LOINC Panel and HL7 ANO attachments. Thomas White, MD, MS – NYS Office of Mental Health

50. Thank You Thomas White, MD, MS – NYS Office of Mental Health