Werner CEUSTERS, MD

1. Seminar on the Role of Ontologies in Clinical Medicine Assessment instruments and biomedical reality:examples in the pain domainJune 13, 2012 – Ramada Inn, Buffalo NY Werner CEUSTERS, MD Center of Excellence in Bioinformatics and Life Sciences, Ontology Research Group and Department of Psychiatry University at Buffalo, NY, USA http://www.org.buffalo.edu/RTU

2. Presentation overview • Context of the work: OPMQoL • Background on assessment instrument • How Ontological Realism and Referent Tracking might improve the design and use of assessment instruments

3. OPMQoL

4. Context of this work • OPMQoL: Ontology for Pain-related disablement, Mental health and Quality of Life, • Funded by grant 1R01DE021917-01A1 from the National Institute of Dental and Craniofacial Research (NIDCR). The content of this presentation is solely the responsibility of the author and does not necessarily represent the official views of the NIDCR or the National Institutes of Health.

5. Collaborators Werner Ceusters – Richard Ohrbach UB (PIs) Mike T. John – Eric L. Schiffman University of Minnesota Vishar Aggarwal Manchester, UK Joanna Zakrzewska London, UK Thomas List Malmö, Sweden Rafael Benoliel Hadassah, Israel

6. Project goals • to obtain better insight into: • the complexity of pain disorders, pain types as well as pain-related disablement and • its association with mental health and quality of life, • to develop an ontology for this subdomain incorporating a broad array of measures consistent with a biopsychosocial perspective regarding pain, • to integrate five existing datasets that broadly encompass the major types of pain in the oral and associated regions.

7. Considered datasets • ‘US Dataset’ (724 patients) resulted from the NIH funded RDC/TMD Validation Project, • ‘Hadassah Dataset’ (306 patients) from the Orofacial Pain Clinic at the Faculty of Dentistry, Hadassah, • ‘German Dataset’ (416 patients) of patients seeking treatment for orofacial pain at the Department of Prosthodontics and Materials Sciences, University of Leipzig, • ‘Swedish Dataset’of 46 consecutive Atypical Odontalgia (AO) patients recruited from 4 orofacial pain clinics in Sweden as well as data about age- and gender-matched control patients, • ‘UK Dataset’ (168 patients) of facial pain of non dental origin present for a minimum of three months.

8. Most important challenge • The data sets cover more or less the same domain, but … • the data within each data set are collected independently from each other, with distinct, partially overlapping data collection and data organization tools.

9. Tools used If not used for data collection and organization, these sources can be used post hoc to document, and perhaps increase, the level of data clarity and faithfulness in and comparability of existing data collections.

10. Ontologies and data collections: simplified view Linking the variables of distinct data collections to a realism-based ontology.

11. Linking data organization tools Terminology component used for uses 1..* uses 1 0..* terminology 1..* assessment instrument uses 1..* used for 1..* used in 0..* term used for used-in 0..* 1 1 data dictionary expressed-by uses 0..* explained-in 0..* Data component uses 1 1..* uses used-for 1 1..* 1 used-for assessment instrument ontology data collection means 1 1..* broader explains concept 1..* uses 1 1..* data item data collection ontology expresses 0..1 narrower 1..* 1..* application ontology used for uses uses used-for representational artifact bridging axiom corresponds-to 0..* 1..* 1..* 1 0..* ontology 1 denotes denoted by 1..* entity denotator Ontology component 0..* 1 reference ontology

12. Linking data collections using distinct assessment instruments

13. Assessment instruments

14. McDowell, Ian & Newell, ClaireMeasuring health a guide to rating scales and questionnaires Free 2006 copy: http://a4ebm.org/sites/default/files/MeasuringHealth.pdf

15. Assessment instrument design strategy • Identify the (human) feature of interest, • Select questions answers to which provide a means to quantify the presence in a subject, • Design a scoring algorithm, • Identify baselines, • Assess the quality of the instrument towards its design objectives.

16. Example: Perceived Stress Scale (PSS) Cohen, S., Kamarck, T., and Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 386-396.

17. Perceived Stress Scale: score calculation • In the last month, how often have you been upset because of something that happened unexpectedly?........................................................................................................................................... 0 1 2 3 4 0 1 2 3 4 • In the last month, how often have you felt that you were unable to control the important things in your life?........................................................................................................................................................... 0 1 2 3 4 0 1 2 3 4 • In the last month, how often have you felt nervous and “stressed”? ......................................................... 0 1 2 3 4 0 1 2 3 4 • In the last month, how often have you felt confident about your ability to handle your personal problems?.................................................................................................................................................. 0 1 2 3 44 3 2 1 0 • In the last month, how often have you felt that things were going your way?............................................. 0 1 2 3 44 3 2 1 0 • In the last month, how often have you found that you could not cope with all the things that you had to do? .................................................................................................................................................................. 0 1 2 3 4 0 1 2 3 4 • In the last month, how often have you been able to control irritations in your life?..................................... 0 1 2 3 44 3 2 1 0 • In the last month, how often have you felt that you were on top of things?................................................ 0 1 2 3 44 3 2 1 0 • In the last month, how often have you been angered because of things that were outside of your control? .................................................................................................................................................................. 0 1 2 3 4 0 1 2 3 4 • In the last month, how often have you felt difficulties were piling up so high that you could not overcome them?........................................................................................................................................................ 0 1 2 3 4 0 1 2 3 4 ∑ Cohen, S., Kamarck, T., and Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 386-396.

18. Perceived Stress Scale: norm table Cohen, S. and Williamson, G. Perceived Stress in a Probability Sample of the United States. Spacapan, S. and Oskamp, S. (Eds.) The Social Psychology of Health. Newbury Park, CA: Sage, 1988.

19. Quality of instruments • Validity: • how well does the instrument measure what it is intended to measure?  property of the instrument • what can one conclude given a certain score  property of the interpretation • Reliability (consistency): • how well reproduces the instrument the same results?

20. Validity versus reliability • This instrument measures very reliably but has poor validity; • However, it might be measuring something else ! • Assessing validity – especially in absence of a gold standard – is a difficult and error-prone procedure driven by phenomenological and statistical considerations.

21. Construct validity • ‘For variables such as pain, quality of life, or happiness, gold standards do not exist and thus validity testing is more challenging. • For such abstract constructs, validation of a measurement involves a series of steps known as “construct validation.” • This begins with a conceptual definition of the topic (or construct) to be measured, indicating the internal structure of its components and the way it relates to other constructs. • These may be expressed as hypotheses indicating, for example, what correlations should be obtained between a quality of life scale and a measure of depression, or which respondents should score higher or lower on quality of life. • None of these challenges alone proves validity and each suffers logical and practical limitations, although when systematically applied, they build a composite picture of the adequacy of the measurement.’ Ian McDowell. Measuring Health: A Guide to Rating Scales and Questionnaires, Third Edition. Oxford University Press 2006; p34

22. Evidence for validity of PSS • Higher PSS scores were associated with (for example): • failure to quit smoking, • failure among diabetics to control blood sugar levels, • greater vulnerability to stressful life-event-elicited depressive symptoms, • more colds.

23. Epistemic value commitments for constructs • ‘values involved in making and advancing epistemologically-relevant claims, such as scientific ones’: Coherence Consistency Comprehensiveness Fecundity Simplicity Instrumental efficacy Originality Relevance Precision JZ. Sadler. Epistemic Value Commitments in the Debate over Categorical vs. Dimensional Personality Diagnosis. Philosophy, Psychiatry, & Psychology 3.3 (1996) 203-222

24. Many caveats • f.i.: way questions are phrased creates bias or confusion; • e.g.: “In the last month, how often have you been upset because of something that happened unexpectedly?..................................... 0 1 2 3 4” • ‘0’ can mean: • nothing unexpectedly happen • there were unexpected events but none caused being upset

25. Construct validity – additional notes • ‘For variables such as pain, quality of life, or happiness, gold standards do not exist and thus validity testing is more challenging. • For such abstract constructs, validation of a measurement involves a series of steps known as “construct validation.” • This begins with a conceptual definition of the topic (or construct) to be measured, indicating the internal structure of its components and the way it relates to other constructs. • These may be expressed as hypotheses indicating, for example, what correlations should be obtained between a quality of life scale and a measure of depression, or which respondents should score higher or lower on quality of life. • None of these challenges alone proves validity and each suffers logical and practical limitations, although when systematically applied, they build a composite picture of the adequacy of the measurement.’ Ian McDowell. Measuring Health: A Guide to Rating Scales and Questionnaires, Third Edition. Oxford University Press 2006; p34

26. Hypothesis networks of constructs represent correlations in reality

27. But: are we discovering or inventing disorders?

28. Real correspondence to entities in reality? denotes? denotes? denotes?

29. Hypothesis • Ontological Realism and Referent Tracking hold – once again – a key to a solution

30. Methodology

31. Remember: Assessment instrument design • Identify the (human) feature of interest, • Select questions answers to which provide a means to quantify the presence in a subject, • Design a scoring algorithm, • Identify baselines, • Assess the quality of the instrument towards its design objectives.

32. Starting point: IASP definition for ‘pain’ • ‘an unpleasant sensory and emotional experience associated with actual or potential tissue damage, or described in terms of such damage’; • what asserts: • a common phenomenology (‘unpleasant sensory and emotional experience’) to all instances of pain, • the recognition of three distinct subtypes of pain involving, respectively: • actual tissue damage, • what is called ‘potential tissue damage’, and • a description involving reference to tissue damage.

33. Thus rather: five pain-related phenomena! Smith B, Ceusters W, Goldberg LJ, Ohrbach R. Towards an Ontology of Pain. In: Mitsu Okada (ed.), Proceedings of the Conference on Logic and Ontology, Tokyo: Keio University Press, February 2011:23-32.

34. Linking the instruments and other tools • analyze data dictionaries, assessment instruments, study criteria and corresponding terminologies, • build realism-based application ontologies to link these sources to realism-based reference ontologies.

35. Example: assessing TMJ Anatomy

36. Panoramic X-ray of mouth

37. Radiology RDC/TMD Examination: data collection sheet

38. RDC/TMD: a collaborator’s data dictionary Fieldnames in that collaborator’s data collection Allowed values for the fields

39. Anybody sees something disturbing ?

40. This data dictionary alone is not reliable! That these variables are about the condylar head of the TMJ is ‘lost in translation’!

41. ‘meaning’ of values in data collections ‘The patient with patient identifier ‘PtID4’ is stated to have had a panoramic X-ray of the mouth which is interpreted to show subcortical sclerosis of that patient’s condylar head of the right temporomandibular joint’ meaning 1

42. Objectives of the ‘sources’ analysis • Find for each value V in the data collections all possible configurations of entities (according to our best scientific understanding) for which the following can be true: • V • ‘it is stated that V’ • Describe these possible configurations by means of sentences from a formal language that mimic the structure of reality.

43. Objectives of the ‘sources’ analysis (2) • For example, • for the value stating that ‘The patient with patient identifier ‘PtID4’ has had a panoramic X-ray of the mouth which is interpreted to show subcortical sclerosis of that patient’s condylar head of the right temporomandibular joint’ to be true, • this statement must have been made, • for the statement to be true, there must have been that patient, an X-ray, etc, … • BUT! It is not necessarily true that that patient has indeed the sclerosis as diagnosed.

44. Methodology • Formulate for each variable in the data collection a sentence explaining as accurately as possible what the variable stands for, • list the entities in reality that the terms in the sentence denote, • list recursively for all entities listed further entities that ontologically must exist for the entity under scrutiny to exist, • classify all entities in terms of realism-based ontologies (RBO), • specify all obtaining relationships between these entities, • outline all possible configurations of such entities for the sentence to be true.

45. RBO (1): Ontology of General Medical Science produces bears realized_in etiological process disorder disease pathological process produces diagnosis interpretive process signs & symptoms abnormal bodily features produces participates_in recognized_as Scheuermann R, Ceusters W, Smith B. Toward an Ontological Treatment of Disease and Diagnosis. 2009 AMIA Summit on Translational Bioinformatics, San Francisco, California, March 15-17, 2009;: 116-120. http://www.referent-tracking.com/RTU/sendfile/?file=AMIA-0075-T2009.pdf http://code.google.com/p/ogms/

46. No conflation of diagnosis, disease, and disorder The diagnosis is here The disorder is there The disease is there

47. RBO (2): (cleaned up) Ontology of Biomedical Investigations

48. Step 1: formulate a statement ‘The patient with patient identifier ‘PtID4’ is stated to have had a panoramic X-ray of the mouth which is interpreted to show subcortical sclerosis of that patient’s condylar head of the right temporomandibular joint’ meaning 1

49. Step 2 (1): list the entities denoted • 1(The patient) with 2(patient identifier ‘PtID4’)3(is stated)4(have had) a 5(panoramic X-ray) of 6(the mouth) which 7(is interpreted) to 8(show)9(subcortical sclerosis of 10(that patient’s condylar head of the 11(right temporomandibular joint)))’ notes: colors have no meaning here, just provide easy reference, this first list can be different, any such differences being resolved in step 3

50. Step 2 (2): provide directly referential descriptions