INFO-I642 (Clinical Decision Support Systems)

1. INFO-I642 (Clinical Decision Support Systems) Knowledge Generation Lecture #4

2. Lecture in a Nutshell • Evidence-Based Knowledge • Meta-Analysis & Systematic Reviews • Methodology • Protocols • Research Question • Literature Search • Data Extraction • Assessment • Data Combination • Issues in Meta-Analysis • Publication Bias • Large Trial vs. MA of Small Trials • Updating • Meta-Analysis Use and Access • Knowledge Acquisition (KA) • Introduction • Theoretical Basis • Cognitive Task Analysis • Knowledge Elicitation Methods • Data Analysis Methods • Representational Methods • Computer Based Knowledge Acquisition • Knowledge Generation (KG) • Introduction • Supervised Learning Models • Decision Trees • Logistic Regression • Artificial Neural Network • Nearest Neighbor • Evaluation • Unsupervised Learning Models • Cluster Analysis • Examples

3. Knowledge Acquisition (KA)

4. Introduction • Experts have: • The factual knowledge (what) that is required to solve the relevant problems and • The judgmental knowledge (why) that gets to the heart of a problem • Why Knowledge Acquisition (KA)? • Knowledge preservation we lose it once the expert retires or leaves • Knowledge sharing  can be reused in training programs • Knowledge to form the basis for decision aids  practitioners make better decisions • Knowledge that reveals underlying skills  reveals underlying strategies and skills • KA  the process of identifying and eliciting knowledge from existing sources—from domain experts, from documents, or inferred from large datasets—and subsequently encoding that knowledge so that it can be verified and validated. • Knowledge-Base in CDSS • Conceptual or Factual Knowledge  potential findings and diagnoses • Procedural Knowledge guidelines or algorithms used to operate on this knowledge • Strategic Knowledge knowledge structure with application logic used to apply these guidelines and algorithms to the underlying conceptual structure

5. Introduction cont. The classical view of knowledge engineering (top) vs interactive transfer via a computer model (bottom)

6. Theoretical Basis • Areas of research in psychology  what distinguishes outstanding individuals in a domain from less outstanding individuals and to characterize the development of expertise. • Classification of levels of expertise: • Beginner  has only everyday, lay knowledge  typical patient • Novice  has begun to acquire the prerequisite knowledge  medical student • Intermediate  above the beginner level but below the subexpert  senior medical student or a junior resident • Subexpert  possesses generic knowledge and experience that exceeds that of an intermediate but lacks specialized knowledge of the medical subdomain in question • Expert has specialized knowledge of the subdomain in addition to broad generic knowledge  Cardiologist • Intermediate effect degradation in performance as a subject moves from novice to expert  lacking the connections in knowledge pieces • Experts  superior perception of patterns; they learn many ‘‘heuristics’’ or rules of thumb; can skip steps; have meta-cognetive awareness;

7. Cognitive Task Analysis • Cognitive Task Analysis (CTA) general approach that cognitive scientists use in analyzing the basis for human performance to capture the way the mind works (cognition): • Knowledge Elicitation Methods • KA is often complex and resource intensive. • The source of the knowledge to be elicited  Experts  assumption that they are individuals with sufficient domain knowledge, interest in participating in the KA process, and minimal bias. • Collecting knowledge from multiple experts (adv > disadv) • Straightforward interview techniques is mostly used. Observing experts in real world limits introspective opinions  think-aloud protocols. • Ethnography evaluations  minimizing bias/intervention from knowledge engineer and maximizing the role of collecting information in context • Disadvantage of interview or observation  time-intensive and less quantitative data • Group techniques: brainstorming, nominal group studies, presentation discovery and …  consensus-based knowledge  better than individual

8. Cognitive Task Analysis cont. • Biases in Logical and Probabilistic Reasoning  poor estimation of probabilities by human beings + bias in their probabilistic reasoning • Bias in probabilistic reasoning  • (a) cognitive availability (recency) of information affects frequency, • (b) anchor judgments on initial estimates • (c) assess the likelihood of an event based on familiarity than objective frequency • (d) overestimate the frequency of rare events • Knowledge engineers should avoid the use of probabilistic judgments • Bias in laboratory observations  • A tendency to assign undue weight to the first evidence obtained • Over-reliance on variables that have taken on extreme values • The tendency to seek evidence that confirms the current hypothesis • The tendency to reason about only one or two hypotheses at a time • The tendency to be overconfident • The desire to maintain consistency (devaluing important information) • Belief in illusory correlations • The tendency to be overly conservative

9. Cognitive Task Analysis cont. • Data Analysis Methods • Protocol and Discourse Analysis • Elicitation of knowledge from individuals while they are engaged in problem-solving or reasoning tasks (e.g. think aloud)  determine the conceptual entities and relationships between them used by individuals while they reason • Experts generate high-level hypothesis which partitions problems in manageable units while subexperts keep generating hypotheses mostly at a lower level without evaluating them • Discourse analysis  the process by which an individual’s intended meaning within a body of text or some other form of narrative discourse is analyzed into discrete units of thought (propositions) and the subsequent analysis of the contexts in which those units appear (propositional relations in semantic structures), as well as the quantification and description of the relationships existing between those same units • Concept (Mapping) Analysis • The modern idea of concept map can be interpreted as a ‘‘user-friendly’’ expression of meaning in a text • The researchers help the domain practitioners build representation on their domain knowledge, merging the activities of knowledge elicitation and representation

10. Cognitive Task Analysis cont. • Verification and Validation of Knowledge Acquisition • Ideally applied throughout the entire knowledge engineering spectrum • Verification  the evaluation of the perceived requirements of the end users or application domain (consistency and completeness) • Validation  the evaluation of whether that system meets the real-world requirements • Heuristic Methods • The most commonly used approach to evaluating knowledge is the use of heuristic evaluation criteria (manually reviewing knowledge-base and determining its consistence with heuristics)  they are difficult to automate. • Representational Methods • Abstract frameworks that assume particular types of knowledge structures • Verbal Data  Propositional Representations and Semantic Networks • Propositional Representation  explicitly represents and classifies the embedded ideas and their relationships in a discourse  a proposition is denoted as a relation (predicate) over a set of arguments (concepts)  nodes and links  semantic net.

11. Cognitive Task Analysis cont. Semantic analysis of a clinical text. In the diagram, solid rectangles indicate cues from the text, broken lines indicate diagnostic hypotheses, and arrows indicate directionality of relations. COND: conditional relation, CAU: causal relation, RSLT: resultive relation. In this case, the text is taken from an explanation protocol provided by a psychiatrist who had been challenged by a case from the field of cardiology: ‘‘The patient has been reacting to stress likely by his injecting a drug (or drugs), which has resulted in tachycardia, a fall in blood pressure, and elevated temperature. These findings are due to the toxic reaction caused by the injected drugs. He is in or near shock. The flame-shaped hemorrhage may represent a sequel of an upsurge in blood pressure possibly as a result of his injection of drugs.’’

12. Computer Based KA • Large knowledge-bases are impractical to manage manually • Knowledge engineers typically begin with the creation of a basic ontology for a field (e.g. Protégé System) and then build inferential structures and relationships that allow a knowledge system to draw conclusions and generate advice (e.g. MYCIN) • Teiresias the creation of a notion known as rule models, metalevel descriptions of the knowledge categories and relationships that characterized the domain in question and guided the interactive knowledge acquisition process • Gradual move toward creating powerful knowledge authoring and editing tools that could be used by knowledge engineers after they had elicited the pertinent knowledge from human experts  OPAL (from ONCOIN) • Today it is rare to have a knowledge elicitation tool that is designed and successfully implemented for use directly by physicians

13. Computer Based KA cont. A mixed-initiative knowledge-acquisition dialog between MYCIN and an infectious disease expert

14. Computer Based KA cont. The expert indicates what corrections need to be made and is able to verify that the revised rule is what was intended

15. Knowledge Generation (KG)

16. Introduction • Most clinical decision support systems in current use do not learn from data and still rely on the rule-based paradigm: • Data are simply not available or not structured enough to allow knowledge to be ‘‘learned’’ from them • Techniques to discover patterns from data are not well disseminated or not well evaluated in the biomedical community • Nonprobabilistic rules that are defined by experts are more clearly understandable by clinicians • Simple understandable models (e.g., linear and logistic regression, score systems) have far outweighed in number and utilization the more sophisticated machine learning models (e.g., support vector machines, neural networks), many of which remain limited to research applications. • Statistical and machine learning pattern recognition algorithms have been in existence for several decades recognize regularities in data and construct a model that can be utilized in new cases.

17. Introduction cont. • Data mining techniques are pattern recognition techniques intended to find correlations and relationships in the plethora of data  convert data into meaningful information  description (find) and prediction. • Data mining tools do not require a priori knowledge on the part of the decision maker (i.e. non-knowledge based) compared to rule-based systems (i.e. knowledge-based) • Supervised learning models classify objects with predefined labels representing classes of interest using the data at hand • Unsupervised learning models are not based on predefined classifications, and are used frequently for exploratory data analyses

18. Supervised Learning Models • Supervised learning models determine how to best classify objects with predefined labels representing classes of interest (e.g., malignant versus benign cases) using the data at hand • Requires training of the models with training sets • Establishes a relationship or predictive model between the dependent and independent variables • A model is called a classification model if the target variable is discrete; and a regression model if the target variable is continuous • Mapping of data items into one of the predefined classes. • A Priori Probability Problem the training set/sample may not represent the general population • Sample approaches/models are decision trees, logistic regression, neural networks, and nearest neighbor approach

19. Supervised Learning Models cont. • Decision (Classification) Trees • Easiest to understand and the most widely used method  adopts a top-down strategy in searching for a solution • Each node is a classification question and the branches of the tree are partitions of the data set into different classes • Recursively and univariately partition cases into two subgroups. A simple decision tree with the tests on attributes A and B.

20. Supervised Learning Models cont. Classification tree for the bivariate outcome problem. Cases are recursively partitioned according to the attribute-value pair that best divides the cases into ‘‘euthyroid’’ or not. The resulting partitions easily can be visualized in this simplified two dimensional problem.

21. Supervised Learning Models cont. Computer-Derived Decision Tree for the Classification of Patients with Acute Chest Pain

22. Supervised Learning Models cont. • Logistic Regression • Models data in which the target or dependent variable is binary, i.e., the dependent variable can take the value 1 with a probability of success p, or the value 0 with the probability of failure 1 − p. • Rather than predicting the values of the dependent variable, logistic regression estimates the probability that a dependent variable will have a given value Logistic regression model

23. Supervised Learning Models cont. Without variable transformation, logistic regression will not work for all cases because the problem is not linearly separable.

24. Supervised Learning Models cont. • Artificial Neural Networks (ANN) • Inspired by the way the brain recognizes patterns • ANN can be supervised or unsupervised • In supervised neural networks, examples in the form of the training data are provided to the network one at a time  if the output of the neural network is the same as the actual value  no further training is required • ANN could work on the training set but not generalize to broader samples Neural network

25. Supervised Learning Models cont. Artificial neural network with a hidden layer of nodes. For didactic purposes, activation functions in this example correspond to step functions that define partitions similar to the ones in the classification tree.

26. Supervised Learning Models cont. • There is no theoretical advantage of using ANNs over logistic regression in binary classification problems unless the ANNs have a hidden layer of nonlinear neurons • Nearest Neighbor (NN) Classifier • NN rule assumes that observations which are the closest together (based on some form of measurement) belong to the same category • The NN rule is often used in situations where the user has no knowledge of the distribution of the categories. • k-nearest neighbor approach (KNN)  k-neighboring points when classifying a data point  works very well in cases where a class does not form a single coherent group but is a collection of more than one separate group. • The supervised (K)NN uses a training set to identify a set of classes and then uses those classes in testing sets  KNN can be used for pattern recognition and classification in medical imaging (e.g. melanoma)

27. Supervised Learning Models cont. KNN NN Nearest neighbor (NN) classifier. There are two classes: A (triangles) and B (diamonds). The circle represents the unknown sample, X. For the NN rule, the nearest neighbor of X comes from class A, so it would be labeled class A. Using the k-NN rule with k = 4, three of the nearest neighbors of sample X come from class B, so it would be labeled as B.

28. Supervised Learning Models cont. • Evaluation of Supervised Models • Discrimination assesses how well the models can potentially discriminate positive and negative cases. Area under ROC curve can be used. • Calibration assesses how close the model’s estimated probability is to the ‘‘true’’ underlying probability of the outcome of interest. • In logistic regression models, calibration is usually assessed by the Hosmer-Lemeshow goodness-of-fit test. • Poor calibration is often caused by limited representation of the population to which models will be applied at the model construction phase • Receiver Operating Characteristic (ROC) Graph • In pattern recognition the goal is to map entities to classes classification accuracy is compared among methods • Classification  true positive, false positive, true negative, false negative • Sensitivity = TP / TP + FN  TP Rate = sensitivity • Specificity = TN / TN + FP  FP Rate = 1 - specificity

29. Supervised Learning Models cont. • The ROC graph is a two-dimensional graph that depicts the trade-offs between benefits (true positive rate) and costs (false positive rate). Benefit Liberal Perfect Conservative Poor No benefit or Type I error Cost or Type II error

30. Supervised Learning Models cont. Plotting the ROC point for each possible threshold value results in a curve.

31. Supervised Learning Models cont. • The area under the ROC curve (AUC) provides a single statistic (the C-Statistic) for comparing classifiers  accuracy of the classifiers. • The value of AUC ranges from 0.5 (flipping a coin) to 1 (best class). Calculation can be based on trapezoids under the curve.

32. Unsupervised Learning Models • Unsupervised learning models are not based on predefined classifications, and are used frequently for exploratory data analyses in domains in which knowledge is sparse • No information is available as to how to group the data into more meaningful classes  the objective is to unveil hidden patterns in the data that were not previously anticipated (a posteriori) • Cluster Analysis • Clustering  a general term to describe methodologies that are designed to find natural groupings or clusters based on measured or perceived similarities among the items in the clusters (no need to identify groups) • Clustering provides a starting point for exploring further relationships • Clustering methods: hierarchical and nonhierarchical • The selection of an appropriate measure of similarity (density versus feature) to define clusters is a major challenge in cluster analysis

33. Unsupervised Learning Models cont. Two clusters of data (left); three clusters (right) using the same set of data

34. Examples • Prognosis of ICU Mortality • The Acute Physiology and Chronic Health Evaluation series (APACHE I, II and III)  widely used logistic regression-based predictive models • These models remain useful in research, but limitations in calibration and across disparate patient populations have restricted use in some clinical situations. • Simplified Acute Physiologic Score SAPS-II and Mortality Prediction Model MPM-II  more used in Europe • Cardiovascular Disease Risk • Framingham cohort  estimates of the risk of developing future heart disease in patients (logistic regression methods) • One of the limitations of the cohort was the lack of racial diversity • Prognosis in Interventional Cardiology • Logistic regression prediction models for post-procedural mortality following angioplasty

35. Evidence-Based Knowledge

36. Systematic Review and Meta-Analysis • Evidence-Based Medicine (EBM)  formalizes the principles and methods of reviewing and synthesizing evidence that have been developing over several decades. • Systematic Review (SR): comprehensive, rigorous, and unbiased review and synthesis of up-to-date evidence provides the most reliable information to inform health practice. • Meta-Analysis (MA): a systematic review that uses statistical methods to combine results across several studies to address specific questions • Meta-Analysis Example  A meta-analysis combining 33 studies found a highly statistically significant result of approximately 20 percent reduction of overall mortality by the adoption of streptokinase therapy; however, FDA did not approve its use till two larger studies showed the same results in 1988. Thousands of lives could have been saved meanwhile!

37. Systematic Review and Meta-Analysis cont. Standard forest plot (left panel) and a cumulative meta-analysis (right panel) of intravenous streptokinase therapy for acute myocardial infarction;

38. Methodology • A systematic review differs from traditional narrative review  it follows a well-defined protocol to identify, appraise, and synthesize the available evidence to minimize bias and to arrive at reliable conclusions • SR Protocol  SRs and MAs generally are retrospective analyses of published data. The protocol should clearly describe the specific research question(s), literature search strategy, selection criteria, approach to critical appraisal of the studies, methods of statistical analyses, and interpretation of the results • Research Question  Formulating the research question is the most critical step. PICO (Population, Intervention, Comparator, Outcome) can be used to guide the search. • Example: ‘‘What drugs should be used to treat hypertension?’’ should be reformatted and refined (based on PICO) to something such as “How does nifedipine compare with hydrochlorothiazide in patients with moderately elevated diastolic blood pressure (100–110mmHg) in clinical trials evaluating long-term (one-year or more) mortality and morbidity?’’

39. Methodology cont. • A now common use of MAs is to provide data for decision analyses (e.g. probabilities in decision trees) • Developing an analytic framework (sometimes called evidence model or causal pathway) can be very useful in some settings to help formulate research questions  used by Evidence-based Practice Centers (EPCs) • Literature Review SR should be comprehensive and all relevant literature should be reviewed. • The literature search generally begins with a search of the MEDLINE database because it is free, readily available electronically, and it indexes over 4,000 biomedical journals • Many authors also search the Cochrane Central Register of Controlled Trials, which has indexed over 400,000 controlled trials • The importance of including non-English language articles in a systematic review is also likely to be topic-dependent

40. Methodology cont. • Data Extraction Data needed for analyses in systematic reviews and meta-analyses must be extracted from the original studies • Considerable skill and experience are needed to ensure the reliability of the data extraction process (e.g. more than one individual to reduce inter-rater bias) • Data reported across studies often are not standardized, and important information is often missing. • The need for subjective judgment in the collection of data potentially contributes bias to the systematic review process • Quality Assessment  SRs should use high quality studies. • Various methods have been proposed to assess the methodological quality of a study, which include checklists of specific study elements, and numerical quality scores based on some schemes of weighting of elements believed to contribute to quality • Guidelines, such as the CONSORT statement, have been published to improve the conduct and reporting of future clinical trials  similar approaches can be developed for systematic reviews

41. Methodology cont. • Data Combination MA is a systematic review in which the reviewers have decided that sufficient data are available, from studies meeting inclusion criteria, to address a specific question, and that it is reasonable to combine them to provide an overall answer. • The common form of meta-analysis aims to determine an overall weighted average of the effect size, by combining data using a fixed-effect or a random-effects model • The fixed-effect model assumes that all studies are estimating a single true value  large studies and studies with more events tend to receive the most weights • The random-effects method incorporates both the fixed-effect weight and the between-study weight due to heterogeneity of results across studies  it distributes the weight more evenly (i.e., small studies receive more weight) across studies when there is heterogeneity across studies. • Fixed-effect model or a random-effects model fail when significant heterogeneity of treatment effect among trials is present  Subgroup analysis and meta-regression could be used to explore heterogeneity across studies • Meta-Regression limitations  difficulty with aggregated values such as age and gender; ecological bias; and lack of consistently reported covariates in clinical trials

42. MA Issues • Publication Bias  unpublished studies with negative results threaten the validity of a meta-analysis • The inverted funnel plot is the most popular method used to detect publication bias  small studies have greater variability (wide end of funnel) and larger studies have smaller variability (narrow neck of funnel)  missing publications will show an asymmetric funnel suggesting publication bias Large size studies Small size studies

43. MA Issues cont. • Large Trial vs MA of Small Size Trials  Large clinical trials often are considered as the definitive last word in clinical evidence. • Study 1  Disagreements between large trials and the corresponding meta-analysis of small trials occur in about 10 to 30 percent of the comparisons. The high rate of discrepancy raises the question of validity of meta-analyses! • Study 2  disagreements among large trials were just as common as disagreement between results of large trials and meta-analyses of small trials  heterogeneity across clinical trials (same topic) is very common, regardless of study size. • The interpretation of the results of a meta-analysis should be made with respect to all factors that might affect the results • Updating MA and SR  The need to routinely update meta-analyses has been amply demonstrated. • With the publication of new research findings, the conclusions of these reviews may change and their usefulness may become obsolete • The Cochrane Library is the only entity that has a built-in mechanism to routinely update published systematic reviews.

44. MA Issues cont. Factors that contribute to the observed treatment effects in a randomized controlled trial.

45. MA Issues cont. Estimated treatment effect in a meta-analysis

46. MA Uses and Access • MA is being used not only clinical medicine but also other fields such as dentistry, nursing, genetics and etc • Types of studies that have been synthesized in these reviews included randomized controlled trials, cohort studies, case-control studies, as well as case reports. • Compared with meta-analyses of interventions, which number in the thousands, there are only several hundred meta-analyses of diagnostic tests • Meta-analyses of associations of factors with health outcomes (e.g. second-hand smoking and cancer) should be interpreted with greater care compared to randomized trials (due to cofounders) • Meta-analyses published in journals indexed in MEDLINE or other major electronic databases (e.g., EMBASE) can readily be identified by using the term “meta-analysis”  even included in MeSH  however, SR is not included in MeSH • The Cochrane Library currently represents the single most comprehensive source of high quality systematic reviews that are routinely updated • AHRQ  EBM Program for SR (http://www.ahrq.gov/clinic/epc) and CPG repository at the National Guideline Clearinghouse (NGC) (www.guidelines.gov)

47. Summary • Evidence-Based Knowledge • Meta-Analysis & Systematic Reviews • Methodology • Protocols • Research Question • Literature Search • Data Extraction • Assessment • Data Combination • Issues in Meta-Analysis • Publication Bias • Large Trial vs. MA of Small Trials • Updating • Meta-Analysis Use and Access • Knowledge Acquisition (KA) • Introduction • Theoretical Basis • Cognitive Task Analysis • Knowledge Elicitation Methods • Data Analysis Methods • Representational Methods • Computer Based Knowledge Acquisition • Knowledge Generation (KG) • Introduction • Supervised Learning Models • Decision Trees • Logistic Regression • Artificial Neural Network • Nearest Neighbor • Evaluation • Unsupervised Learning Models • Cluster Analysis • Examples