1 / 38

Course overview, the diagnostic process, and measures of interobserver agreement

Course overview, the diagnostic process, and measures of interobserver agreement. Thomas B. Newman, MD, MPH September 24, 2009. Announcement. The Giants have a day game next week (10/1), starting at 12:45 PM Avoid driving and parking here. Overview. UC Faculty walkout Administrative stuff

ozzie
Download Presentation

Course overview, the diagnostic process, and measures of interobserver agreement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 24, 2009

  2. Announcement • The Giants have a day game next week (10/1), starting at 12:45 PM • Avoid driving and parking here

  3. Overview • UC Faculty walkout • Administrative stuff • The diagnostic process • Interobserver agreement • Continuous variables • Categorical variables • Concordance • Kappa • Regular • Weighted

  4. UC Faculty Walkout • The trigger • Underlying agenda • Deeper underlying problem

  5. The Trigger • UC Regents granted President Yudof emergency powers, over objections of Council of UC Faculty • He implemented furlough plan • Academic senates of all 10 campuses voted that furloughs take place on instructional days • Rejected by Chancellors and UCOP, citing UC’s “paramount teaching mission” • One response:Paramount?“Perhaps President Yudof could identify a single University of California faculty member who has attained tenure solely by the excellence of his or her teaching.” From UCB Professor Catherine Cole, Open letter to students. (http://berkeleycuts.org/?p=21)

  6. Underlying agenda? • “Privatize” UC • Governor’s (and his predecessors’) vision: University education as a private, rather than a public good • Therefore, shift costs to students, wealthy donors and corporations • Data: • State spending per student down 40% from $15,860 in 1990 to $9560 today* • UC Undergrad Tuition $700/yr (1972-6), $3600/yr (~2000), ~$10,302 (Proposed 2009-10)** • Medical School $800 (1980); $10,177 (2001); $26,114 (2009) *Saving the University of California. http://api.ning.com/files/*chnzbv3Cp*Ax3IGthxsbwTyNPFU*rD-Flhu*zEpnRQ9xVqEUcY6wyyf7lm-qB4dZgXar3EY2HzL-OoxWfmH30qxra2ZDLFi/UC_EducationalHandout.pdf **Asimov N. UC president recommends huge tuition increases. SF Chronicle 9/11/09 http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2009/09/11/MNAB19L7Q9.DTL

  7. Deeper underlying problem • Increasing internalization in the US (spreading to other countries) of market economy values as opposed to a sense of community • Lack of universal health care is a symptom • This is reflected in the practice of medicine (focus on individual vs public health interventions) and at UCSF • “You eat what you kill.” --Lee Goldman, Former Chair of Medicine, UCSF • Evidence that income

  8. Recommended reading • The Kept University (Atlantic Monthly article from 2000; link on course website) • Money and the Changing Culture of American Medicine. P Hartzband and J Groopman, NEJM 2009;360:101-3 • Books by David Korten, Rabbi Michael Lerner, Robert Reich • Sermons on TN’s website

  9. Administrative stuff • Introductions • Basic structure of course • New material each week in lecture • Read material before lecture if possible • HW on that material due the following week in section • Exceptions: • No class on Thanksgiving • Penultimate class 12/3 – Chapter 12 (Challenges for EBD) and course review: pass out take-home exam; no HW on Ch 12 • Last lecture 12/10: review of take-home exam • Lectures: mixture of PPT and Whiteboard

  10. SECTIONS • Section assignments: Click ROSTER on Epi 204 website • Section rooms: Click SCHEDULE on website • Faculty will rotate; students, rooms and TA's will be constant for the quarter

  11. Homework • Required – key way of learning material • Bring the homework to section on paper • Not graded if late, but can still be turned in; answers on web and in the book • Will be graded by section leaders and returned the following week • Which problems are assigned will be announced each week in SECTION and (later) posted on web

  12. Homework Problems • Issues • Many of our best problems are in the book, with answers • Excess of pediatrics and emergency medicine examples • Solutions: • We will assign some of them anyway. Write our your answer before reading ours • We’re writing new problems all the time • We may ask you to help us create problems from your own fields!

  13. Getting help • Classmates, then section leaders, then faculty • Questions, ambiguous/confusing problems – send e-mail to section leader or me • Unless you indicate otherwise, we will assume we can cc: the whole class when we respond if we think the question is of general interest

  14. Books • Essential: TBN and MAK’s “Evidence-based Diagnosis” (Cambridge University Press, 2009) • Other texts listed on the web • Copies of other books are in the bookstore and on reserve in the library and available for browsing here

  15. Grading, honor code, etc. • Worst HW score dropped; all other HW count equally • Grade based on HW and take home final (50:50) • Write out answers to problems in your own words before looking at the back of the book • Acknowledge help from others and use of the answers • Do NOT collaborate on the final exam • Honor code taken seriously

  16. Diagnostic process • Why do we want to assign a name to this person’s illness? • Different reasons lead to different classification schemes

  17. Examples • Acute nephrotic syndrome • Acute leukemia • Attention deficit disorder • Dysuria worth a course of antibiotics • SLUBI=Self-limited undiagnosed benign illness

  18. Simplified Generic Decision Problem • Patient either has the disease or not • If D+, net benefit of treatment • If D, better not to treat • (“Treat” could just mean doing more tests)

  19. Simplifying assumptions (often wrong) • Test results are dichotomous • Most tests have more than two possible answers • Disease states are dichotomous • Many diseases occur on a spectrum • There are many kinds of “nondisease”

  20. Evaluating diagnostic tests • Reliability • Accuracy • Usefulness • Today we do reliability

  21. Types of variables • Categorical • Dichotomous – 2 values • Nominal – no intrinsic ordering • Ordinal – intrinsic ordering • Continuous (infinite number of values) vs Discrete (limited number)

  22. Measuring interobserver agreement for categorical variables What is agreement?

  23. Concordance • What percent of the time do the 2 observers agree (exactly) • Advantage: easy to understand • Disadvantage: may be misleading if observers agree on prevalence of abnormality and it is high or low

  24. Concordance problem • What is agreement? • Are you impressed?

  25. Definition of Kappa • The amount of agreement beyond what would be expected from the marginals* • Formula: • Practice • Obs = 90%, Exp = 80%, K = • Obs = 70%, Exp = 60%, K = • Obs = 60%, Exp = 70%, K = Observed agreement – Expected agreement 1 – Expected agreement *Row and column totals

  26. Calculation of Expected Agreement from Marginals

  27. Why does multiplying row total by column total and dividing by N give you the expected agreement?

  28. Calculation of Kappa

  29. Kappa for variables with more than two values • Calculate observed and expected agreement the same way • Calculate Kappa the same way • If the variable is ordinal use weighted kappa

  30. What is going on here? Look for lack of balance above and below diagonal Results when observers have different thresholds Unbalanced Disagreement

  31. GCS Eye opening- Observed

  32. GCS Eye Opening: Expected 17 x 78/116 = 1326/116 = 11.4

  33. Weighted Kappa • Weighted kappa • Linear • Quadratic – “penalty” for being wrong is squared • Custom

  34. Real-life illustration: Rating of neurological examination • Types of weights, Stata illustration . tab ex1 ex2 . kap ex1 ex2, w(w) . kap ex1 ex2, w(w2) • (See Appendix 2.1)

  35. What does observed Kappa depend upon? • How well people agree • SPECTRUM within classifications • E.g., are the abnormal ones VERY abnormal? • Difficult cases can be excluded or over-sampled • PREVALENCE of classifications by the various observers (and whether they agree on prevalence) • Chance (random error; people can get lucky/unlucky) • Weighting scheme used

  36. Extra slides

  37. TICR Professional Conduct Statement • I will maintain the highest standards of academic honesty • I will neither give nor receive aid in examinations or assignments unless such cooperation is expressly permitted by the instructor • I will conduct research in an unbiased manner, reports results truthfully, and credit ideas developed and work done by others • I will not use answer keys from prior years • I will write answers in my own words, and, when collaboration is permitted, acknowledge collaborators when answers are jointly formulated

More Related