1 / 56

Evaluation of surveillance systems

Evaluation of surveillance systems. 17th EPIET Introductory Course Lazareto, Menorca, Spain September – October, 2011. Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada. Günter Pfaff. The surveillance loop. Health Care System. Public Health Authority. Reporting.

lita
Download Presentation

Evaluation of surveillance systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of surveillance systems 17th EPIET Introductory Course Lazareto, Menorca, Spain September – October, 2011 Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada Günter Pfaff

  2. The surveillance loop Health Care System Public Health Authority Reporting Data Event Analysis & Interpretation Decision Information Intervention (Feedback)

  3. Importance of evaluation • Obligation • Does the system deliver? • Credibility of public health service • In reality • Often neglected • Basis for improvements • Learning process • EPIET training objective • ”Do not create one until you have evaluated one”

  4. Does the surveillance system… • Detect trends? Epidemics? • Provideestimatesofmorbidityandmortality? • Identifyriskfactors? • Stimulateepidemiologicresearch? • Assess effects of control measures • Leadto improvedclinicalpractice? • Lead to new/improved control measures? • Lead to betteradvocacyandincreasedfunding?

  5. Criteria to look at • Simplicity • Flexibility • Acceptability • Data quality • Sensitivity and Predictivevaluepositive (PvP) • Capture-recapture • Representativeness • Timeliness CDC guidelines Footnote

  6. Simplicity Assimpleaspossiblewhilemeetingtheobjectives • Structure • Information needed • Number and type of sources • Training needs • Number of information users • Functionality • Data transmission • System maintenance • Data analysis • Information dissemination Footnote

  7. Components of system • Population under surveillance • Period of data collection • Type of information collected • Data source • Data transfer • Data management and storage • Data analysis: how often, by whom, how • Dissemination: how often, to whom, how Confidentiality, security

  8. Flowchart (HIV in Norway)

  9. Flexibility • Abilityofthesystemtoaccommodatechanges • New event to follow-up • New data about an event • New sources of information Footnote

  10. Acceptability • Willingnesstoparticipateinthesystem • Participation (%) of sources • Refusal (%) • Completeness of report forms • Timeliness of reporting

  11. Acceptability • Factorsinfluencingthewillingnesstoparticipate • Public health importance • Recognition of individual contribution • Responsiveness to comments/suggestions • Time burden • Legal requirements • Legal restrictions Footnote

  12. Completeness Proportion ofblank / unknown responses Simple counting Validity True data? Comparison Records inspection Patient interviews ... Data quality

  13. Completeness of information

  14. Sensitivity = reported true cases total true cases = proportion of true cases detected

  15. Sensitivity Disease - + - Total notified True + False + + Notified Total not notified - False - True - - Total sick Total not sick Sensitivity = True + / Total sick Specificity = True - / Total Not sick PVP = True+ / Total notified

  16. Sensitivity versus specificity The tiered system: confirmed, probable, possible

  17. ConsequencesoflowPvP • Frequent "false-positive" reports • Inappropriate follow-up of non-cases • Incorrect identification of epidemics • Wastage of resources • Inappropriate public concern (credibility) Footnote

  18. Measuring sensitivity • Find total true cases from other data sources • medical records • disease registers • special studies • Capture-recapture study

  19. Capture-recapture • Used for counting total number of individuals in population using two or more incomplete lists • Originally used in wildlife counting(birds, polar bears, wild salmon…)

  20. Uses in epidemiology • Estimate prevalence or incidence from incomplete sources • Evaluate completeness of a surveillance system

  21. Principles • Two/more sources of cases with disease • Lists, registries, observations, samples • Estimate total number in the source population (captured and uncaptured) from the numbers of captured in each capture

  22. Assumptions • The population is closed • No change during the investigation • Individuals captured on both occasions can be matched • No loss of tags • For each sample, each individual has the same chance of being included • Same catchability • Capture in the second sample is independent of capture in the first • The two samples are independent, pYZ = pY pZ

  23. Daddy, how many fish are in the aquarium? Seaworld Oberhausen, August 2010

  24. Your options as a scientist • Don‘t answer => Expect repeat question • Answer something => „How do you know?“ • Consult an expert • Estimate yourself

  25. Meet the expert - „Pulpo Paul“ • Has nine brains and three hearts • Managed to predict all German games during the 2010 Football World Cup right • Predicted accurately the finale Netherlands-Spain Binomial distributions only

  26. http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Teshttp://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

  27. Two-source model N=? Y1 Source Y Z1 Source Z b a c x=? N= a + b + c + x

  28. Two-source analysis N = Y1 Z1 / a Sensitivity of Y Ysn = Y/N = (a+c)/N Sensitivity of Z Zsn = Z/N = (a+b)/N

  29. How many persons are in the EPIET 2011 Introductory Course? Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“

  30. Hand does not meet our case definition This is our first view 1 4 4 5 2 3 4 4 3 How many persons are in the EPIET 2011 Introductory Course? 3 Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“, n=33

  31. This is our second view 4 3 2 6 3 How many persons are in the EPIET 2011 Introductory Course? Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=18

  32. How many participants at the course? • Capture: Source ”View #1” • Recapture: Source ”View #2” • Estimations  • Assumptions hold? 

  33. Number of participants Source View #2 – After Dinner Tutorial Yes No Source View #1 Dinner Yes 13 20 View #1 = 33 x No 5 View # 2 = 18 N = 13 + 20 + 5 + x N = 33 * 18 / 13 = 47 Sensitivity of View # 1 Sn1 = 33/47 = 70.2% Sensitivity of View # 2 Sn2 = 18/47 = 38.3%

  34. + 2 This is our second view (revisited) 4 3 2 6 3 How many persons are in the EPIET 2011 Introductory Course? Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=20

  35. Number of participants Source View #2, revised – After Dinner Tutorial Yes No Source View #1 Dinner Yes 13 20 View #1 = 33 x No 7 View # 2 = 20 N = 13 + 20 + 7 + x N = 33 * 20 / 13 = 51 Sensitivity of View # 1 Sn1 = 33/51 = 64.7% Sensitivity of View # 2 Sn2 = 20/51 = 39.2%

  36. So, just how many are there? 9 9 9 25 18 30 2 5 off screen Isla del Lazareto, Katharina‘s Lecture, Monday, 11 October 2010 – Case definition: “Persons in room“, N=53

  37. The problem with the X:Finding a comprehensive view

  38. Assumptions may not hold • The population is closed • Usually possible • Individuals captured on both occasions can be matched • OK if good recording systems • For each sample, each individual has the same chance of being included • Rarely true • Capture in the second sample is independent of capture in the first • Rarely true

  39. Sources are independent(most important condition) Being in one source does not influence the probability of being in the other source OR > 1 (positive dependence): underestimates N OR < 1 (negative dependence): overestimates N

  40. Dependent sources • Estimation of number of IVDU in Bangkok in 1991 (Maestro 1994) • Two sources used: • Methadone programme (April – May 1991) • Police arrests (June – September 1991) • Methadone  Need for drugs   Probability of being arrested  = negative dependence, overestimation of N

  41. Usefulness of capture-recapture • If conditions are met • Great potential to estimate population size by using incomplete sources • Cheaper than exhaustive registers or full counting • Two sources • Impossible to quantify extent of dependence • Multiple sources • Can adjust for dependence and variable catchability

  42. Examples of capture-recapture • STDs in The NL • Reintjes et al. Epidemiol Infect 1999 • Foodborne outbreaks in France • Gallay et al. Am J Epidemiol 2000 • Pertussis in England • Crowcroft et al. Arch Dis Child 2002 • Invasive meningococcal disease • Schrauder et al. Epidemiol Infect 2006

  43. Representativeness • A representative system accurately describes • Occurrence of a health event over time • Distribution in the population by place and time • Difficult to determine • Compare reported events with actual events • Characteristics of the population • Natural history of condition, medical practices • Multiple data sources • Related to data quality, bias of data collection, completeness of reporting Footnote

  44. Timeliness Analysis and interpretation Reporting Action taken Disease onset Diseasediagnosed ofevent Public Health Authorities Clinician, labs

More Related