Getting and making sense of “the best available” scientific evidence

1. Getting and making sense of “the best available” scientific evidence EBMgt Helping Managers Make Better Decisions

2. What is evidence? • Evidence is not the same as ‘proof’ or ‘hard facts’ • Evidence can be • - so strong that no one doubts its correctness, or • - so weak that it is hardly convincing at all

3. Don’t confuse • Evidence of effect (do!) • Evidence of no effect (don’t!) • No evidence of effect (research!)

4. Assignment CAT Critical Appraised Topic

5. CAT: Critical Appraised Topic A critically appraised topic (or CAT) is a structured, short (2 pages max) summary of evidence on a topic of interest, usually focused around a practical problem or question. A CAT is like a “quick and dirty” version of a systematic review, summarizing the best available research evidence on a topic. Usually more than one study is included in a CAT.

6. CAT: structure • Question • PICOC • Background • Search strategy • Results • Conclusion • Comments • Recommendation • References

7. Part 2 Asking the right questions

8. 5-step approach Gathering Best ScientificEvidence is a 5-step approach • Formulateananswerablequestion (PICOC) • Search for the best availableevidence • Criticalappraise the quality of the foundevidence • Integrate the evidencewith managerial expertise andorganizational concerns and apply • Monitor and evaluate the results

9. Asking the right question? Does team-building work? Does leadership development training work? Does management development improve the performance of managers? Does employee participation prevent resistance to change? Is 360 degree feedback effective?

10. Answerablequestion: PICOC P= Problemorpopulation I = Interventionorsuccessfactor C= Comparison O = Outcome C= Context

11. Answerablequestion: PICO(C) Scenario: You are a consultant, yourclient is aninsurancecompany, there are plans for a merger, you have heardthat the othercompany has a different culture, you want to knowifthiswill effect the outcome P= Organizationswith a different corporate culture I = Merger C= Organizationswith a similarcorporate culture O = Long term profitability (C))= Profitorganizations, competitivemarket)

12. Searchingevidence The problem with finding evidence: the abundance of literature

13. Article 1 Article 3 Article 4 Article 5 Article 7 Article 8 Article 6 Article 2 Searching evidence There are about 1350 articles published on HRM every year. For an HR professional to keep up this means reading 3 to 4 articles every day! (most of these publications are not valid or irrelevant)

14. Searching evidence Evidence-basedsearching • In a systematic and transparant waysearchingfor the “best” evidence • Part of EBMgtwheredecision maker is not a ‘subject matter expert’

15. Searchingforscientificevidence What kind of evidence are we lookingfor? • Studies with a design that best suits the research question • Studies with the highest level of evidence

16. Which design for which question? Explanation

17. Levels of internal validity

18. Searching evidence Where do we search?

19. Peer reviewedjournals

20. Whichjournals are peer reviewed?

21. Company Annual Reports, Datastream, Factiva.com, Amadeus Wall street Journal, Financial Times, Business week, Financieel Dagblad ABI/INFORM, Business Source Premier, Emerald, PsychInfo, Science Direct Current Information Overview of a subject General background Academic Information Statistical Information Theories about a subject Company information CBS Statline, Eurostat Textbooks and encyclopedias Textbooks and popular books Encyclopedias, yearbooks & book reviews Information sources Type of Information Source

22. Searching evidence How do we search? Search Strategy

23. Why do we need a search strategy? Search strategy Promotes deeper learning about your question Leads to better yield of quality research. Saves time in the long run. Source: Inky Bob, www.flickr.com, Creative Commons, April 2006.

24. Building blocks method Snowball method Search strategy Two types of search strategies

25. Snowball method Starting from one book or article, you search for other literature on the same topic. • Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article). • Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge or Google Scholar).

26. OR OR OR Building blocks method Keyword 1 Keyword 2 Keyword 3 Keyword 4 Synonyms or related terms • …. • …. • …. • …. Synonyms or related terms • …. • …. • …. • …. Synonyms or related terms • …. • …. • …. • …. Synonyms or related terms • …. • …. • …. • …. AND AND AND

27. Answerable question: PICOC P= back office employees I = merger, integration, back office C= status quo O = economy of scale C = healthcare, different organizational culture, unequal • Underline the keywords • Number the order of importance from 1-4

28. Answerable question: PICOC P= back office employees I = 1. merger, 3. integration, back office C= status quo O = 4. economy of scale C = 5. healthcare, different 2. organizational culture, unequal • Underline the keywords • Number the order of importance

29. Search terms Operationalise your Pico elements! O= long term profitability?

30. Select keywords The keywords of your PICOC may be enough. If not, select more words by using: • synonyms • alternate spelling, translations • related terms / words / subjects • narrower or broader terms corporate culture: organizational behavior/character, corporate identity merger: acquisition, take-over, fusion, combination, unification profitability: profit, advantage, return on investment, shareholder value

31. Search Query • Search with #1 PICOC term (incl. alternative terms, synonyms, alternate spellings, truncations, etc.) in the thesaurus, title or abstract • Combine the results with OR (use the history function!) • Search with #2 PICOC term (incl. synonyms, etc.) • Combine the results with OR • Combine the results of step 2 and 4 with AND

32. Merger • Fusion • Combination • Take over • Acquisition • Unification • … • Corporate culture • Organizational behavior • Organizational character • Corporate identity • Core beliefs • Shared values • Healthcare organization • Non profit • Not for profit AND AND 2. Corporateculture OR OR OR 3. Integration 1. Merger 4. Health care organization Search Query: an example I I O C • Integration

33. Search Query Use the history function to combine results

34. Search Query

35. Boolean operators • AND = both terms (apples AND oranges) • OR = either one of these terms (apples OR oranges) • NOT = without this term (fruit NOT oranges) • NEAR = near this term (apples NEAR oranges) • * = replaces 0,1 or more characters (apple*= apple, apples, applejack, applejuice, applepie, etc.) • ?= replaces 1 character (organi?ation= organisation, organization)

36. oranges oranges apples apples Boolean operators apples AND oranges apples OR oranges

37. Justify your search strategy Search Strategy • Why? To help the reader of your paper: • Follow the steps of your search process • Understand the end results • How? • Including keywords used for the search actions • Justify information sources used (literature list)

38. Include literature references Search Strategy • Why? • To give other authors the credit they are due. • To show that you have made use of reliable sources • To show the relationship between your work and that of others. • To show that you have studied the subject in depth • To make it possible to check your work. • To avoid committing plagiarism !!! • How? • Cite & include references to acknowledge all your sources carefully. • Include sufficient own / new ideas in your work. • You can make use of Reference ManagerorEndnote

39. Start up select ‘advanced’ select ‘peer-reviewed’ select ‘ABI/INFORM Global’

40. ABI / Inform

41. Learningthroughplay ! • Try all buttons • Makelots of mistakes • Have fun! • Go do it & report back next week.

42. Levels ofScientificEvidence Levelsof evidence=A hierarchical order for research designs based on their internal validity Internalvalidity=Degree the results maybe unbiased. Higher when conditions demonstrating causality are present (1. control over “cause”, 2. temporal order, and 3. control over or no plausible alternative explanation for findings). • Careful design of primary studies promotes these three conditions but seldom eliminates them. Threats to internal validity are overcome when accumulated studies with different designs yield comparable findings.

43. Levels of internal validity

44. Levels of internal validity It is shown that … It is likely that … There are signs that … Experts are of the opinion that …

45. Internal validity But … sometimes observational studies are as good as RCT’s When the size of effect is very large (swamps the bias)

46. Generalizability • Degree findings hold across populations, settings, procedures etc. (external validity). • Reasons for rejecting generalizability must be logical and evidence-based (not mere dislike of findings) • Logical threats to generalizability include: • Person/Treatment interactions: e.g., incentives based on dice throw that work for gamblers and not Baptists • File draw problem: Studies only published if show significant effects (why unpublished sources matter)

47. Internal validity These treatments have not been tested in RCTs: are they supported by poor evidence? Cardiac arrest: AED Heimlich manoeuvre Dehydration: drinking water

48. But …. Better than a single study: a replication studyBetter than a replication study:a systematic review / meta analysis If there were 100 studies, 99 of which gave a ‘negative’ result (where, say, the new intervention appeared to be not effective), while one had a ‘positive’ result (were the intervention appeared effective), it would obviously be a mistake to consider only the single positive study.

49. Research designs • Systematicreviewormeta-analysis • Randomizedcontrolledstudy (experiment) • Non-randomizedcontrolledstudy (quasi-experiment) • Observationalresearch: cohort-, panel-, case-control and cross-sectionalstudy • Before-afterstudy (pretest – posttest design) • Qualitativeresearch

50. Systematicreview The intention behind a systematic review is to identify as fully as possible all the scientific studies of relevance to a particular subject and to assess the validity and authority of the evidence of each study separately. As the name indicates, a systematic review takes a systematic approach to identifying studies and has the methodological quality critically appraised by multiple researchers independently of each other, as a consequence of which the review is transparent and reproducible and can be monitored. The use of statistical analysis techniques in a systematic review to pool the results of the individual studies numerically in order to achieve a more accurate estimate of the effect is termed a “meta-analysis”.