ECRAN – EBM guide for healthcare journalists

1. European Communication on Research Awareness NeedsWP6 – EBM guide for healthcare journalists to the critical interpretation of clinical trials

2. ECRAN – EBM guide for healthcare journalists What is quality of scientific information? Let’s sort out the wheat from the chaff … The huge amount of research studies is a big challenge for health care journalists, who have to sort out the wheat of scientific excellence from the chaff of low quality or interested driven science. The purpose of this guide is to ensure excellence in medical journalism by means of: • applying the core principles of journalistic coverage and investigation in the field of science and medicine; • training journalists to properly cover news events, trends, and issues in all aspects of health care, sorting out the tangle of data about clinical trials and transferring them to medical practice in a comprehensive and multifactorial perspective; …

3. ECRAN – EBM guide for healthcare journalists WP3 task 3.2 – Others (1) … • enabling properly trained journalists to better convey the complex dimensions of health care challenges from an overall point of view; • ensuring an independent coverage about health and research, analyzing health related data with fairness, accuracy and balance; in order to: • increase the capacity of individuals and policymakers to adopt health care attitudes and policies that can improve the health of all the citizens; • improve the ability of citizens to make wise decisions about behaviors that promote health, treatment options, and their choice of health care providers.

4. ECRAN – EBM guide for healthcare journalists WP3 task 3.2 – Others (2) What is science? «A system of certain knowledge in a period of time built up, arranged and ensured by research, teaching and accumulated data». (translation from the German definition in Brockhaus 2003:2193) Which is the goal of science? The ultimate goal of science must always be to verify or reject existing (content-specific or methodical) knowledge and to replace it with new knowledge. Science is a cumulative process. The touchstone for assessing new knowledge is the comparison with existing knowledge. Systematic reviews, that collect and evaluate knowledge of several scientific studies, have been developed and implemented for 20 years as methodical tool for assessing research.

5. ECRAN – EBM guide for healthcare journalists WP3 task 3.2 – Others (3) How should good science look like? Assessing good science according to its neutrality and objectivity is unrealistic, because of its several infrastructural, financial and performance-related implications. In the last decade the call of transparency has been playing a central role in this process, in order to ensure the integrity of research. Assessment of the framework conditions for research should be therefore integrative part of each journalistic investigation, in order to set an indicator for a preliminary assessment of the reliability of research outcomes. How should good journalistic investigation look like? Basic scientific understanding of scientific studies and their weak points enables journalists to facilitate orientation and to avoid lurking pitfalls in this field (e.g. look at the section “Tips for understanding studies – Common pitfalls” of this guide).

6. ECRAN – EBM guide for healthcare journalists Toolkit for journalists There are many challenges to achieving accuracy, balance and completeness in health care journalism. For a reporter, there are deadlines, editors, and corporate financial pressures. Though quality in health care journalism is far more than this, because it plays a major role in educating consumers! Consumers need to be well informed to make sound choices that affect their health and well being. The following rating criteria are an internationally validated methodology, that helps conjugate general and medical journalistic criteria to review stories claiming about specific treatments, tests, products or procedures (i.e. drugs or devices, vitamins or nutritional supplement, diagnostic and screening tests, dietary recommendations, surgical procedures).

7. ECRAN – EBM guide for healthcare journalists 1 Quantification of benefits (treatment/test/product/ procedure) 2 Quantification of harms of intervention 3 Proper evaluation of quality of evidence Rating criteria for healthcare journalists 4 Further independent source and declaration of conflicts of interest 5 Story solely or largely based on news release? 6 Establishing true novelty of approach 7 Comparing new approach with existing alternatives 8 Establishing availability of treatment/test/product/ procedure 9 Adequate estimate of costs of intervention 10 Is there potential for disease mongering?

8. ECRAN – Rating criteria for healthcare journalists 1 Quantification of benefits (treatment/test/product/procedure) Stories are expected to describe the effectiveness of a benefit. Few stories provide helpful numbers to back the real benefit of a new treatment, test, product or procedure. Empirical investigations have shown that news reports, drug ads and some studies coverage tend to report benefits as the relative reduction instead of the absolute risk reduction in the frequency of failures. Another challenging question may be: «How many patients have to be treated in order to prevent the problem for one patient?». This is the Number Needed to Treat (NNT): it is defined as the inverse of the absolute risk reduction and is the average number of patients who need to be treated to prevent one additional bad outcome.

9. ECRAN – Rating criteria for healthcare journalists 1 What does “significantly increased” mean? How was this measured? Apart from this, statistical significance may not be equal to clinical relevance. The real question is: «What difference does it make in peoples’ lives?» Reports indicating surrogates and intermediate endpoints, e.g. changes in blood test scores, may not influence individual health scores (increased life expectancy or lower mortality rate) or quality of live. Moreover, anecdotal storytelling makes engaging reading but is not scientific data. Glowing personal stories or single unchallenged exaggerated success stories may not be a representative example of how a benefit works and require a critical appraisal for a balanced framing of it.

10. ECRAN – Rating criteria for healthcare journalists 1 Criterion will not be fulfilled, in case of … • Data of possible benefit is not available at all. • Using only relative, not absolute, risk reduction or benefit data. • Relying too heavily on what may be unrepresentative patient anecdotes about benefits. • Citing statistics appropriately, but allowing even a single unchallenged exaggerated quote to throw the story out of balance. Question to clear at the end of the day Is the benefit really as much more or less significant as it is presented?

11. ECRAN – Rating criteria for healthcare journalists 2 Quantification of harms of intervention Be aware of appearances! When you hear someone saying that a new approach «appears to be safe», look for evidence supporting it. Stories may emphasize or exaggerate potential benefits of interventions, while minimizing or completely ignoring their potential harms. For some patients slight side effects may even have dramatic impact on their lives. An overall healthcare report should always consider possible – even the slightest – adverse effects of interventions, ideally reporting their frequency and severity. Any healthcare decision implies tradeoffs, all treatments may have side effects, risks and complications.

12. 2 ECRAN – Rating criteria for healthcare journalists Even a screening test may turn out to be harmful to the individual getting it: in fact, it may produce a false positive result, which causes anxiety, and will probably lead to more testing (i.e. biopsy) and possibly treatment (i.e. surgical intervention) that comes with its own harms. Moreover, enthusiasm for early tests in people without symptoms is often not supported by evidence. PSA testing, for example, often turns out to cause more harms than benefits, including false positive results and overdiagnosis, both of which could lead to unnecessary treatments associated with significant side effects.

13. 2 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • Failing to mention potential harms. • Failing to quantify potential harms. • Failing to describe the severity of potential harms. • Failing to account for “minor” side effects that could have a significant impact on a patient’s life. • Relying too heavily on a patient anecdote about safety. • Relying too heavily on a researcher’s comment that an approach appears to be safe – without supporting data. Question to clear at the end of the day Which are the risks and the side effects of this intervention or product?

14. 3 ECRAN – Rating criteria for healthcare journalists Proper evaluation of quality of evidence Healthcare coverage should critically evaluate the evidence and not merely take published or presented research as gospel. Quality of evidence has to be classified according to evidence hierarchy:

15. 3 ECRAN – Rating criteria for healthcare journalists Science works cyclically with ebbs and flows: knowledge stream keeps flowing and keeps changing in the course of time. So, ask yourself: At which point in the stream did I grasp the evidence? Not all evidence is bullet-proof, even if it is published in high impact journals, because not all studies are equal. Eminence is not evidence and study stories often turn out to be ongoing stories. A solid journalistic investigation is expected to scrutinize limitations of evidence, number of probands, duration of the study, relevance of outcomes; to assess whether what appears as the latest story is already existing in larger and better-designed stories on the same topic or whether the statements generalize small group data (e.g. a single medical center with experienced surgeons).

16. 3 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • Failing to dig out the limits of the study design (i.e. lack of control group, lack of blinding, missing drop out rate). • Not cautioning on the interpretation of uncontrolled data. • Conflating causation and association (i.e. limitations of observational studies). • Generalizing small group data (i.e. at one medical center with experienced surgeons). • Confounding surrogates with real health outcomes (illness or death vs. markers or scores). • Not mentioning that scientific meeting outcomes lack adequate peer review. • Not pointing out that findings from animal or lab experiments are not directly applicable to human health. • Presenting anecdotic data as evidence for intervention. Question to clear at the end of the day Does this study design justify these allegations?

17. 4 ECRAN – Rating criteria for healthcare journalists Further independent source and declaration of conflicts of interest Claims about new treatments, tests, products, and procedures should be based on evidence, not on beliefs or business hopes. In health care journalism, it is therefore essential that second opinions be sought from independent experts who have no vested interest in the approach being discussed. For this scope two things must occur: • there must be an independent expert source quoted (someone not directly connected with the research), • there must be some attempt to let the public know about potential conflicts of interest.

18. 4 ECRAN – Rating criteria for healthcare journalists Hyperbolic healthcare claims made by drug companies, medical device manufacturers and academic research centers are to be confronted with this question: Do they have something to gain? May be that: a trial is paid for by the drug manufacturer; researchers are getting fees from a drug company; claims are made by a spokesman for a device manufacturer; doctors are being early adapters of a new device; there are underlying inventors’ interests.

19. 4 ECRAN – Rating criteria for healthcare journalists A study funded by industry doesn’t necessarily mean that findings lack integrity, but evidence from recent studies shows that potential conflicts might lurk around every corner in health care. So, the next time you go to your doctor’s office that is cluttered with drug company promotions, it is reasonable for you to ask him if he/she had any financial relationship with the makers of drugs, devices or test advertised for. Moreover, seek independent perspectives and beware of statements like: «Experts believe …, doctor think …». Instead Give them a name, because vagueness is nameless!

20. 4 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • Failing to identify the source of the story (news release, journal article, editorial, scientific meeting presentation, etc.) and failing to make obvious the extent to which that source is likely to be conflicted (e.g. «A PR consultant working for the company said…» or «Dr. Smith, who received a company grant to perform the study, said…») • Failing to include the input of a trusted independent source with expertise on the subject to comment on the claims made (ideally, more than one such independent source would be used). Question to clear at the end of the day To what extent are the information independent and how plausible is the expertise?

21. 5 ECRAN – Rating criteria for healthcare journalists Is the story solely or largely based on news release? It is simply not sound journalistic practice to lift material directly from a news release without stating that this is the case. If we know that a news release was sent out and if we can get a copy of that release, we will review it along with the story. Many news organizations facing financial pressures may be lead to become more open to using news releases as a way to publish content while cutting costs. Although news releases can be valid sources of some information, journalism is being asked to collect independent information with the consequence that it is unacceptable to rely on a news release as the sole source of information.

22. 5 ECRAN – Rating criteria for healthcare journalists Particularly in health care, there are many vested interests trying to influence consumer choices. Journalists are expected to use independent verification relying neither on news releases nor on company spokesmen! A team from Dartmouth Medical School and the Veterans Affairs Outcomes Group published in 2009 an analysis on the quality of press releases by medical centers concluding that «Press releases from academic medical centers often promote research that has uncertain relevance to human health and do not provide key facts or acknowledge important limitations.»

23. 5 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • There is clear evidence that the exact same wording from a news release appears in a story. • Lacking evidence of a press release and the worst a story can be rated is “not applicable”. Criterion will be fulfilled, in case of … • A story has several independent sources and proves NOT to rely only on a news release. Question to clear at the end of the day Is the news based only on a press release?

24. 6 ECRAN – Rating criteria for healthcare journalists Establishing true novelty of approach Many alledgedly new treatments, tests, products or procedures are not really novel. The agent may be another member of a well-established therapeutic class of drug. Even if it represents a new class, it may offer no more than the drugs that are widely available. In the press release for a new drug this essential information may be lost in the hype and the drug is portrayed as “novel” in order to increase initial sales. So ask yourself: Is it novel or simply new? Journalists’ reporting should be able to make these distinctions on the real novelty of a product by researching or asking independent sources, e.g. specifiying that there are previous studies on this agent for the same indication.

25. 6 ECRAN – Rating criteria for healthcare journalists In health care, newer isn’t always better, the first in a new class of drugs may offer no more than the drugs that are widely available. Drug companies are very good in promoting the alledged novelty of a drug in order to increase initial sales. Moreover, some claims for new drugs are made for “off-label use”, i.e. for not officially approved uses and under doctors’ careful monitoring. Off-label uses might also result in wasting money and harming patients and the respective costs may not be covered by the state or private insurance. By focusing on a new idea journalists should tell how well it works putting it into the context of existing alternatives.

26. 6 ECRAN – Rating criteria for healthcare journalists Mostly it is rare that a study appears out of nowhere and the “big picture”, where the “novelty” comes from, is often missing. To find it out journalists should search for the opinion of independent experts or reliable scientific sources like.

27. 6 ECRAN – Rating criteria for healthcare journalists Criterion is will not be fulfilled, in case of … • Failing to accurately describe the new product/intervention and, as a consequence of this, misleading the consumer. • Failing to explain the real novelty of the product/intervention (preferably comparing it to existing alternatives on the market). • Failing to mention that the described new product/intervention is, in fact, for off-label uses. Question to clear at the end of the day Is the product/intervention a real novelty?

28. 7 ECRAN – Rating criteria for healthcare journalists Comparing new approach with existing alternatives It is mandatory for a healthcare journalist to put the new treatment into the context of existing alternatives that have a longer, more proven track record. Whatever the new approach, the report should try to put it into the context of already available interventions, in order to compare the effectiveness of the various options. News stories on new treatments, tests, products and procedures should also take into consideration the value for money ratio. In case a treatment should have been developed from scratch, the journalist is expected to explain that there are no alternative treatments or only supportive or soothing remedies.

29. 7 ECRAN – Rating criteria for healthcare journalists “To wait and see” is also a plausible alternative to the proposed treatment/test/product/intervention. “Active surveillance”, declining immediate aggressive therapy, is a viable and rationale option. It is wise for patients to know their treatment options. They may also consider to make lifestyle changes rather than taking a drug, or else they may pursue a drug or non-surgical option rather than having surgery. Promotions of screening tests should also consider the option for individuals to choose NOT to be screened.

30. 7 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • Discussing a surgical approach without mentioning non-surgical alternative approaches. • Discussing a new test without mentioning other tests that are available, including the option of not being screened in the case of a screening test. • Failing to discuss the advantages/disadvantages of the new idea compared with existing approaches. • Failing to discuss how the new treatment, test, product or procedure fits into the realm of existing alternatives. Question to clear at the end of the day Are there any other treatment options besides this procedure/product?

31. 8 ECRAN – Rating criteria for healthcare journalists Establishing availability of treatment/test/product/procedure This criterion shall mostly be applied to coverage about a new drug in clinical trials (or in a trial on the new use of an established drug). A journalist is expected to tell his/her readers/public, if the drug/agent is available at the chemist’s (or may be soon) or if it is really only in an early-phase trial (e.g. pilot study or stage before approval). It is important for a journalist to know that there may be a lot of marketing hype that suggests that something is well-studied and ready for use. Questions should also be raised by the fact that something should have been approved in the U.S. and not or not yet in Europe.

32. 8 ECRAN – Rating criteria for healthcare journalists Some early studies focus on safety and not on how well something works. Some other focus on early efficacy and may not be large enough or designed to show safety of a drug/treatment in the long term. Predictions about «something to be approved by …» in a certain period should also be taken with a grain of salt, they usually come from someone who stands to benefit. New approaches, devices or operations should be scrutinized for their availability (no one wants to be patient number one with a new device/operation). Finally, have the personnel required to deliver the new approach properly been trained and are there enough of them? What is the learning curve for this?

33. 8 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • Making not clear that the drug is not available now. • The approval of a drug is treated as a closed issue. • Something «expected to be approved by …» and its availability on the market sounds more like a crystal ball prediction. • Time predictions about the market availability of the drug are solely based on declarations sourced to a company spokesperson. • Failing to describe to what extent a treatment is available (in every hospital or only in specialty clinics). • Promoting a special intervention without providing a sense of how many trained doctors there are in the field (e.g. a hi-tech surgical intervention). Question to clear at the end of the day Is this item available now or not?

34. 9 ECRAN – Rating criteria for healthcare journalists Adequate estimate of costs of intervention Journalists are expected to appropriately report on intervention/product costs, though it may be difficult to estimate costs of an experimental approach early in its development. Even if the real cost is probably known at the time of market entry, health reporters could at least cite costs of existing alternatives. Journalists should be able to get a reasonable estimate from their source/experts on the topic. The rule of thumb is: If it is not too early to talk about how well something might work, then it is not too early to start discussing what it may cost

35. 9 ECRAN – Rating criteria for healthcare journalists A large amount of healthcare stories do not appropriately report on the cost issue. Healthcare takes a large part of the Gross Domestic Product in developed countries and we all better start thinking about costs of treatments, tests, products and procedures, we all pay for. So, it is not enough to say The cost is much lower than … but you should better ask What does this actually cost? How much lower? What about the cost-effectiveness, if compared to …? Journalists should also consider additional expenses for psychologists, dieticians, physiotherapists involved in the intervention or that the convalescence and rehabilitation may be more time-consuming. Moreover, they should investigate, if insurances cover the intervention/drug or it is affordable for the patients needing them.

36. 9 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • Costs of an intervention are not mentioned at all. • Costs are played down without giving a satisfactory explanation therefore. • Not comparing the costs of a procedure with other available alternatives (more or less expensive ones). • Not mentioning, if the new procedure/drug will be covered by public/private health insurance. Criterion will not apply, in case of … • An experimental procedure in early development with no mention of the related costs. • An expert claiming that costs are quite unimportant (e.g. speaking of well known products like aspirin or weight-loss products). Question to clear at the end of the day Is it covered by health insurance or must I pay out of my pocket? Question to clear at the end of the day:

37. 10 ECRAN – Rating criteria for healthcare journalists Is there potential for disease mongering? This criterion assesses whether a condition is exaggerated or oversold. There are different forms of “mongering”: • turning risk factors into diseases (e.g. low bone mineral density becomes osteoporosis); • misrepresentation or inaccurate description of the natural history and/or severity of a disease (e.g. early-stage low-grade prostate cancer); • medicalization of minor or transient variations in function (e.g. temporary erectile dysfunction or female ‘sexual dysfunction’); • medicalization of normal states (baldness, wrinkles, shyness, menopause); • exaggeration of the prevalence of a disease/disorder (e.g. using rating scales to ‘diagnose’ erectile dysfunction).

38. 10 ECRAN – Rating criteria for healthcare journalists Some statements are trying to make all of us feel like there’s something wrong with us – even if there isn’t. That’s what we call disease-mongering. This is dangerous because it is an unnecessary emotional and financial burden for society and may distract us and precious public resources from important healthcare issues that should be addressed. Also turning a normal state or variation of health into a medical problem is disease mongering, like baldness, wrinkles, shyness or menopause. Pseudodiseases, that lower the threshold for common diseases but, in fact, only open new markets, may turn millions of people overnight into patients needed to be treated with drugs or vitamins against high cholesterol or osteoporosis. Often, women are targeted because of premenstrual dysphoric disorders, female sexual dysfunctions, overactive bladders or menopause; men equally get their share being hammered because of low testosterone levels, balding etc.

39. 10 ECRAN – Rating criteria for healthcare journalists Identifying disease mongering is a matter of judgment. Sometimes it is obvious. Sometimes there’s a fine line about whether a statement on irritable bowel disorder, erectile dysfunction, restless leg syndrome or osteoporosis (all of which can be serious for some sufferers) is misrepresenting the condition to the public. Sometimes there may be “multiple mongering” with many hyperboles for the same issue.

40. 10 ECRAN – Rating criteria for healthcare journalists Criterion will not be fulfilled, in case of … • Spurious statistics are presented. Question prevalence estimates! • Statistics inflate the seriousness of a condition (e.g. «millions of people `suffering` in silence with toenail fungus…») • Someone “selling sickness” wants to scare you into testing or treatment. • Risk factors are treated as diseases to convince us that we should be concerned about “a score” (i.e. surrogate markers, e.g. changed blood parameters) instead of outcomes like illness or death. • Holding up “worst-case patients experiences” as representative examples of all with this conditions.

41. 10 ECRAN – Rating criteria for healthcare journalists Criterion will not apply, in case of … A therapy/procedure/test/drug possibly being overestimated. There is no «treatment mongering» as such. This item should be best scrutinized under criterion Quantification of benefit or Proper evaluation of quality of evidence. Question to clear at the end of the day Should I really be concerned about this disease/circumstance?

42. 11 ECRAN – General journalistic criteria Choosing the right issue Is it a current, relevant and original contribution? Topicality of an issue may be related to the event and the frequency of medium publication, the importance or the originality (i.e. curiosity) of its contents. Relevance may occur for instance, if the issue directly or indirectly regards a large part of the population, if a therapy determines high costs for the health system, if a new development has an ethical dimension, if the topic has political importance or an exemplary nature. If not relevant, an issue may instead be curios, humorous, surprising etc. Novelty of an issue may be characterized by latest data published e.g. today, yesterday, this week in a scientific journal, or outcomes of recently held conferences. The rule of thumb is: If the issue should not be current, then it must be relevant. If not relevant, then it must be at least curios/original!

43. 12 ECRAN – General journalistic criteria Delivery of message Has the issue been successfully delivered or does it even have model character for its format? A successful message must be delivered in an intelligible language, with a plain structure avoiding long and complicated sentences. Specialist or technical terms should only be used in exceptional cases and, anyway, accordingly explained by means of context or examples. Formal factors also play an important role: the story must follow dramaturgical principles (personalisation, narrative elements, relation to everyday life); form and content must harmonise; abstract thinking must be explained by pictures, illustrations, clear specimen texts; good sound clips in a radio report should invite to listen to medical issues. In case of reports, WH questions should all be answered. Rule of thumb: delivery standard is conform with the format

44. 13 ECRAN – General journalistic criteria Factual accuracy Does the story report essential facts? Factual accuracy means reporting historical facts without manifest or essential errors, e.g. accurate reporting of study conclusions or data, correct description of study design (i.e. number of participants enrolled/allocated/followed up/analysed in clinical trials). Accuracy is neither synonymous with objective truth nor completeness, e.g. a report must not comprise each and every aspect of the study but the described facts must be true. A single name error is not enough to invalidate this criterion, but many little mistakes may create the impression of a real sloppy work! Apart from purely data errors or hyperboles in the headlines or teaser beware of surrogate markers, association vs. correlation, absolute vs. relative risks, NNT, off-label use, phases of clinical trials, confidence interval etc ...

45. ECRAN – Tips for understanding studies – Common pitfalls Surrogate markers A surrogate marker is an event or a laboratory value that researchers hope can serve as a reliable substitute for an actual disease. A common example is blood cholesterol levels which are surrogates, or substitutes, for heart disease. As a matter of fact, surrogate studies are indirect by definition and a journalist should be skeptical before celebrating the last medical breakthrough, because what is sound medical dogma today may fade over time. Surrogate research is an important research tool, that can open important therapeutic avenues and stimulate additional research. Those who earn their living and reputation from medical research may have a different view on surrogacy, but it is important to understand that exaggerations may damage patient care by highlighting surrogate outcomes instead of important results such as death, quality of life or functional capacity.

46. ECRAN – Tips for understanding studies – Common pitfalls Association vs correlation and language use Look at this wording describing the results of an observational study: «Frequent fish consumption was associated with a 50% reduction in the relative risk of dying from a heart attack.» And look at the editor’s streamline “adaption” detrimental to meaning: «Women who ate fish five times a week cut their risk of dying later from a heart attack by half.» The new wording suggests that the subjects’ fish consumption is responsible for their dying less frequently from heart attacks, but the original study does not support a conclusion of cause and effect. Epidemiologic – or observational – studies examine the association between what is known in epidemiologic jargon as an exposure (e.g. a food, environmental influences, lifestyle) and an outcome (often a disease or death). Because of all the other exposures occurring simultaneously in the complex lives of free-living humans that can never be completely accounted for, such studies cannot provide evidence of cause and effect; they can only provide evidence of some relationship (between exposure and outcome) that a stronger design could explore further.

47. ECRAN – Tips for understanding studies – Common pitfalls The only study design involving humans that does rise to the level of demonstrating cause and effect is a randomized trial. In this design, study subjects are assigned an exposure (or a control condition) at random, irrespective of any other exposures in their lives, and all such other exposures are then assumed to even out between the treated group and the control group of subjects (and this can be demonstrated). As a result, the only difference between the groups is whether they receive the exposure under study or the control condition. This approach is a true experiment. Any difference in outcome seen between the control and the experimental group should be due to the one factor or variable that differs. A subtle trap occurs in the transitionfrom the cautious, nondirectional, noncausal, passive scientifical language in reporting the results of observational studies to the active language used in mass media. Active language is fine in general (who wants to write like a scientist?), but problems can arise when the use of causal language is not justified by the study design. In practice, a shift to causal language can occur at any stage: writing, editing, or headline composing, with similar effects on meaning, that could lead readers to overestimate the meaning of a given study and possibly even make life choices that the evidence does not warrant. Sometimes even scientists and press-release writers slide into causal language in expressing results of observational studies. 

48. ECRAN – Tips for understanding studies – Common pitfalls Absolute vs relative risks Researchers, clinicians or journalists often report only relative differences in making claims about a new idea, but they should tell the rest of the story! Absolute differences make the real sense out of such claims. Look at this example about blindness in diabetic patients in a 5-year-period: • If the risk for blindness is 2 in 100 (2%) in a group of patients treated conventionally and 1 in 100 (1%) in patients treated with a new drug, the absolute difference is derived by simply subtracting the two risks: 2% – 1% = 1%. • Expressed as an absolute difference, the new drug reduces the 5-year risk for blindness by 1%. • The relative difference is the ratio of the two risks. Given the data above, the relative difference is: 1% ÷ 2% = 50% • Expressed as a relative difference, the new drug reduces the risk for blindness by half. Data are accurate in both cases, but: if you are a marketing manager for the new drug, you are likely to only use the relative risk reduction; if you are a journalist, you would better do your job by citing the “rest of the story” (absolute risk reduction) often missing in news releases and direct-to-consumer prescription drug ads.

49. ECRAN – Tips for understanding studies – Common pitfalls Number needed to treat (NNT) The NNT is defined as the inverse of the absolute risk reduction or the number of patients who need to be treated to prevent one additional bad outcome, calculated as 1/absolute risk reduction. Applied to our hypothetical diabetes blindness drug example: • the risk for blindness in a patient with diabetes over a 5-year period is 2 in 100 (2%) in a group of patients treated conventionally and 1 in 100 (1%) in patients treated with a new drug. So the absolute difference is derived by simply subtracting the two risks: 2% – 1% = 1%. • the number needed to treat would be 1 / 1% (or .01) = 100. • so you would need to treat 100 people with diabetes for 5 years in order to prevent one case of blindness. You can see that this is an important way to look at new claims about new drugs.

50. ECRAN – Tips for understanding studies – Common pitfalls Off-label use Prescription drugs are officially approved by regulatory agencies for specific indications to treat specific conditions or diseases. Physicians may prescribe a drug for a use that is not described in the approved labeling if it seems reasonable or appropriate to them. This is what’s called off-label use. For example: it is not uncommon for physicians to prescribe low doses of beta blocker drugs to help people overcome jitters before public speaking. Beta blockers are not formally approved for this use, but in such circumstances doctors are advised that they have the responsibility to be well informed about the product, to base its use on firm scientific rationale and on sound medical evidence, to inform the patients and get its consent for the use of the drug. A study in the Archives of Internal Medicine found that about 21% of all estimated uses for commonly prescribed medications were off-label, and that 15% of all estimated uses lacked scientific evidence of any useful or beneficial effect. Authors of the study recommended more careful monitoring of off-label uses that might waste money and harm patients. Another problem of the off-label prescribing is that its cost may not be reimbursed by public or private health insurances. Finally, whilst off-label use is permitted, marketing of a drug for an off-label use is not.