Critical Appraisal

Critical Appraisal Or Making Reading More Worthwhile www.bradfordvts.co.uk

The Problem • Vast and expanding literature. • Limited time to read. • Different reasons to read – mean different strategies. • Keeping up to date. • Answering specific clinical questions. • Pursuing a research interest. Bruce Davies

Stages • Clarify your reasons for reading. • Specify your information need. • Identify relevant literature. • Critically appraise what you read. Bruce Davies

Clarify Your Reasons for Reading. • Keeping up to date. • Skimming the main journals and summary bulletins. • Answering specific clinical questions. • Finding good quality literature on subject. • Pursuing a research interest. • Extensive literature searching. Bruce Davies

Specify Your Information Need. • What kind of reports do I want? • How much detail do I need? • How comprehensive do I need to be? • How far back should I search? • The answers to these questions should flow from the reasons for reading. Bruce Davies

Identify Relevant Literature. • There are many ways of finding literature. • Remember to ask a librarian – they are the experts. • Selectivity is the key to successful critical appraisal. Bruce Davies

Critically Appraise What You Read. • Separating the wheat from the chaff. • Time is limited – you should aim to quickly stop reading the dross. • Others contain useful information mixed with rubbish. • Simple checklists enable the useful information to be identified. Bruce Davies

Questions to Ask • Is it of interest? • Why was it done? • How was it done? • What has been found? • What are the implications? • What else is of interest? Bruce Davies

Questions to Ask • Is it of interest? • Title, abstract, source. • Why was it done? • Introduction. • Should end with a clear statement of the purpose of the study. • The absence of such a statement can imply that the authors had no clear idea of what they were trying to find out. • Or they didn’t find anything but wanted to publish! Bruce Davies

Questions to Ask • How was it done? • Methods. • Brief but should include enough detail to enable one to judge quality. • Must include who was studied and how they were recruited. • Basic demographics must be there. • An important guide to the quality of the paper. Bruce Davies

Questions to Ask • What has it found? • Results. • The data should be there – not just statistics. • Are the aims in the introduction addressed in the results? • Look for illogical sequences, bland statements of results. • ? Flaws and inconsistencies. • All research has some flaws – this is not nit picking, the impact of the flaws need to assessed. Bruce Davies

Questions to Ask • What are the implications? • Abstract / discussion. • The whole use of research is how far the results can be generalised. • All authors will tend to think their work is more important than the rest of us! • What is new here? • What does it mean for health care? • Is it relevant to my patients? Bruce Davies

Questions to Ask • What else is of interest? • Introduction / discussion. • Useful references? • Important or novel ideas? • Even if the results are discounted it doesn’t mean there is nothing of value. Bruce Davies

NOT JUST A FAULT FINDING EXERCISE ! Bruce Davies

What Is the Method? • The first task – alternative check lists for different methods. • How was the study conducted to confirm the method? • Authors sometimes use the wrong words to describe their work! Bruce Davies

Surveys • Describe how things are now. • Samples of populations or special groups. • The samples must be randomly selected. Bruce Davies

Surveys • Do not have separate control or comparison groups. • Comparisons may be made between subgroups – but this is not control. • Use of the term survey should identify the method – beware the use in what is really a cohort study. Bruce Davies

Surveys • Cross-sectional is seldom used to describe other methods. • There are many ways of selecting a sample: e.g: stratified, cluster and systematic. Bruce Davies

Cohort Studies • Used to find out what happens to patients. • A group is identified and then watched to see what events befall them. • May have comparison or control groups – who must be identified from the start. • Not an essential feature tho.’ Bruce Davies

Cohort Studies • Must have the element of time flowing forward, from the point at which they are identified. • This is sometimes called a retrospective cohort study. • The term cohort should be diagnostic. Bruce Davies

Clinical Trials • Testing. • Always concerned with effectiveness. • The focus should always be on the outcome. • The outcomes may not be beneficial – in other words side effect trials. • Sometimes cohort trials are used to assess effectiveness. This is very poor research and can usually be dismissed. Bruce Davies

Clinical Trials • When more than two things are compared it makes the study more complex and harder to get right. • The key words to look for are: random allocation, double blind, single blind, placebo-controlled. • The term outcome is sometimes used in cohort studies as well. Bruce Davies

Case-control Studies • Ask what makes groups of patients different. • Select a set of patients with a characteristic – eg a disease. • The characteristics of this set are then compared with a control group who do not have the characteristic being studied – but all all other respects must be as identical as possible. Bruce Davies

Case-control Studies • Case studies look backward – not forward as cohort studies do. • Other terms used are: case-referrent, case-comparator, case-comparison. • May also be called retrospective – as is used for some cohort studies. Bruce Davies

The Results • The major mental challenge. • What do I think this really means? • CAUTION. • Large unexpected results are rare. • Flawed studies and misleading findings are common. Bruce Davies

Statistics • A subject in itself. • Some general thoughts are worth emphasising. Size matters. Bruce Davies

Probability • Is just that. • It is not proof. • Think of horse racing and the lottery. • Think of the odds. Bruce Davies

Pitfalls • All statistical tests make assumptions about the raw data. • If there is no raw data presented you cannot know if the tests are meaningful. • Outliers. Bruce Davies

Pitfalls • Skew. • Non-independence. • Serendipity masquerading as hypothesis – or data trawling! Bruce Davies

Pitfalls • Black box analyses. Modern computers make statistical testing easy – the authors may not know what they are doing! • Bias – play devils advocate. • Confounding. • A very common problem in medicine. • Colour televisions do not cause increases in hypertension. Bruce Davies

Checklists • Checklists for particular types of literature are a quick and easy way of learning critical appraisal. • They all have 3 stages: • Basic questions. • Essential appraisal. • Detailed appraisal. Bruce Davies

BUT, BUT, BUT • Checklists do not tell you about the quality or usefulness. • This is still a subjective question. • All the lists do is enable a more structured and thoughtful response to the subjective question. Bruce Davies

References • The Pocket guide to critical appraisal. • By: Iain Crombie. • Pub BMJ Books, 1996. • How to lie with statistics. • By: Darrell Huff. • Pub: Pelican, 1989. Bruce Davies

Critical Appraisal