Quantitative Research Methods Day 1 Research, Quantification and Outcome

1. Quantitative Research Methods Day 1 Research, Quantification and Outcome Vaughan Bell

2. Course Outline – Day 1 • Lecturer: Dr Vaughan Bell • Why quantitative research? • Psychometric scales and outcome • Describing data and testing ideas • Research on assessing psychotherapy outcome

3. Course Outline – Day 2 • Lecturer: Dr Katerina Fotopoulou • Quick revision of quantitative research • Evaluating the quantitative approach • Strengths and weaknesses vs qualitative research

4. Course Outline – Day 3 • Lecturer: Dr Vaughan Bell • Randomised controlled trials • Publication issues

5. Two Important Types of Evidence • Evidence from clinical intuition • Evidence from research • In good clinical practice, each should inform the other.

6. What is Research? • Pretty much anything, but… • Good research should have a number of features: • Contribution to knowledge – not wasting your and the participants’ time and resources • Methodical – choosing the right tool for the job • Rigorous – using the tool correctly

7. Quantitative Research • Quantitative research relies on several assumptions. • Perhaps the most important is that the thing your studying can be meaningfully quantified. • This is particularly important in psychology… • …where there is an ongoing debate about whether human experience or attributes can be meaningfully represented as numbers.

8. Cognitive Psychology • Quantification is now central to psychology because of the dominance of cognitive psychology. • Cognitive psychology has some core features: • Information processing view of the mind. • Positivist / scientific approach to research • Experimental • Quantitative • Reductionist tendency • Aim for objectivity

9. Information Processing • To varying degrees, cognitive psychology seeks to explain the mind in terms of information processing. • Pylyshyn (1979, p435) argues that “computation is not a metaphor but part of a literal description of cognitive activity”. • Parkin (2000) argues that cognitive psychologists use information processing purely as an analogy for mental function.

10. Has this become a dogma ? • Still and Costall (1987) would agree and have argued that cognitive psychology has become “restrictive as well as complacent”. • Even Don Norman (1981) has criticized cognitive psychology for “sterility” and for descriptions which “do not fit actual behaviour”. • Criticisms of cognitive psychology come from two main sources: Phenomenology and Algorithmic Information Theory

11. Meaningful Quantification • Whether you buy into these arguments or not, there is no doubt that quantification has proved useful. • The crucial question is “is quantification useful for what I am doing / trying to understand?” • Which brings us back to the question “what can be meaningfully quantified?” • Outcome is a major focus of quantitative methods.

12. Outline • Part1 • What is outcome? • Types of measure • What attributesdoes a good measure need ? • Possible confounding factors and caveats. • Ethical issues • Part 2 • Workshop on creating / assessing measures

13. What is Outcome ? • In the schizophrenia literature (Brekke et al, 1993), outcome is classified in three ways: • Clinical outcome: signs, symptoms, service utilisation. • Functional outcome: social, vocational, independent living. • Subjective outcome: patient’s experiences of outcome. • It is important to be clear on what sort of outcome you want to assess and why.

14. Types of Measure • Clinical metrics: length of hospital stay, readmissions, number of sessions with therapist. • Psychometric scales: quantifies some aspects of psychological function or behaviour. • Self-report • Structured / semi-structured interview • Multi-rater measures: multiple people are asked to rate the same material. • Tasks: e.g. experimental / neuropsychogical tests • Physiological measures: e.g. brain scans

15. Purpose • Screening tool: Determines whether a symptom or psychological trait is present. • Aid to diagnosis: typically increases the reliability of ‘bedside’ diagnoses. • Quantification: allows symptoms or traits to be quantified by intensity, duration etc. • Dimensional scale: measures an attitude or trait through its range in the population.

16. Attributes of a Good Measure • The two essential features of a good measure, are: • Reliability: measures consistently. • Validity: measures what it is designed for. • Each of these have various sub-categories that need to be fulfilled.

17. Reliability • Reliability must be established first, as validity relies on it. • i.e. A reliable scale could measure nonsense, but do so consistently… • …but it is impossible for a measure to be inconsistent and measure what it is designed for.

18. Reliability • Common forms… • For psychometric scales: • Internal reliability – “are similar items answered in similar ways ?” • Test-retest reliability – “does the test produce similar results when used on the same people on different occasions ?” • For multi-rater observational measures: • Inter-rater reliability – “does the measure produce similar results when used by different observers”

19. Internal Reliability “Are similar items answered in similar ways?” • Typically tested with Cronbach’s Alpha. • An alpha above 0.7 is usually considered satisfactory (Kline, 1993) • If a measure has multiple independent factors, they may need to be tested separately.

20. Test-Retest Reliability “Does the test produce similar results when used with the same people on different occasions?” • Typically tested with Pearson correlation. • Results from first occasion are correlated with results from second occasion. • Correlation should be above 0.8 (Kline, 1993) • Assumes that the object of measure is stable between occasions. • Therefore, this is usually tested on non-clinical groups.

21. Inter-rater Reliability “Does the measure produce similar results when used by different observers?” • Typically tested with Cohen’s Kappa. • Cohen’s Kappa controls for problems with directly comparing multiple raters’ scores. • e.g. one rater consistently scoring five-points more than the other will correlate despite not agreeing. • A two point rating (e.g. symptom present / absent) will have 25% agreement just by chance.

22. Cohen’s Kappa • Kappa values and level of agreement between raters (Landis and Koch, 1977): • Fair: 0.21 - 0.40 • Moderate: 0.41 - 0.60 • Substantial: 0.61 - 0.80 • Almost perfect: 0.81 - 1.00

23. Validity • Face validity – “does the perception of the measure influence the outcome ?” • Content validity – “does the measure cover everything it needs to cover ?” • Construct validity – “do the results of the measure agree with what theory predicts ?” • Criterion validity – “does the measure fulfil expected criteria ?” – usually the performance of a certain group • Incremental validity – “does it measure anything new ?”

24. Face Validity “does the perception of the measure influence the outcome?” • If participants or testers misperceive the nature of the measure, it may affect the results. • e.g. if someone takes a verbal memory test but thinks it is a creative thinking test, it may look like they are confabulating. • Similarly, asking ‘are you depressed ?’ may have good face validity for depression • …but asking ‘are you deluded ?’ has poor face validity for delusions.

25. Content Validity “does the measure cover everything it needs to cover?” • i.e. it is comprehensive ? • If a measure of anxiety asks only about social anxiety, it doesn’t have good content validity. • This is usually assessed by comparing, or generating the measure, based on • Known phenomena • Literature reviews

26. Construct Validity “do the results of the measure agree with what theory predicts?” • There are two ways of assessing this: • Convergence – correlates with measures of things known to be associated • Divergence – negatively correlates with things known to be mutually exclusive. • e.g. a good measure of depression should correlate with a measure of low mood • …but negatively correlate with a measure of self-esteem.

27. Construct Validity Or, a good measure or anxiety should predict performance, as per the Yerkes-Dodson Law (1908)

28. Criterion Validity “does the measure fulfil certain criteria?” • Often, this can be the same as construct validity (e.g. correlates with similar measures) • It is often tested by asking members of a certain group to take the test… • …who are known to have high levels of the measure being attributed. • e.g. people with psychosis should score higher than the general population on a good measure of anomalous perceptual experience.

29. Incremental Validity “does it measure anything new?” • An assessment of what the measure adds to the ‘toolkit’ of psychological assessment. • If it measures exactly the same as something else, in the same way… • …there may not be any point in developing it.

30. Wider Validity Issues • Rarely tackled in the textbooks, but it is important to assess how the validity tests were carried out. • Has validity been established for: • different cultural groups ? • different ages ? • someone with a disability ? • etc.

31. Other Ongoing Changes • Particularly in the clinical environment, there may be a number of difficult-to-disentangle influences. • Children present a particular challenge as it might be difficult to separate the effects of: • Cognitive development • Psychopathology • Treatment • Task engagement / motivation

32. Covariates • One way of controlling for this is to use covariates in statistical analysis, particularly ANOVA. • e.g. I want to see if my Special Brain Power Training™ boosts children’s intelligence. • I gave class A the training and see if they score more highly on an IQ test than class B. • However, class A are, on average, older, so they are likely to do better anyway. • I introduce age as a covariate into the analysis, to cancel-out its effects and make a fairer comparison.

33. Confounding Factors • Other factors can sometimes be more difficult to deal with: • Floor / ceiling effects: Where the test is so hard or easy that it is not possible to differentiate between participants. • Therefore, it is important to pilot the measure, and compare performance with norms. • Emotional impact: Testing can be stressful for healthy individuals. • For patients, especially so, particularly if they see themselves as ‘failing’ the tests.

34. Ethical Issues • There are distinct ethical (and potentially legal) implications when using such tests, e.g.: • Are you using the measure for clinical decisions ? • If so, are you qualified and competent to develop / deploy / interpret the measure ? • If not, do you have adequate supervision from someone who is ?

35. Ethical Issues • Is the measure being used for research? • If so, has the research been given ethical approval? • Are you asking for informed consent ? • Does the patient know this is not part of their standard care ? • Are the results being kept private or anonymised ?

36. Conclusions • The purpose of measurement and the type of measure are crucial. • Measures need to be reliable and valid. • You need to be aware of possible confounding factors. • You need to be clear whether the context is research or clinical… • …and know and abide by the ethics of each.

37. Exercise • Completing the Magical Ideation Scale • Data analysis and discussion • Exercise in creating a measure of childhood psychosis-like experience