1 / 37

Quantitative Research Methods Day 1 Research, Quantification and Outcome

Quantitative Research Methods Day 1 Research, Quantification and Outcome. Vaughan Bell. Course Outline – Day 1. Lecturer: Dr Vaughan Bell Why quantitative research? Psychometric scales and outcome Describing data and testing ideas Research on assessing psychotherapy outcome.

hans
Download Presentation

Quantitative Research Methods Day 1 Research, Quantification and Outcome

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Quantitative Research Methods Day 1 Research, Quantification and Outcome Vaughan Bell

  2. Course Outline – Day 1 • Lecturer: Dr Vaughan Bell • Why quantitative research? • Psychometric scales and outcome • Describing data and testing ideas • Research on assessing psychotherapy outcome

  3. Course Outline – Day 2 • Lecturer: Dr Katerina Fotopoulou • Quick revision of quantitative research • Evaluating the quantitative approach • Strengths and weaknesses vs qualitative research

  4. Course Outline – Day 3 • Lecturer: Dr Vaughan Bell • Randomised controlled trials • Publication issues

  5. Two Important Types of Evidence • Evidence from clinical intuition • Evidence from research • In good clinical practice, each should inform the other.

  6. What is Research? • Pretty much anything, but… • Good research should have a number of features: • Contribution to knowledge – not wasting your and the participants’ time and resources • Methodical – choosing the right tool for the job • Rigorous – using the tool correctly

  7. Quantitative Research • Quantitative research relies on several assumptions. • Perhaps the most important is that the thing your studying can be meaningfully quantified. • This is particularly important in psychology… • …where there is an ongoing debate about whether human experience or attributes can be meaningfully represented as numbers.

  8. Cognitive Psychology • Quantification is now central to psychology because of the dominance of cognitive psychology. • Cognitive psychology has some core features: • Information processing view of the mind. • Positivist / scientific approach to research • Experimental • Quantitative • Reductionist tendency • Aim for objectivity

  9. Information Processing • To varying degrees, cognitive psychology seeks to explain the mind in terms of information processing. • Pylyshyn (1979, p435) argues that “computation is not a metaphor but part of a literal description of cognitive activity”. • Parkin (2000) argues that cognitive psychologists use information processing purely as an analogy for mental function.

  10. Has this become a dogma ? • Still and Costall (1987) would agree and have argued that cognitive psychology has become “restrictive as well as complacent”. • Even Don Norman (1981) has criticized cognitive psychology for “sterility” and for descriptions which “do not fit actual behaviour”. • Criticisms of cognitive psychology come from two main sources: Phenomenology and Algorithmic Information Theory

  11. Meaningful Quantification • Whether you buy into these arguments or not, there is no doubt that quantification has proved useful. • The crucial question is “is quantification useful for what I am doing / trying to understand?” • Which brings us back to the question “what can be meaningfully quantified?” • Outcome is a major focus of quantitative methods.

  12. Outline • Part1 • What is outcome? • Types of measure • What attributesdoes a good measure need ? • Possible confounding factors and caveats. • Ethical issues • Part 2 • Workshop on creating / assessing measures

  13. What is Outcome ? • In the schizophrenia literature (Brekke et al, 1993), outcome is classified in three ways: • Clinical outcome: signs, symptoms, service utilisation. • Functional outcome: social, vocational, independent living. • Subjective outcome: patient’s experiences of outcome. • It is important to be clear on what sort of outcome you want to assess and why.

  14. Types of Measure • Clinical metrics: length of hospital stay, readmissions, number of sessions with therapist. • Psychometric scales: quantifies some aspects of psychological function or behaviour. • Self-report • Structured / semi-structured interview • Multi-rater measures: multiple people are asked to rate the same material. • Tasks: e.g. experimental / neuropsychogical tests • Physiological measures: e.g. brain scans

  15. Purpose • Screening tool: Determines whether a symptom or psychological trait is present. • Aid to diagnosis: typically increases the reliability of ‘bedside’ diagnoses. • Quantification: allows symptoms or traits to be quantified by intensity, duration etc. • Dimensional scale: measures an attitude or trait through its range in the population.

  16. Attributes of a Good Measure • The two essential features of a good measure, are: • Reliability: measures consistently. • Validity: measures what it is designed for. • Each of these have various sub-categories that need to be fulfilled.

  17. Reliability • Reliability must be established first, as validity relies on it. • i.e. A reliable scale could measure nonsense, but do so consistently… • …but it is impossible for a measure to be inconsistent and measure what it is designed for.

  18. Reliability • Common forms… • For psychometric scales: • Internal reliability – “are similar items answered in similar ways ?” • Test-retest reliability – “does the test produce similar results when used on the same people on different occasions ?” • For multi-rater observational measures: • Inter-rater reliability – “does the measure produce similar results when used by different observers”

  19. Internal Reliability “Are similar items answered in similar ways?” • Typically tested with Cronbach’s Alpha. • An alpha above 0.7 is usually considered satisfactory (Kline, 1993) • If a measure has multiple independent factors, they may need to be tested separately.

  20. Test-Retest Reliability “Does the test produce similar results when used with the same people on different occasions?” • Typically tested with Pearson correlation. • Results from first occasion are correlated with results from second occasion. • Correlation should be above 0.8 (Kline, 1993) • Assumes that the object of measure is stable between occasions. • Therefore, this is usually tested on non-clinical groups.

  21. Inter-rater Reliability “Does the measure produce similar results when used by different observers?” • Typically tested with Cohen’s Kappa. • Cohen’s Kappa controls for problems with directly comparing multiple raters’ scores. • e.g. one rater consistently scoring five-points more than the other will correlate despite not agreeing. • A two point rating (e.g. symptom present / absent) will have 25% agreement just by chance.

  22. Cohen’s Kappa • Kappa values and level of agreement between raters (Landis and Koch, 1977): • Fair: 0.21 - 0.40 • Moderate: 0.41 - 0.60 • Substantial: 0.61 - 0.80 • Almost perfect: 0.81 - 1.00

  23. Validity • Face validity – “does the perception of the measure influence the outcome ?” • Content validity – “does the measure cover everything it needs to cover ?” • Construct validity – “do the results of the measure agree with what theory predicts ?” • Criterion validity – “does the measure fulfil expected criteria ?” – usually the performance of a certain group • Incremental validity – “does it measure anything new ?”

  24. Face Validity “does the perception of the measure influence the outcome?” • If participants or testers misperceive the nature of the measure, it may affect the results. • e.g. if someone takes a verbal memory test but thinks it is a creative thinking test, it may look like they are confabulating. • Similarly, asking ‘are you depressed ?’ may have good face validity for depression • …but asking ‘are you deluded ?’ has poor face validity for delusions.

  25. Content Validity “does the measure cover everything it needs to cover?” • i.e. it is comprehensive ? • If a measure of anxiety asks only about social anxiety, it doesn’t have good content validity. • This is usually assessed by comparing, or generating the measure, based on • Known phenomena • Literature reviews

  26. Construct Validity “do the results of the measure agree with what theory predicts?” • There are two ways of assessing this: • Convergence – correlates with measures of things known to be associated • Divergence – negatively correlates with things known to be mutually exclusive. • e.g. a good measure of depression should correlate with a measure of low mood • …but negatively correlate with a measure of self-esteem.

  27. Construct Validity Or, a good measure or anxiety should predict performance, as per the Yerkes-Dodson Law (1908)

  28. Criterion Validity “does the measure fulfil certain criteria?” • Often, this can be the same as construct validity (e.g. correlates with similar measures) • It is often tested by asking members of a certain group to take the test… • …who are known to have high levels of the measure being attributed. • e.g. people with psychosis should score higher than the general population on a good measure of anomalous perceptual experience.

  29. Incremental Validity “does it measure anything new?” • An assessment of what the measure adds to the ‘toolkit’ of psychological assessment. • If it measures exactly the same as something else, in the same way… • …there may not be any point in developing it.

  30. Wider Validity Issues • Rarely tackled in the textbooks, but it is important to assess how the validity tests were carried out. • Has validity been established for: • different cultural groups ? • different ages ? • someone with a disability ? • etc.

  31. Other Ongoing Changes • Particularly in the clinical environment, there may be a number of difficult-to-disentangle influences. • Children present a particular challenge as it might be difficult to separate the effects of: • Cognitive development • Psychopathology • Treatment • Task engagement / motivation

  32. Covariates • One way of controlling for this is to use covariates in statistical analysis, particularly ANOVA. • e.g. I want to see if my Special Brain Power Training™ boosts children’s intelligence. • I gave class A the training and see if they score more highly on an IQ test than class B. • However, class A are, on average, older, so they are likely to do better anyway. • I introduce age as a covariate into the analysis, to cancel-out its effects and make a fairer comparison.

  33. Confounding Factors • Other factors can sometimes be more difficult to deal with: • Floor / ceiling effects: Where the test is so hard or easy that it is not possible to differentiate between participants. • Therefore, it is important to pilot the measure, and compare performance with norms. • Emotional impact: Testing can be stressful for healthy individuals. • For patients, especially so, particularly if they see themselves as ‘failing’ the tests.

  34. Ethical Issues • There are distinct ethical (and potentially legal) implications when using such tests, e.g.: • Are you using the measure for clinical decisions ? • If so, are you qualified and competent to develop / deploy / interpret the measure ? • If not, do you have adequate supervision from someone who is ?

  35. Ethical Issues • Is the measure being used for research? • If so, has the research been given ethical approval? • Are you asking for informed consent ? • Does the patient know this is not part of their standard care ? • Are the results being kept private or anonymised ?

  36. Conclusions • The purpose of measurement and the type of measure are crucial. • Measures need to be reliable and valid. • You need to be aware of possible confounding factors. • You need to be clear whether the context is research or clinical… • …and know and abide by the ethics of each.

  37. Exercise • Completing the Magical Ideation Scale • Data analysis and discussion • Exercise in creating a measure of childhood psychosis-like experience

More Related