1 / 18

(Don’t) be afraid of the conceptual validity

(Don’t) be afraid of the conceptual validity. Introduction. ESCOP stresses the importance of providing proof of quality But what is quality ? Quality statistics provides evidence for something that is correct and true This is philosophy !. Typology of validity & ESCOP.

rollo
Download Presentation

(Don’t) be afraid of the conceptual validity

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. (Don’t) be afraid of the conceptual validity Gró Einarsdóttir, Ph.D. Statistics Iceland Gro.Einarsdottir@hagstofa.is

  2. Introduction • ESCOP stresses theimportance of providingproof of quality • Butwhatis quality? • Qualitystatisticsprovidesevidence for somethingthat is correct and true • This is philosophy!

  3. Typology of validity & ESCOP • Statistical conclusion validity • Internal validity • External validity • Construct validity construct measure Source: Cook et al. (2002)

  4. Typology of validity & ESCOP • Statistical conclusion validity • Internal validity • External validity • Construct validity Is the measured relationship a true relationship? construct measure Source: Cook et al. (2002)

  5. Typology of validity & ESCOP • Statistical conclusion validity • Internal validity • External validity • Construct validity Is the measured relationship a true relationship? construct measure Source: Cook et al. (2002)

  6. Typology of validity & ESCOP • Statistical conclusion validity • Internal validity • External validity • Construct validity Is the measured relationship a true relationship? Does A cause B? construct measure Source: Cook et al. (2002)

  7. Typology of validity & ESCOP • Statistical conclusion validity • Internal validity • External validity • Construct validity Is the measured relationship a true relationship? Does A cause B? construct measure Source: Cook et al. (2002)

  8. Typology of validity & ESCOP • Statistical conclusion validity • Internal validity • External validity • Construct validity Do theresults generalize? Is the measured relationship a true relationship? Does A cause B? construct measure Source: Cook et al. (2002)

  9. Typology of validity & ESCOP • Statistical conclusion validity • Internal validity • External validity • Construct validity Do theresults generalize? Is the measured relationship a true relationship? Does A cause B? construct measure Source: Cook et al. (2002)

  10. Constructvalidity • ESCOP • Many ESCOP principlesseemtoaddressconstructvalidity • Closerlookshowsthatremediesarereformulations of reliabilitymeasures • Exclusivefocusonreliabilitycanleadustomeasurethewrongthingwithgreatconsistency and precision • Do wemeasurewhatweclaim to be measuring? construct measure Source: Cook et al. (2002), ESCOP (2017)

  11. Constructvaliditycasestudy • The Icelandic social indicators systemhasbeenpublishedsince 2012 • Between45-67 indicators for differentyears • Differentdimensions for differentyears • demography, equality, sustainability, health, social cohesion, living standards and welfare, education, employment, children • The same indicators have been categorised into different dimensions • These changes in dimensions and indicators have not been well documented, justified, or explained through the years. Statistics Iceland. 2019 a. Revision of Social Indicators. Statistical Series – Working Papers, 104 (1), p. 1- 14. ISSN 1670-4770

  12. Conceptual diagram Statistics Iceland. 2019 a. Revision of Social Indicators. Statistical Series – Working Papers, 104 (1), p. 1- 14. ISSN 1670-4770

  13. The dimensions of social well-being Statistics Iceland. 2019 a. Revision of Social Indicators. Statistical Series – Working Papers, 104 (1), p. 1- 14. ISSN 1670-4770

  14. Quality criteria for indicators • The unit of measurement is people • The indicator measures well-being outcomes that are directly important for people, and not the cause or consequences of these outcomes • Has the potential to change and be influenced by policy • Generally accepted measurement that is widely used • Repeated measurement with a timeline • A timely measurement • Provides the possibility to look at subgroups and distributions • A precise measurement • A conceptually valid measure of the dimensions. Indicators should be chosen to adequately cover the dimension with as few measurements as possible. Ideally, 3-5 indicators should measure each dimension. • That Statistics Iceland can guarantee the quality of the measurement and that it can be enriched with registry and population data • The social indicators meet the needs of the users Statistics Iceland. 2019 a. Revision of Social Indicators. Statistical Series – Working Papers, 104 (1), p. 1- 14. ISSN 1670-4770

  15. Lessons • Thecasestudydescribesa method for improvingfacevalidity of an indicatorsystem • Facevalidity is supportedbyemergingconsensusamongstatisticsoffices and followingexplicitqualitycriteria • Therearehoweverothermethodsthatcanbeusede.g. • Cronbach alpha, Omega or greatest lower bound (GLB) • Exploratoryfactor analysis (EFA) • Confirmatoryfactor analysis (CFA)

  16. Pitfalls • Addressing one type of validity often introduces trade-offs with other types of validity • Construct confounding: Measurement of an entity typically consists of multiple constructs and we may fail to describe the entire construct

  17. Take home message • Theinterest is often not in thestatistic, but in theconstructthat it represents • Statisticsofficesneedtobeaware of this and explicitlyevaluatetheconstructvalidity of statisticaloutput and aimtoimprove it

  18. Thankyou! Gro.Einarsdottir@hagstofa.is

More Related