1 / 24

Comparative Policy Evaluation: Constructing Indices

Comparative Policy Evaluation: Constructing Indices. Alison Smith Social Policy University of Edinburgh. Seminar Structure. Using indices in comparative policy analysis: the UNICEF example Building indices Harmonisation of variables Where to find comparative survey data for your own work

kourtney
Download Presentation

Comparative Policy Evaluation: Constructing Indices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Comparative Policy Evaluation: Constructing Indices Alison Smith Social Policy University of Edinburgh

  2. Seminar Structure • Using indices in comparative policy analysis: the UNICEF example • Building indices • Harmonisation of variables • Where to find comparative survey data for your own work • Design your own index!

  3. “UK is accused of failing children” • Headline BBC news 14/2/07 – UK comes bottom of league table for child well-being across 21 industrialised countries • Find the report online: “Child Poverty in Perspective: An Overview of Child Well-Being in Rich Countries”, UNICEF Innocenti Research Centre, Report Card 7 http://www.unicef-icdc.org/

  4. Netherlands Sweden Denmark Finland Spain Switzerland Norway Italy Republic of Ireland Belgium Germany Canada Greece Poland Czech Republic France Portugal Austria Hungary United States United Kingdom Child Well-Being Table

  5. Measurement and Policy (adapted from Innocenti report card 7:3) • ‘to improve something, first measure it’ • ‘measurement serves as the handrail of policy’ • ‘measurement and comparison shows what is achievable in practice’ • ‘comparisons demonstrate that given levels of child well-being are not inevitable but policy-susceptible’

  6. How to Use the Index • As a stand alone piece of research • As a macro level indicator in cross-national multivariate analysis

  7. Child Well-Being in Rich Countries

  8. Dimensions of Variation • Material well-being • Health and safety • Educational well-being • Family and peer relationships • Behaviours and risks • Subjective well-being

  9. Graphical Interface (not provided by UNICEF) • http://esl.jrc.it/dc/child_wb/index.htm • http://esl.jrc.it/dc/mdg_unsd/index.htm

  10. Components of the Dimensions • The dimension scores are derived from information on the components • Each component draws on one or more indicators • The component scores are averaged to arrive at each country’s overall rating

  11. Choice of Indicators • For each indicator, countries are given a score which reveals how far that country stands above or below the OECD average • Where more than one indicator has been used, scores are averaged giving an equal weighting to each indicator • Evaluating indicators: reliability and validity

  12. Building Indices • Indices are composite measures of a concept • What to do with the different elements of the index • Add them • Average ranking • Product scores • Factor scores from factor analysis

  13. Standardising Variables • Creating a common scale to facilitate comparison • A z-score is a unit’s score expressed as the number of standard deviation units that it lies from the mean of the variable • the z-score tells us two things for each unit/country • how far that individual unit’s score is from the mean • the individual unit’s position in the group

  14. Complicating Issues • Interpretation of index score • Equivalence of items • Missing data

  15. Measuring Concepts • Nominal definition • Operational definitions • http://qb.soc.surrey.ac.uk/resources/authors/authors.htm

  16. The Ladder of Abstraction • Index of the concept of child well-being • Dimensions of variation within this concept • Components of each dimension • Indicator measures taken from survey data for each component

  17. Survey Data • Systematic measurement across cases to permit comparison • Cases are collected as units of analysis: individual, firm, country • The logic of survey analysis is that variation in one variable is matched with variations in other variables (following J. S. Mill A System of Logic)

  18. Comparative Data Sets • MTUS - http://www.timeuse.org/mtus/ • CNEF - http://www.human.cornell.edu/che/PAM/Research/Centers-Programs/German-Panel/cnef.cfm • ECHP • ISSP - http://www.issp.org/ • Eurobarometer - http://ec.europa.eu/public_opinion/index_en.htm • European Time Use • LIS - http://www.lisproject.org/

  19. Cross-National Harmonisation of Variables • No international standard for measuring complex concepts like income, education, or employment. • “Researchers interested in doing cross-national work must investigate the institutions, laws and cultural patterns of a country in order to ensure that the variables they create for their analyses are equivalently defined across countries.” CNEF 2000

  20. Question Bank • Secondary analysis of existing data: finding questions that enable you to produce a satisfactory scale • Does not cover all cross-national data sources but good for UK data • http://qb.soc.surrey.ac.uk/

  21. ZA Online Study Catalogue (ZACAT) • http://www.gesis.org/Datenservice/zacat.htm • Access to data and documentation of around 200 studies with a focus on international comparative studies and election studies

  22. Additional Reading Bradshaw, J., Ditch, J., Holmes, H. and Whiteford, P., (1993): “A comparative study of child support in fifteen countries”, Journal of European Social Policy, 3 (4), 255-271. Bruning, G. and Plantenga, J. (1999): “Parental leave and equal opportunities experiences in eight European countries”, Journal of European Social Policy, 9 (3), 195-209. Gornick, J. C., Meyers M. K. and Ross, K. E. (1997): "Supporting the employment of mothers: Policy variation across fourteen welfare states", Journal of European Social Policy, 7, 45-70. Gornick, J. C. and Meyers, M. K. (2003): Families That Work. Policies for Reconciling Parenthood and Employment, Russel Sage Foundation: New York. Smith, A. J. and Williams, D. (2007): "Father friendly legislation and paternal time across Western Europe", Journal of Comparative Policy Analysis, 9 (3), 175-192.

  23. Technical Reading • De Vaus, D. 2002: Surveys in Social Research (5th Edition), Chapter 1: The Nature of Surveys; Chapter 4: Developing Indicators for Concepts; Chapter 12: Overview of analysis; Chapter 11: Building Scales (needs to be read after chapter 12).

  24. Design An Index! • Put together a short presentation about an index you might like to develop. Consider the points below. (Groups of 3: 15 minute presentation) 1. Identify the concept you want to measure in an index (e.g. child well-being, father-friendliness of policy, support for women’s employment, any other topic of interest to you) 2. Develop a nominal definition of this concept 3. Identify relevant dimensions of the concept (not more than 4 for today’s exercise) 4. Specify appropriate indicators for each dimension (operational definitions). Reflect upon the validity and reliability of the indicators. 5. Which data sets might you use? 6. How would you construct your index?

More Related