What is ‘secondary analysis’?

DTC Quantitative Methods Data Sources: Secondary Analysis (& Official Statistics) Thursday 16th January 2014

What is ‘secondary analysis’? • Hakim: “…any further analysis of an existing dataset which presents interpretations, conclusions or knowledge additional to, or different from, those presented in the first report on the inquiry as a whole and its main results”. • Dale et al.: “secondary analysis implies a re-working of data already analysed”. • Hyman: “the extraction of knowledge on topics other than those which were the focus of the original surveys”. Online course extracts: Dale et al. 1988; Dale et al. 2008

Sources for secondary analyses • Surveys • The Census • Administrative and/or public records • Longitudinal studies • (Qualitative studies) The UK Data Archive (http://www.data-archive.ac.uk) now catalogues data from surveys and qualitative studies, as well as the Census, historical data, international country-level databases, etc.

Some sources specifically geared towards secondary analysis • The British Social Attitudes Survey • Understanding Society http://www.understandingsociety.org.uk/ • The Timescapes qualitative longitudinal study: http://www.timescapes.leeds.ac.uk/

Benefits of secondary analysis • It avoids costs in money and time that would make primary research impractical, especially for a lone researcher. • It allows one to benefit from the fieldwork expertise of professional organizations. • Cross-national and historical research become more of a practical possibility. • Secondary analyses of longitudinal data facilitate studies of change over time. • Large, nationally representative samples facilitate sophisticated, generalisable analyses, and (sometimes) analyses relating to small/relatively inaccessible minorities.

Bridging the quantitative/qualitative ‘divide’ • Dale et al. comment that “qualitative research can greatly enhance the value of secondary analysis by providing greater depth of information, particularly by suggesting the underlying processes that are responsible for the observed relationships”.

Why is there a shortfall in secondary analyses in the UK? (particularly in some disciplines, e.g. Sociology) • A lack of quantitatively-orientated researchers. • The legacy of critiques of quantitative research methods. • More specifically, the legacy of critiques of official statistics. • Although it’s now more of a question of inertia than of ongoing scepticism?

Themes within critiques of official statistics • Concerns about coverage • Concerns about measurement • Epistemological concerns • ‘Political concerns’

A ‘damning’ quote? “It’s [i.e. the state’s] economic and political functions are embedded in the production of official statistics, structuring both what data are produced and how this is done... only by understanding that statistics are produced as part of the administration and control of a society organised around exploitative class relations can we grasp their full meaning” (Miles and Irvine, 1979).

However… • Analyses of official data can produce substantively interesting results • The producers and users of official statistics are normally very concerned about the errors in data and the data’s limitations, • The conceptual issues arising from the use of official statistics are not dissimilar to those arising in other forms of sociological research. • Analyses of official data have been used to critique governments with respect to issues such as unemployment, health inequalities, etc. (The first three of the above bullet points are suggestions by Bulmer)

... nevertheless As Hindess commented: • “Official statistics are never mere givens to be taken as they are or else dismissed as inadequate. Like other productions they must be explained in terms of the conditions and instruments of their production”. • “As structured social products they [i.e. official statistics] can [and should be!] be critically assessed”.

Official statistics or official data? • Published official statistics have justifiably been viewed with some scepticism. • However, the analysis of official data by a secondary analyst can avoid some of the problems. • Given access to the ‘raw’ official data, she or he can manipulate them in ways different to how they were processed to produce published official statistics.

Are UK official statistics getting more independent? • A Statistics Board resulting from the Statistics Bill of July 2007, renamed the UK Statistics Authority in February 2008 (see: http://www.statisticsauthority.gov.uk/) is: • “... an independent body operating at arm's length from government as a non-ministerial department, directly accountable to Parliament. … [its] overall objective is to promote and safeguard the quality of official statistics that serve the public good. It is also required to safeguard the comprehensiveness of official statistics”.

Some more specific developments: I • OPCS [now ONS] Disability Surveys were criticised for not adequately reflecting disabled people’s perspectives on their disabilities. (see Abberley’s chapter in Levitas and Guy, 1996). • However, they were nevertheless used for some interesting and useful secondary analyses (see Pole and Lampard, 2002, Ch. 7). • More recently, the Office for Disability Issues (ODI) brought together a group of disabled people as a reference network, in part to facilitate the effective design of a new longitudinal disability survey, the Life Opportunities Survey (LOS): (see http://www.ons.gov.uk/about/surveys/a-z-of-surveys/life-opportunities-survey/index.html)

Some more specific developments: II • A number of UK government surveys now (since 2009) ask a question on sexual orientation, following a question being asked in the 2007 Citizenship Survey (and resulting from ONS's Sexual Identity Project, established in 2006). • This development reflects more general governmental concerns about the availability of ‘equality data’. • However, it does not seem that a question will be asked on this topic in the 2011 Census! (and it wasn’t…) • The consensus also seems to be that the results generated by the question will under-estimate non-heterosexual orientations.

“What is the moral? Must have a moral…” • Whether the source of their data is official or non-official, secondary analysts should gain an extensive knowledge of the research design and data collection process. • This allows the secondary analyst to adopt an informed and suitably critical approach to their assessment of the validity and value of their data source(s). The data source for the slide title (I think) is “A Funny Thing Happened on the Way to the Forum” (Sondheim)

Key issues in secondary analysis(according to Dale et al.) • What was the original purpose of the study and what conceptual framework was used? Who was responsible for collecting the data? • What data did the study collect and how were variables such as occupational class operationalized? • What was the sample design that was used and what was the level and pattern of non-response?

Is there documentation available in relation to? • Sample selection • Patterns of (non-)response • Interview schedules [Questionnaires] • Instructions to interviewers • The coding of answers • The construction of derived variables

Some other relevant questions... • Is secondary analysis an appropriate approach given the researcher’s objectives? • Does the secondary analyst know the topic area well enough to be able to interpret and evaluate the information available? • What similarities and differences are there between the conceptual frameworks of the original researchers and of the secondary analyst? • Are the data recent and extensive enough for the secondary analyst’s purposes? • How consistent is the information with information from other sources? • Is the information representativeness enough to support generalisations? Is weighting needed to correct for a lack of representativeness?

An example: the General Household Survey Advantages of the GHS include • a large sample size • the fact that it has been repeated more or less annually since the early 1970s, which allows trends to be examined • a broad agenda which means that relationships between concepts belonging to different policy areas can be examined • a hierarchical structure, which allows linkages between different members of the same household to be examined (Dale et al.)

Disadvantages of the GHS include • That it no longer exists, although some of its components have been reassigned to other, less satisfactory surveys...

Some examples of sources and issues from Richard’s research • Social Change and Economic Life Initiative (SCELI) main survey (1986) Representativeness • National Survey of Sexual Attitudes and Lifestyles II (2000) [and various other couple-related surveys] Comparability of measures • General Household Survey (1991 & 2005) Conceptual coverage/validity of measures • British Election Study (1987) Validity of measures See also Pole and Lampard, 2002, Ch. 7.

Reasons for the end of a cohabiting or marital relationship (as shown on a NATSAL II showcard) • Unfaithfulness or adultery • Money problems • Difficulties with our sex life • Different interests, nothing in common • Grew apart • Not having children • Lack of respect or appreciation • Domestic violence • Arguments • Not sharing household chores enough • One of us moved because of a change in circumstances (for example, changed jobs) • Death of partner • Another reason (please say what)

...and the categories that had to be added • Drink, drugs or gambling problem • Mental health or related problem • Problem with children/step-children • Never at home (e.g. always out with friends) • Problems with parents/in-laws/family • Age-related problems (e.g. big age difference) • Another relationship involved • Lived in/moved to a different country/area • Still in relationship, but stopped living together • Change of mind/feelings/personality • Partner just left without any explanation

What is ‘secondary analysis’?