RACcER: Re-use and Archiving of Complex Community Based Evaluation Research

1. RACcER: Re-use and Archiving of Complex Community Based Evaluation Research Irish Qualitative Data Archive Tallaght West Childhood Development Initiative

2. Irish Qualitative Data Archive • Funded by the Irish Government under the Programme for Research in Third Level Institutions (Cycle 4) • Several major datasets being archived, including – • Growing up in Ireland study • Life Histories and Social Change in Ireland in 20th Century Ireland • But... • Limitedawareness/understanding of qualitative archiving in Irish research community • Lack of literature about challenges of archiving qualitative data generated in evaluation research • Re-use & Archiving of Complex Community-based Evaluation Research (RACcER) project explores issues specific to archiving community-based evaluation data, also aims to raise awareness throughout research community

3. RACcER • RACcER co-funded by IRCHSS and CDI: demonstrator project for qualitative archiving • Objectives: • Scope concerns, issues and requirements in qualitative data archiving through interviews • Explore issues and challenges for researchers, participants, funders, potential data users • Test feasibility of archiving evaluation data • Develop protocols and procedures for archiving evaluation data • Research • Interviews with 30+ individuals from policy and research community, including those involved in TWCDI evaluation • Evaluation researchers were all university based

4. Tallaght West Childhood Development Initiative (CDI) • 1 of 3 Prevention and Early Intervention Programmes jointly funded by government & philanthropy • Aims to improve the education and well-being outcomes for children and families in historically disadvantaged areas of Tallaght West • 5 constituent projects: early years, literacy, pro social behaviour, healthy schools, community safety • Major emphasis on evaluation • Mainly quantitative approach to evaluation of RCT interventions, but qualitative process data collected for all constituent projects and overall evaluation

5. Findings – qualitative data gathered in evaluations • Range of qualitative material collected • One-to-one interviews, discussion groups, meeting minutes, site observations, project log books • Variety of participant groups • Service providers, practitioners, school, police, health service personnel, service users (children and parents), community reps • Most material recorded, anonymised and transcribed

6. Findings – archiving policy • Evaluators’ research centres didn’t generally have archiving policies for qualitative material • Range of experience of what happens to qualitative data after project • Destroyed; stored privately but not used; returned to commissioner; archived internally • No evidence of researchers sharing datasets informally • Default ethics position is to destroy after 5 years • In some cases 10 years

7. Findings – archiving experience • No evidence of formal archiving • Limited experience of accessing qualitative archives, • One example for use in teaching • Some had archived quantitative material • Some had accessed quantitative archives • Range of views on resource implications of prepare qualitative material for archiving

8. Findings – issues with archiving • Difficulties of anonymising in small community • Level of anonymisation would reduce re-use value • Perception of limited reuse value of material collected for evaluation purposes • Secondary analysts wouldn’t understand the nuances of evaluation context • What’s the added value of raw material vs. final report? • Lack of awareness/understanding/trust in newly established archive • Compromises fully informed consent • Viability of archive, level of resources queried • Reluctance to ‘hand over’ data to be governed by unknown institution

9. Findings – issues(2) • Consent not sought • Unethical to archive without consent • Retrospective consent not feasible • ‘Ethically dubious’ to archive documents like meeting minutes, no consent • Constraints of ethics committees limit archiving • Loss of control means consent can’t be fully informed • Prospect of archiving reducing quality of the research a major concern • Reduce willingness to participate in research • Constrain what participants willing to share • “There wasn’t a hope in hell of anyone speaking to us unless there was anonymity.” • Archiving seen to equate with loss of anonymity

10. Findings – issues(3) • Wide perception of fundamental difference between evaluation research and broader topics • Necessarily ‘political’ • At its core it is a judgement on the management and delivery of services • Interview concerns 3rd parties, consent not sought of them • Services, jobs, funding are in question, highly sensitive • Usually geographically bounded, problems with anonymisation

11. Findings – issues(4) • Lack of clarity on who can access data and how it could be used generates fear and reduces inclination to archive • Question of what constitutes a ‘bona fide researcher’? – • How would bona fides be established? What assurances? • Issue of policy researchers’ & communities’ eligibility to access • Concern over data being used out of context, ‘twisted’, sensationalising populations, journalists gaining access • Archiving seen as a breach of trust between the researcher and researched

12. Findings – issues(5) • Researchers concerned about being opened up to scrutiny • Sense of archiving being used to open up research to scrutiny makes researchers feel they aren’t trusted • Potential for commissioners/others to access the archive to undermine research when they don’t like what it has said • Influence of that on evaluation research process “And if they’re going to check my work I might as well write what they want to hear.” • PhD/masters students using archived material could reduce valuable learning opportunities gained from fieldwork • Resource implication of archiving • Benefits not worth gain if funders aren’t supporting it

13. Findings – advantages • Potential to explore alternative research questions • Macro level implementation questions across PEIP sites • Language used by different groups in schools • Impact of setting on parents’ willingness to disclose information • Intangible factors in project success such as charisma • Contributing to cumulative knowledge of implementation of children’s services • Facilitate comparative analysis over time or place • Although some qual researchers dispute extent to which comparison is appropriate

14. Findings – advantages (2) • A valuable teaching resource • But disputed by some as being unethical • Increase the standing of qualitative research if it is open to external examination, improved validity and reliability • But concerns about how it could be used in this way • Contribute to corporate memory of commissioning organisation • But is this the purpose of archiving?

15. Findings - risks • Reputational risk for organisations involved at all levels • Particularly if initiatives haven’t been successful • Equally if commissioner is unhappy with research findings • Negative publicity for researched community • User agreement could be breached • Anonymity could be compromised – • Longer term implications for researchers accessing certain communities • But... • Need to consider likelihood of risk occurring and how serious it would be if it happened

16. Findings – mitigating risks • Strict controls on user access • Ethics approval • Sign-off of original researcher on publications • Archive to facilitate collaboration between researchers • As an explicit function of the archive, builds trust • Hierarchy of access, tailored depositor and user agreements • Some accept that time delay on access reduces risk • Others say time doesn’t make a difference, especially in with research on children • Time delays reduce reuse-ability in policy context • ‘Sunset clause’ may also be useful in some cases

17. Other project issues • Data ownership issue in evaluation context was very sensitive • Massive underlying issue in this project – part of the genesis of RACcER • Commissioners claim legal ownership, evaluation teams dispute this, some suggest participants have moral ownership • Underestimated importance of anonymity to research participants • Implications for archiving & reporting–anonymisation burden, higher level unit of analysis • Some participants refused archiving

18. Anticipated project outputs • Report on RACcER research  • Development and dissemination of best practice guidance • Response to concerns identified in research • Workshop with research participants • Raise awareness about archive and provide support • Archiving of CDI internal material • Archiving of RACcER interviews  

19. RACcER Team • Irish Qualitative Data Archive • Jane Gray • Julius Komolafe • Hazel O’Byrne • Aileen O’Carroll • Tallaght West CDI • Tara Murphy