Exploring the use of QSR Software for understanding quality - from a research funder’s perspective

1. Exploring the use of QSR Software for understanding quality - from a research funder’s perspective Janice Fong Research Officer Strategies in Qualitative Research - using QSR software 2006 Conference

2. Structure of my presentation • Comment on quality in qualitative research • The Disability Rights Commission as a funder of research • Assessment of quality in a research tender exercise and look at some examples • The relevance of QSR • Conclusion

3. Quality in the process of qualitative data analysis No explicitly agreed standards, different people have different perceptions • However the notion of validity and reliability are thought to be important • Although the meanings of these concepts varies • Information need to be recorded and presented in a transparent way. • Documentation of research process should be clear in order to provide an ‘audit trail’ so that one can assess its reliability and validity.

4. DRC as a research commissioner • DRC commissions research to inform decision making in relation to policy and practice. • Information needs to be robust and rigorous. • There may be belief that quantitative data superior to qualitative data; procedures for managing and analyzing the former are perceived to be more straightforward. • Different methodologies for different purposes. • In general, there is a lack of understanding of methods to assess quality of qualitative research

5. Commissioning research in the DRC • The role of research manager: writing project specification, evaluating tender, drafting contract, managing the research process and communicating with researchers. • Issues of quality often come up when the report is submitted. • However, a high quality report should not be assessed only by the final report, but the whole research process and procedure, right at the beginning when the proposal is drawn.

6. Assessing quality : Some examples Example of a DRC research project put out to tender: • The context of the research specification clearly mentioned the aims and objectives of the research. • However, the methods is far less prescriptive because we want to encourage creativity. • Timeline: six months

7. Tender evaluation criteria • We explicitly requested: an explanation of analysis procedures. • a typical quality for the tender evaluation criteria include: clarity & feasibility of the methodology; experience; staff quality and management; clarity of tender documents, understanding of task, quality control systems, completion within the timescale; understand commission needs and value for money, etc.

8. Proposal One A. Methods • telephone interview • 25 respondents B. Data • Type of data: • Not clear (notes? transcript?) • Volume of data: • Don’t know

9. Proposal One (Con’d) C. Timetable • Interview lasting 4 months • There is no time assigned for the qualitative data analysis at all. D. Project management • The principle investigator takes overall responsibility for project management, including the production and delivery of the final report. • Two other research assistants taking up the fieldwork but there is no mention of how does the team work together and who is going to do the analysis.

10. Proposal One (Con’d) My thoughts • Since the process of qualitative data analysis is not mentioned at all, I could not assess its potential quality on the output. • No time for analysis • What will the data look like? • How much is its volume?

11. Proposal Two A. Methods • Semi-structured individual interviews • 72 respondents B. Data • Type of data: • all interviews will be taped and transcribed. • Volume of data: • Full transcription of individual interview X 72 respondents.

12. Proposal Two (Con’d) C. Timetable • One month for data analysis • Data will be sorted and thematically analysed using “standard approaches to the analysis of qualitative data.” D. Project management • Two principle investigators and their experiences in the fields are mentioned and two part-time research fellows would be employed to undertake data collection. Regular team meetings to ensure on-going co-ordination and collaboration between team members.

13. Proposal Two (Con’d) My thoughts • How data analysis is conducted and by whom? • How can I be confident that the data analysis is going to be consistent and transparent? (because there are 4 people in the team) • Are there in fact standard approaches to the analysis of qualitative data?

14. Proposal Three A. Methods • Case studies and structured questionnaire with some open-ended responses. • 70 respondents B. Data • Type of data: • Case studies, open ended written accounts. • Volume of data: • Written accounts X 70 respondents.

15. Proposal Three (Con’d) C. Timetable • Two months for the analysis with an interim report after the 1st month. • “QSR NVivo 7 will be used…this is a qualitative software package designed to manage data…five randomly selected transcripts will be independently examined by three of the researchers in the team to identify potential ‘nodes’…These transcripts will then be compared and a consensus of ‘nodes’ established…All non-coded data will be examined as an additional check to the validity of the analysis.” D. Project management • The project manager, project co-leads and research staff will from the ‘Operational Management Team’. The OMT will undertake on-going monitoring of the project by measuring progress against key activities as scheduled in the timetable.

16. Proposal Three (Con’d) My thoughts • This proposal has demonstrated to me that the researchers have a plan for qualitative data analysis • I can have some ideas of the amount of data and the details of analysis they are going to do from the proposal • I could also expect that there will be a proper transcription of the written accounts produced because one needs to input the data into NVivo 7 before themes can be drawn out from the coding. • Therefore, I have some confidence that the data will be analyzed more consistently and transparently. • Moreover, the timetable and the project management design give more confidence that some systematic project management is in place to make sure delivery.

17. Relevance of QSR • QSR embeds certain principles in its design which can assist systematic thinking in the process of research. • It can tell us clearly what you are going to do • how you plan to manage different types and volume of data • how you analysis the data through the coding design and how you plan to interrogate the data • how you test and develop theory • It can tell us clearly how you plan to conduct quality checking • It can tell us how you plan to project manage the research when more than one person is involved • Summary: • Information that helps us assess quality can be recorded and reported systemically and transparently

18. Conclusion However, there are also some risks • Software may not necessarily increase the ‘power’ of the study • ‘Broad brush’ statements claiming that data will be analysed using a specific computer program is not sufficient. • Does not in itself explain data analysis and management. • You are the one who needs to analyse and do the thinking • Most importantly, a clear-cut research question and the skill, vision and integrity of the researcher are essential to a high quality analysis of qualitative data