Data sources and collection methods

1. Data sources and collection methods Ken Mease Cairo, June 2009

2. What types of Data? A thorough assessment may well include: • Archival and secondary data • Survey data • Quantitative and qualitative approaches and data • It also will likely include de jure and de facto information

3. Qualitative and Quantitative data • There are basically two types of data: qualitative and quantitative • Qualitative data are usually text or words and quantitative data - numbers • Qualitative approaches, if conducted in a rigorous manner, require more skill than most quantitative approaches

4. Qualitative Data and Approaches • Qualitative approaches provide text data, but increasingly audio, video and images • They are more time consuming to analyze • Text management software, such as NVIVO, AtlasTI and AnSWR (available free at http://www.cdc.gov) • Coding is a very subjective process and open to various problems, such as investigator bias or a lack of inter-coder reliability

5. Quantitative Data and Approaches • Quantitative approaches provide increased rigor by investigating relationships at known levels of probability • They are easier to analyze because researchers use standard, replicable techniques • Common software includes SPSS, Stata and SAS

6. Country reports/desk studies Cross-country (region)comparative surveys Expert assessments Government data Household surveys Mass opinion surveys Key Stakeholder Surveys Media Indicators Business surveys Sources and Types of Governance Data

7. Combining different sources and methods • Combining methodologies and types of data often provides the most useful results. • Combining archival information and administrative data with original qualitative and quantitative survey data allows for triangulation • This approach can increase the level of professionalism, credibility and legitimacy.

8. Administrative, archival and secondary These data, both qualitative and quantitative, objective, reported events, perception and proxies come from a variety of sources: • Narrative reports, administrative data and other information routinely collected by government ministries • The constitution, laws, policies and legislation • Statistics and data gathered by NGOs, international organizations and academics.

9. Data Mapping • It is the best way to identify existing data - accessibility, quality and gaps • One tool available to assist with this process is the IMF’s Data Quality Assessment Framework • A senior academic conducted the data mapping exercise in Zambia and was very valuable

10. Survey Data • Qualitative and Quantitative • Sample sizes can range from 20 to 20,000 or more • Costs can range accordingly • There are a range of options for who collects the data – independent surveys firms, academics, NSOs • Perception and reported events data

11. Different Types of Interviewing • Structured interviews use an identical instrument for each respondent, • Interviewers are trained and have explicit instructions. • This technique uses primarily structured questions with fixed response sets and very few open-ended questions

12. Types of Interviewing • Semi-structured interviews use a written list of questions that need to be covered in a particular order • The questions are often developed from informal discussions and focus groups. • They can include open-ended and/or more structured questions • Ideal when working with elites, managers, bureaucrats and other people who have limited time

13. Types of surveys and data collection methods • Face-to-face data collection is likely the best option in most developing countries • It is also the most expensive and time-consuming • It requires professional management of trained interviewers, sampling and other aspects of the study

14. Mail surveys • Work well only if the postal system is reliable • The questionnaire must be carefully designed for self-administration, and there should not be too many language issues • The cost is usually quite reasonable • It misses the homeless and other vulnerable groups who may not have a valid mailing address

15. Internet surveys • Require special planning and design • Very good for certain populations • These surveys can suffer from low response rates. • Developing and implementing internet surveys has become very reasonable in terms of cost • Very reasonably priced internet services are available - Survey Gizmo at www.surveygizmo.com

16. Telephone surveys • May have coverage issues in most countries. • In many countries, people do not have a phone in their home, and if they do have a phone, it is usually a mobile or cell phone • Most cell phone providers do not make telephone numbers available for use in random-digit-dial surveys.

17. Focus groups • Can generate information about the background conditions surrounding governance issues. • Focus groups are usually efficient in terms of time and money. • They are highly participatory and have the potential of generating solutions to the problems identified by the group members • It is very demanding and requires highly skilled coordinators.

18. Focus groups • Make individual ratings insignificant • Accuracy can suffer, as some individuals may not feel comfortable to speak up in public, while others are hard to keep quiet • Focus groups yield less systematic results • Focus groups are best used to identify issues and develop surveys rather than as the only source of data

19. Who collects the data • Outside contractors – surveys and desk studies • Local survey researchers • Local academics for desk studies • National Statistical Offices • Government ministries • NGOs/CSOs

20. Data collection and dissemination exercise • Pick aGovernance issue in your country • Choose the levels and types of the data you will collect – Macro, Micro, De jure and De facto • Identify the types and sources of data • Choose the data collection method/s • Identify who will collect the data • Select several options for sharing the results with different audiences