I. What is a survey?

1. I. What is a survey?

2. What is a survey? A tool to collect information from or about people to describe, compare, or explain their knowlege, feelings, values, and behaviour . Source: Fink, A.(2009),How to conduct surveys, SAGE.

3. What is a survey? • Mostly a quantitative research tool …. • Questionnaires sent by post • Online questionnaires • Face-to-face structured interviews • Telephone interviews

4. Doing a survey – main steps Design survey process Develop questions Test questions/tool and train interviewers Collect data Analyze data

5. II. Survey design

6. Target group • Who do I want to interview?(who should be in and who out) • Do I want to interview the whole target population or its sample? Tracer study Target population: graduates Sample: all graduates (of selected fields) or their sample

7. Timing • When is the best time to contact your target population? • How much time is needed to collect responses? Tracer study Timing: at least 6-9 months after graduation

8. Source: The ETF Transition Study in the Kyrgyz Republic 2011/2012

9. Timing (Repeated) Cross-sectional design - population of graduates asked only once Panel design - population of graduates asked repeatedly at different time points

10. Method of data collection Self-administered interviews (e.g. web-based, paper-pencil ) Face-to-face interviews Telephone interviews

11. Face-to-face interviews • High response rate • Possibility to clarify ambiguous answers/provide follow-up information - More time needed - More resources needed (most expensive)

12. Telephone interviews • High response rate, but lower than in case of face-to-face • Less time consuming (in comparison with face-to-face interviews • Less expensive (in comparison with face-to-face interviews) - Possibility of sample bias (not everybody has a phone)

13. Self-administered interviews • Saves money and time • People tend to be more truthful while responding - Lower response rate - Possible sample bias (web-based interviews)

14. Doing a survey – testing Once a questionnaire is ready, it must be tested in the field! - Do people understand to the questions? - Do people complete the questionnaire? - Do they often drop-out? - How long does it take to fill the questionnaire? - Can interviewersadminister the questionnaireproperly? - Are allproceduresstandardised?

15. Doing a survey – testing - Try to anticipate the actual circumstances in which the survey will be conducted - Choose respondents similar to the ones who will eventually complete the survey - Enlist as many people in the trial as reasonable, considering also your resources - Focus on the clarity of the questions and the general format of the survey

16. Doing a survey – things to consider • Reliability • Validity • Comparability • Usefulness and credibility of results • Costs

17. Doing a survey – things to consider • Reliability • Areliable survey providesconsistent information(i.e. repeated measurement yields the same results). …… makesurethateachquestionmeans the samething to everyone.

18. Doing a survey – things to consider • Validity • A validsurvey providesaccurate information. Do ourquestionsmeasurewhatthey are supposed to measure? How were the respondentsselected? Can wegeneraliseourresults? …………

19. III.How to increase a response rate?

20. What is a response rate? • The proportion of people who return the questionnaire out of the total • number of people to whom a questionnaire was distributed. • Example: If you distribute 100 questionnaires and you get back 80 of them, • your response rate is 80%.

21. Response rate matters! • If we get responses only from a small share of respondents → • results can be biased and not representative • results are not significant and useful

22. What to do? 1. Study preparation Think about study feasibility! Think how to reach your respondents/graduates! - Do you know their addresses? Or their parents’ addresses? - Do you know their telephone numbers? - Can you contact them via internet? - Do you need to use a snowball technique?

23. What to do? 2. Designing a questionnaire - Add a pre-notification letter before contacting the respondents • Add a cover letter while distributing the questionnaires - Makesurethatrespondentssee the valueof the survey Personalisedrequests and communication are important!

24. What to do? 2. Designing a questionnaire - Limit the number of questions to those which are relevant - Add clear instructions - Use clear and simple language - Add ‘thankyou’ at the end of the questionnaire

25. What to do? 3. Fieldwork Reminders! • Contact people severaltimes • Timeallcontacts • Usedifferentstrategies e.g. postal survey – send reminders after 2 and 5 weeks e.g. face-to-face/telephone interviews – contact respondents at least 3 times at different hour/day

26. Follow up of respondents is an important part of the survey fieldwork and can influence the quality of your results!

27. IV.How to ask a good question?

28. Doing a survey – preparing a questionnaire • What information is needed? • Can such information be obtained? • Format of questions: open vs. closed, scales etc. • Length of the questionnaire • The order of questions • Piloting

29. Questionnaire – basic rules • Introduction – provide a context of the survey • Assure anonymity • Simple format • Clean and consistent lay-out

30. Questionnaire – questions order • Begin with questions reflecting the subject of the study • Ask people to recall things in a chronological order • Group items that are of a similar topic • Personal and demographic questions should be placed at the end

31. Questionnaire – length • Too long questionnaire – negative effect on a response rate • Too short questionnaire – limited possibilities to respond our objectives Be efficient and askquestionsthat are relevant and needed!

32. Doing a survey – format of questions Closed questions Example: How would you rate your overall career perspectives over the next years? Very good Good Poor Very poor Open questions Example: What are your overall career perspectives over the next years? ---------------------------------------------------------- ---------------------------------------------------------- ----------------------------------------------------------

33. Doing a survey – format of questions Scales Thinking about students' motivation, please rate the following statements: (Likert)1=strongly agree, 2=agree, 3=disagree, 4=strongly disagree

34. What is a good question? • A question should have a meaning • A question should be concrete • A standard language should be used • Avoid biased words and phrases • Avoid double negative questions • Avoid asking two things in one question

35. What is a good question? • How bad is your teacher? Not concrete and biased • How would you rate the quality of your teachers and teaching materials? Twoquestions in one • Would you say that the students are not motivated to study? Double negative question

36. What is a good question? • Please, specify the expenditure on ICT: Abbrevitionused–notclear • How teachers and students evaluate the quality of food provided at school? Twoquestions in one • Do you think that math teachers are better than those teaching languages? Biasedquestion

37. Doing a survey – scales • Number of response categories depends on the survey purposes (usually between 3 to 5). Example: Do you agree or disagree with the statement that students always like to go to school?

38. Doing a survey – scales • Number of response categories must be balanced. Example: Do you agree or disagree with the statement that students always like to go to school? 1=strongly agree, 2=agree, 3=disagree, 4=strongly disagree 1=strongly agree, 2=agree, 3=disagree

39. Doing a survey – closed questions/scales • Number of response categories for each question must be exhaustive. Example:What is your current marital status? 1. Never married 2. Married / Living together

40. Doing a survey – closed questions/scales • Always specify whether one answer or multiple answers are possible. • Example:Why did you leave school before graduation? • 1. Studies too difficult • I wanted to work • I needed to work • I got married • Other • Example:What is your current marital status? • 1. Never married • Married / Living together • Other

41. References A large part of the presentation was based on -Fink, A. (2009), How to conduct surveys, SAGE.