Survey Designs

1. Survey Designs Basic Elements of Questionnaires and Survey Research

2. Questions in the Survey

3. Questions:Demographic & Descriptive • Typically the first few questions in a survey help to describe the respondent in some way that establishes what subgroup of the population he/she represents • Population variables may include things like age, income, years of education completed these are ratio data—they can be averaged place of birth, highest degree, ethnicity these are nominaldata—they name categories • Population variables are typically independent variables

4. 5. How much to you agree or disagree with this statement: “Most days I am enthusiastic about being a student.” • Strongly agree • Agree • Neither agree nor disagree • Disagree • Strongly disagree Opinion questions may be independent or dependent, depending on how they are used in the survey

5. 5. How much to you agree or disagree with this statement: “Most days I am enthusiastic about being a student.” • Strongly agree • Agree • Neither agree nor disagree • Disagree • Strongly disagree The scale used here is a “5-point Likert scale”

6. Likert Scales: • Strongly agree • Agree • Neither agree nor disagree • Disagree • Strongly disagree • Usually 5 point, but some prefer 7 point or 9 point • Typically a Likert scale provides ordinal data • May be reduced to nominal by combining all “Agree” categories and all “Disagree” categories

7. Summary Idea When using surveys, it is important to consider -variable type (dependent or independent) -data type (nominal, ordinal, interval, ratio) These features will affect the analysis plan, which is the procedures that can be used to analyze the data.

8. What research questions?

9. Survey Designs • Longitudinal—changes within a population over time • Cohort Studies—a longitudinal study of a subgroup in a population (over time) • Panel Study—a cohort study of the same group of people (over time)

10. Survey Designs Cross Sectional—data are collected at one point in time • Can measure current attitudes, beliefs opinions • Can measure needs • Can be used to evaluate a program • May be used to study one group, or compare groups

11. Study Designs

12. Who Did I Survey? • UKM Students • Graduate Students • Students Taking Research Class • Students in Prof. Subahan’s Class • Students who have e-mail access • Students who have Internet access • Students willing to do the survey

13. Levels of selection • Population (Who do I want to talk about?) Group of individuals that have one characteristic (independent variable) in common • Target Population (Who can be chosen?) List of individuals that qualify for study (they share one or more an independent variables of importance) (also known as Sampling Frame) • Sample (Who is actually chosen?) The set of individuals selected to be surveyed • Respondents (Who actually takes part?) The group of individuals who actually take part

14. Levels of Selection • Population—UKM Graduate students • Target Population—Graduate students in Prof. Subahan’s class • Sample—Members of Prof. Subahan’s class who have e-mail and Internet access. • Respondents—Member’s of Prof. Subahan’s class in the sample who responded (n=30+)

15. Inferences From Data For n=30 • Masters Student 83.3% • Doctoral Student 20.0% • Age = 35.9 years For n=10 (first 10 respondents) • Masters Student 67.2% • Doctoral Student 35.0% • Age = 34.9 years

16. Limitations of Information Where did you do your undergraduate degree?  • Malaysia. 83.9% • Asian country other than Malaysia. 3.2% • Europe. 3.2% • North America. 0.0% • Africa 0.0% • Other 9.7%

17. Inaccuracies in data Tell which of the following groups you consider yourself to belong to. If more than one applies to you, check all that apply:   • Male 53.3% • Female 60.0% In this case, all respondents fit one category or the other. The total number of responses should be 100%.

18. Forms of Data Collection • Mailed questionnaires • Electronic questionnaires • One-on-one interviews • Telephone interviews • Focus group interviews Focus group—a small group of respondents assembled to respond to survey topics

19. Open-ended Responses Describe how the teaching of mathematics and science in English has affected you. • Thematic analysis researcher reads responses and looks for themes that help to describe the situation, using an external theoretical framework to inform analysis and conclusions • Targeted analysis (Grounded Theory) researcher draws categories from data and looks for statements that relate—building theory that informs further analysis and conclusions

20. Thematic Analysis Describe how the teaching of mathematics and science in English has affected you. I am more confident and dare to talk to people using English. The quality of my teaching seems like it has decreased! I cannot explain some points very clearly and sometimes I cannot give examples spontaneously. However, I like to teach science in English. From this system, I get to know the weaknesses of my English and I have the chance to force myself to improve and practice my English.

21. Thematic Analysis Effect on the person (outside of classroom) • I am more confident and dare to talk to people using English. • I get to know the weaknesses of my English and I have the chance to force myself to improve and practice my English. Effect on teaching • The quality of my teaching seems like it has decreased! I cannot explain some points very clearly and sometimes I cannot give examples spontaneously. Affective (attitude) response • However, I like to teach science in English. • I am more confident and dare to talk to people using English.

22. Designing A Good Survey • Clear questions What would you like to do with research? • Single questions Have you seen people pushing other on trains and not waiting until people have disembarked? • Positive wording Do you think that people should not get on trains until others disembark?

23. Designing A Good Survey • Keep language simple and common How can a concerns-based adoption model explain responses to teaching in English? • Watch out for overlapping responses Would you rather have rice for lunch or fish? • Make sure questions include all in sample, or use branching to clarify. Do you consider riding the Rapid KL to be pleasant?

24. Pilot Testing Describe how the teaching of mathematics and science in English has affected you. “None. I am not a teacher.” “I find the lesson more interesting and I think it is one way to improve on my language proficiency as well.” “At first I found it hard to teach in English. By now I am more comfortable with it.” How might I reword the survey item?

25. Response Rate 137 students in the Sample 30 Respondents 21.89% Response Rate (What % actually respond?) Ways to improve response rate • Direct contact • Use of incentives (but not coercion) • Follow up procedures • Avoid sensitive questions

26. Sensitive Questions What is your age? Of 30 overall respondents, 30 responded Describe your opinion of the decision by the Malaysian government to require that English be used to teach mathematics and science. Of 30 overall respondents, 24 responded.

27. Standards of Practice Ethical practices in research • Human subjects review • Anonymity vs. Confidentiality • Fairness in analysis • Knowledge of data uses • Opportunity to refuse • Member checking of results (when possible)

28. Standards of Practice • Avoid bias within questions, especially in face-to-face interviews. • Train interviewers, especially if there are to be follow-up questions that are unscripted. • Be honest regarding the purposes of the questionnaire, the extent to which the results will be shared, and the level of anonymity/confidentiality.

29. An Abstract: KBSM Study The focus of the study is to determine teachers’ attitudes towards the implementation of the KBSM geography curriculum, and its relationship with age, academic qualification, teaching experience and level taught. This study was carried out on a sample of 125 trained geography teachers. The same group of teachers completed a questionnaire three times; at the beginning, middle and end of the two-year implementation period.

30. Discuss the Study Attributes What is the population? What is the target population? What are the variables? Which variables are dependent and independent? Is the study longitudinal or cross-sectional? What is a Likert scale question that could be used? What is one possible open-ended question? What is a variable that could provide ratio data? What is a variable that could be sensitive? What issues might the research face?

31. An Abstract: MARA Study The objective of this study was to identify the differences in personality factors, family environments, locus of control, socioeconomic status and prior achievement between high and low achievers, and between male and female students. The subjects of the study were 356 high and low achieving fifth formers from eight MARA Junior Science Colleges. The personality traits were measured using the Cattell’s Sixteen Personality Trait Questionnaire, and the locus of control was measured using the Rotter’s Internal-External scale.

32. Discuss the Study Attributes What is the population? What is the target population? What are the variables? Which variables are dependent and independent? Is the study longitudinal or cross-sectional? What is a Likert scale question that could be used? What is one possible open-ended question? What is a variable that could provide ratio data? What is a variable that could be sensitive? What issues might the research face?

33. An Abstract: Islamic Education The aim of this study is to explore the effectiveness of Islamic Education among the Muslim Students of UTM. In the questionnaire, 52 questions were asked. There were 50 questions with predetermined variables and two open-ended questions. The following predetermined variables were used. • Improvement of knowledge about Islam • Attitude of student towards Islamic Education • Moral values in the student’s daily life • The obedience to Islamic teaching among students The variables were viewed with regard to sex difference, duration of exposure to Islamic Education, and achievement.

34. Discuss the Study Attributes What is the population? What is the target population? What are the variables? Which variables are dependent and independent? Is the study longitudinal or cross-sectional? What is a Likert scale question that could be used? What is one possible open-ended question? What is a variable that could provide ratio data? What is a variable that could be sensitive? What issues might the research face?

35. Let’s Summarize What are five terms you should remember? What are three things that can affect a survey? What are three ways that surveys can be used in research? What are three things that can be problems?