Research Seminars in IT in Education MIT6003 Quantitative Educational Research Design 1

1. Research Seminars inIT in Education(MIT6003)Quantitative Educational Research Design 1 Dr Jacky Pow

2. Agenda Identification of research questions in quantitative research Formulation of assumption and hypothesis Sampling of subjects Data collection methods

3. Formulation of research questions A research question is simply a hypothesis stated in question form Formulating a research question allows a researcher to conduct more open-ended inquiries useful if there is little previous research on the topic a wider range of outcomes can be reported may encourage excessive manipulation of findings The research question and hypotheses formulated should be matched

4. Phrasing research questions NO AMBIGUITY! Be specific and straightforward Avoid vague or nebulous wording KEEP IT SIMPLE AND FOCUSED!  Don’t confuse your readers

5. Phrasing research questions The research question must be testable! Not so good: Do students with high GPAs feel better about themselves than students with low GPAs? Better: Do undergraduates with GPAs greater than 3.50 have higher self esteem than undergraduates with GPAs of 1.3 or lower?

6. Assumption and Hypothesis Assumption (2 different uses) Yet to be verified answer to the question To set conditions for the research (e.g., the configurations of the students’ home computers are the same) A hypothesis is a tentative prediction about the nature of the relationship between two or more variables: A hypothesis represents an educated guess about what will happen in the study Hypotheses are always held tentatively

7. Hypothesis Hypotheses are simply if-then sentences that can be categorized in certain logical forms, such as no difference (null hypothesis), associated difference, directionality of difference, and magnitude of difference A good hypothesis implies all these forms in a single sentence, and the trick is to express them as briefly as possible and as simple as possible (e.g. in simple English)

8. Hypothesis All theories contain hypotheses A single theory can generate many hypotheses All hypotheses demonstrate inference by concisely reducing extant (existing) knowledge into manageable and meaningful form Extant knowledge is what you obtain from a literature review

9. Types of hypothesis Null hypothesis No relationship Experimental hypothesis Non-directional Directional

10. Null Hypothesis Conservative approach to conducting research Statement of “no relationship” between two variables, or “no difference” between two groups The null hypothesis is “supported” if the results are non-significant the null hypothesis is never “proven” (at least not by a single study) impossibility of proving a negative

11. Experimental hypothesis Prediction of statistically significant findings significant differences or correlations between groups or among variables Non-directional hypothesis significant difference in any direction Directional hypothesis significant difference in the predicted direction

12. Non-directional hypotheses If Variable A changes, then Variable B changes, or There is a relationship between Variable A and Variable B, or Variable A affects Variable B

13. Directional hypotheses If Variable A increases, then Variable B increases, or If Variable A decreases, then Variable B decreases If Variable A increases, then Variable B decreases, or If Variable A decreases, then Variable B increases

14. Hypothesis examples Null hypothesis: Ho: The use of ICQ will not increase student’s written communication skill Non directional hypothesis: H1: History and Science teachers will differ significantly in their use of IT in their teaching Directional hypothesis Ha: Science teachers will use more IT in their class than history teachers

15. Class activity Identify each of the following types of hypothesis: There are significantly more gifted children in band I schools than in Band II schools There is no correlation between GPA and salaries 10 years post graduation There is a correlation between frequency of misconducts and gender Prolong indulgence in playing computer games will significantly affect children social skills Experienced primary school teachers will display less negative attitudes toward hyper-active students than inexperienced teachers

16. Developing hypothesis What is the general research question?   What specific hypotheses will you address? Is it important? Is it new, exciting, or creative? How does it fit into the existing literature? What work have you already done that is relevant? How can you operationalize the question?   

17. Developing hypothesis What are the independent and dependent variables? What is the design? Are the measures valid and reliable?  Can you collect the data? What analyses will you use?  Do you have enough resources?

18. Steps in formulating hypothesis Decide what you want to explain: choose a dependent variable Your dependent variable must show variation Run Descriptives to see mean and dispersion statistics Even better, run Frequencies, and call for a histogram along with the mean and std. dev

19. Steps in formulating hypothesis Choose independent variables that also show variation One cannot explain variation in a dependent variable with an independent variable that does not vary e.g., the use of computer in primary 6 cannot be explained by age as age does not vary much in primary 6

20. Steps in formulating hypothesis Think of multiple causes of the dependent variable: Do two or more independent variables combine to affect it? Consider using multiple regression to deal with multiple causes Consider alternative measures of both the dependent and independent variables

21. Steps in formulating hypothesis Does a relationship hold for some units of analysis but not others? For male but not for female? For developed nations but not for Third World nations? For Chinese, but not for Westerners? Try to develop your analysis so that it considers all the cases, even if the relationship does not apply equally to them

22. Class activities Construct 2 hypotheses in IT in education topic in each of the followings Non-directional hypotheses Directional hypotheses Inverse directional hypotheses Magnitude with difference hypotheses

23. Variables The basic building blocks of quantitative research are variables. Variables (something that takes on different values or categories) are the opposite of constants (something that cannot vary, such as a single value or category of a variable)

24. Variables Quantitative variables vary in degree or amount (e.g., annual income) Categorical variables vary in type or kind (e.g., gender) Independent variables (symbolized by “IV”) A variable that affects (or is assumed to affect) the dependent variable under study and is included in the research design so that its effect can be determined (e.g., smoking)

25. Variables Dependent variables (symbolized by “DV”) The variable being affected or assumed to be affected by the independent variable (e.g., lung cancer) Dependent variables are influenced by one or more independent variables Intervening variables (also called mediator or mediating variables) Variables that occur between two other variables. e.g., tissue damage is an intervening variable in the following relationship: Smoking---->Tissue Damage---->Lung Cancer

26. Class activity For each of the following, indicate the correct variable type by IV for Independent Variable or DV for Dependent Variable: If use of computer increases as a result of accessibility, then accessibility is what type of variable? If the general public believe that most hackers are male, then being male is what type of variable? If the use of ICQ is inversely related to age, then age is what kind of variable? If the motivation of using a self-learning courseware is a function of the attractiveness of a courseware, then motivation is what type of variable?

27. Class activity - continue If hacking behaviour, mediated by locus of control and lack of appropriate counseling, creates conditions conducive to committing computer crime, then locus of control would be what type of variable? If the corporate ethics climate and computer crime are associated, then ethics climate is what type of variable? If the use of illegal copies of computer programs is linked to the convenience of getting them, then the use of illegal copies is what type of variable? For two groups of pupils, one which is exposed to an educational computer program, and the other which is not, it is shown there are significant differences in learning outcomes, then educational computer program is what type of variable?

28. Sampling of subjects A statistic is a numerical characteristic of a sample A parameter is a numerical characteristic of a population

29. Sampling size Try to get as big of a sample as you can for your study (minimum = 30) If the population size is 100 or less, then include the whole population rather than taking a sample Look at other studies in the research literature and see how many they are choosing

30. Sampling of subjects Probability samples Simple random sampling Systematic sampling Stratified sampling Non-probability samples Convenience sampling Quota sampling Purposive sampling

31. Simple random sampling Equal chance of selecting (by means of probability) from each member of the population (i.e., equal probability sampling method) Randomly selected from the list of the population (i.e., sampling frame) Samples thus selected should contain subjects similar to the population as a whole Produce “representative” samples

32. Simple random sampling Put all the names from the population into a box and then do a “lucky draw” Use computer programs to perform Simple random sampling: Research Randomizer http://www.randomizer.org/ Random assignment of samples http://www.graphpad.com/quickcalcs/randomize1.cfm Use a random table

33. Systematic sampling A modified form of simple random sampling Selected systematically rather than randomly Reorganize the population list (i.e., sampling frame) when combining several ordered lists, to avoid periodicity (i.e., there is a cyclical pattern in the sampling frame)

34. Systematic sampling Determine the sampling interval (k) (i.e., the population size divided by the required sample size) e.g., Population=3,000; sample=300; k=10 Starting point for the selection is chosen at random from 1 to k Include every kth element in the sample starting at the number chosen from 2

35. Stratified random sampling Dividing population into homogeneous groups (i.e., stratify the sampling frame) Each group containing subjects with similar characteristics Stratified sampling can be proportional or disproportional e.g., assume the class sizes of 7A and 7B are 40 and 20 respectively, and the desired sampling size is 12 Proportional = 8 from 7A and 4 from 7B Disproportional = 6 from 7A and 6 from 7B

36. Convenience sampling Choosing the most available or the most easily selected individuals to serve as respondents Continuing the process until the required sample size is obtained For example, the pupils or teachers in your school would be convenience samples of your study

37. Quota sampling The non-probability equivalent of stratified sampling Samples are not selected by probability Attempts to obtain sample that contains the characteristics of the population For example, set a quota to each form in your school but the samples are not selected randomly or systematically (just to select the students you know)

38. Purposive sampling Purposely select the samples that the researcher thinks suitable to the study For example, in studying the student computer usage pattern, you may survey those students who stay in the computer lab after school

39. Response rate No guarantee that the subjects will response to the study (response rate) or providing complete data (quality of response) A low response rate will affect the validity of the data collected (i.e., data are biased) The minimum number of response rate differs when surveying a professional population (70%) and the general public (e.g., 50-60%) Research would need to know the sources of nonresponse and include it in the report if necessary

40. Ways to increase the response rate Perceived reward > perceived cost of responding (in terms of time and effort) Being consulted on an issue of importance to the subject Convenience (e.g., self-addressed stamped envelope for returning questionnaires) Gifts or remuneration (e.g., upon completing a valid questionnaire) Follow-ups (i.e., reminders) A must for almost all questionnaire surveys Should be planned for in advance Include a realistic time cue in completing the questionnaires

41. Data collection methods Tests Questionnaires Quantitative Interviews Quantitative Observations

42. Tests Standardized tests are used in research to measure the variables (e.g., personality, aptitude, achievement, and performance) New test will be used to measure the specific knowledge, skills, behavior, or cognitive activity that is being studied Remember that if a test has already been developed that purports to measure what you want to measure, then you should strongly consider using it rather

43. Strengths of (standardized) tests A wide range of tests is available (i.e. can provide measures of many characteristics of people) Tests are usually already developed Consistent (i.e., the same stimulus is provided to all participants) Strong psychometric properties (high measurement validity) Allows comparability of common measures across research populations Availability of reference group data Can provide “hard,” quantitative data Ease of data analysis because of quantitative nature of data

44. Weaknesses of (standardized) tests Can be expensive if test must be purchased for each research participant Test may not be appropriate for a local or unique population Open-ended questions and probing not available Tests are sometimes biased against certain groups of people Nonresponse to selected items on the test

45. Questionnaires A questionnaire is a self-report data collection instrument filled out by subjects Questionnaires can be paper-and-pencil instruments web-based (online) Remember survey usually use questionnaires but questionnaires are not equal to the ‘survey’

46. 15 Principles of Questionnaire Construction Make sure the questionnaire items match the research objectives Understand the subjects The subject (not you) to fill out the questionnaires Use natural and familiar language Familiar language is comforting; jargon is not

47. 15 Principles of Questionnaire Construction Write items that are clear, precise, and relatively short If the subjects do not understand the items, the data will be invalid Short items are more easily understood and less stressful than long items Do not use "leading" or "loaded" questions Avoid double-barreled questions “Do you acquire the computer skills from classmates and other friends?”

48. 15 Principles of Questionnaire Construction Avoid double negatives “I disagree that students should not acquire the computer skills on their own.” Determine whether an open-ended or a closed ended question is needed In quantitative study, questions are usually closed ended

49. 15 Principles of Questionnaire Construction Use mutually exclusive and exhaustive response categories for closed-ended questions Mutually exclusive categories do not overlap (e.g., age 0-10, 11-20, 21-30 …) Exhaustive categories include all possible responses (i.e., include all possibilities)

50. 15 Principles of Questionnaire Construction Consider the different types of response categories available for closed-ended questionnaire items Rating scale Numerical rating scales (endpoints are anchored; sometimes the center point or area is also labeled) Fully anchored rating scales (all the points on the scale are labeled) It does not matter whether there is a center point (i.e., both 4-point and 5-point scales generally work well)

51. 15 Principles of Questionnaire Construction Rankings (i.e., subjects put their responses into rank order, such as most important, second most important, and third most important) Semantic differential (i.e., where one item stem and multiple scales, that are anchored with polar opposites or antonyms, are included and are rated by the subjects) Checklists (i.e., where participants check all of the responses in a list that apply to them)

52. 15 Principles of Questionnaire Construction Use multiple items to measure abstract constructs To achieve high reliability and validity To use a summated rating scale (Likert Scale)

53. 15 Principles of Questionnaire Construction Consider using multiple methods when measuring abstract constructs The use of multiple methods may yield more valid data Less “method-dependent” (i.e., are the answers corroborated across the methods of measurement or do you get different answers for the different methods?)

54. 15 Principles of Questionnaire Construction Use caution if you reverse the wording in some of the items to prevent response sets A response set is the tendency of a participant to respond in a specific direction to items regardless of the item content Develop a questionnaire that is easy for the participant to use The subjects must not get confused or lost anywhere in the questionnaire Make sure that the directions are clear and that any filter questions used are easy to follow

55. 15 Principles of Questionnaire Construction Always pilot test your questionnaire You will always find some problems that you have overlooked Try best to pilot test with people similar to the subjects to be included in your research study After pilot testing your questionnaire, revise it and pilot test it again, until it works correctly

56. Strengths of questionnaires Good for measuring attitudes and eliciting other content from research subjects Relatively inexpensive and quick turnaround Can administer to probability samples Can be administered to groups Perceived anonymity by respondent may be high Moderately high reliability and validity for well constructed and validated questionnaires Ease of data analysis Useful for exploration as well as confirmation

57. Weaknesses of questionnaires Usually must be kept short Nonresponse to selective items People filling out questionnaires may not recall important information and may lack self-awareness Response rate may be low for mail and email questionnaires Data analysis New measures need validation

58. Quantitative Interviews Are standardized (i.e., the same information is provided to every subject) Use closed-ended questions The key difference between an interview protocol (looks very much like a questionnaire) and a questionnaire is that the interview protocol is read by the interviewer who also records the answers