Experimental Research Designs have Two Purposes:

1. Experimental Research Designs have Two Purposes: • …to provide answers to research questions • ...to control variance (differences) EDU 5900 AB RAHIM BAKAR

2. The main function of the experimental research design is to control variance. • Principle: maximize systematic variance, control extraneous systematic variance, and minimize error variance. • MAX-MIN-CON In other words control variance. EDU 5900 AB RAHIM BAKAR

3. Therefore the researcher attempts to: • maximize the variance of the variable(s) of the research hypothesis (i.e., maximize the difference in the dependent variable [outcome] caused by maximizing the differences in the independent variable [treatment]). • control the variance of extraneous or "unwanted" variables that may have an effect on the experimental outcomes, but which he/she is not interested (limit factors other than the treatment (IV) that could be causing differences in the outcome (DV) . • minimize the error or random variance (i.e., avoid unreliable measurement instruments which have high errors of measurement ). EDU 5900 AB RAHIM BAKAR

4. Maximization of Experimental Variance experimental variance the variance due to the manipulated (i.e., treatment) or attribute (i.e., gender) variables (IV) research precept: design, plan and conduct research so that experimental conditions are as different as possible on the independent variable. EDU 5900 AB RAHIM BAKAR

5. Control of Extraneous Variables (EV) • eliminate the variable(i.e., if you are worried about gender, only include one gender in the study). • randomization(i.e., if you randomly assign subjects to groups, the extraneous variable should be equally distributed among the groups) • build it into the design – make it a moderator variable (i.e., if you are worried about gender, build it into the analysis [2-way ANOVA]—we’ll learn about this later) • match subjects(i.e., match the characteristics of subjects and put one of each matched pair in each group) • statistically equate groups(i.e., use ANCOVA [Analysis of Covariance] to analyze the data with the extraneous variable used as a covariate—we’ll learn about this later) EDU 5900 AB RAHIM BAKAR

6. Minimizing Error Variance has Two Principle Aspects: • reduction of errors of measurement through controlled conditions (i.e., standardize testing procedures) • increase in the reliability of measures (i.e., revise test instruments or find more reliable ones) EDU 5900 AB RAHIM BAKAR

7. Experimental Designs Should be Developed toEnsure Internal and External Validity of the Study EDU 5900 AB RAHIM BAKAR

8. Internal Validity: • Are the results of the study (DV) caused by the factors included in the study (IV) or are they caused by other factors (EV) which were not part of the study? EDU 5900 AB RAHIM BAKAR

9. INTERNAL VALIDITY (KESAHAN DALAMan HASIL KAJIAN). • Does the treatment affect the dependent varibale or so other things affect the dependent variable?? • Misalnya Pelajar Yang Diajar Dengan Kaedah Inquiri Mendapat Keputusan Yang Lebih Tinggi (Test On Critical Thinking) Berbanding Pelajar Yang Diajar Dengan Kaedah Lecture. • Is the difference on achievement due to method of teaching (inquiry vs lecture methods? • What if they are better critical thinkers to begin with? • What if they also taking related course? • What if the teacher is a better teacher than the other teacher? EDU 5900 AB RAHIM BAKAR

10. Hipotesis: For educationally handicapped students, failure rate is related to amount of disruptive behavior. High correlation, so what? Can it be because of something else such as educational ability? Observed difference is directly related to the variables under investigation to be INTERNALLY VALID. kajian menunjukkan wujudnya korelasi yang tinggi (r=.8) di antara ketinggian pelajar dan skor pencapaian dalam ujian matematik bagi pelajar sekolah rendah. Ini bermakna semakin tinggi pelajar semakin tinggi pula pencapaiannya. Apa ertinya disini? Bolehkah ia membawa perubahan kepada amalan di sekolah? Atau amalan pemakanan? EDU 5900 AB RAHIM BAKAR

11. Threats to Internal Validity There are 16 common threats to internal validity. Subject Characteristics (Selection Bias/Differential Selection) -- The groups may have been different from the start. If you were testing instructional strategies to improve reading and one group enjoyed reading more than the other group, they may improve more in their reading because they enjoy it, rather than the instructional strategy you used. EDU 5900 AB RAHIM BAKAR

12. SUBJEK KAJIAN SEMEMANGNYA BERBEZA DARI AWAL LAGI. PERBEZAAN ADALAH TIDAK RAWAK. Misalnya korelasi tinggi antara disruptive behaviour dan kadar kegagalan. Siapa subjek kajian? How about academik ability subject berkenaan? Lain-lain contoh ciri termasuklah: umur, KEKUATAN, kematangan, gender, Ethnik, KECErDASAN, SIKAP, Kebolehan/Keupayaan dan banyak lagi. What to do? Pilih rekabentuk yang sesuai. Jika buat bandingan kumpulan pastikan kedua-dua kumpulan ada persamaan. Gunakan teknik statistiks yang sesuai bagi penyelidikan korelasi misalnya partial correlation EDU 5900 AB RAHIM BAKAR

13. Threats to Internal Validity Loss of Subjects (Mortality) -- All of the high or low scoring subject may have dropped out or were missing from one of the groups. If we collected posttest data on a day when the honor society was on field trip at the treatment school, the mean for the treatment group would probably be much lower than it really should have been. EDU 5900 AB RAHIM BAKAR

14. Kesan yang serious jika kita buat perbandingan kumpulan (dua kaedah mengajar bagi dua kumpulan) apabila bilangan subjek hilang daripada satu kumpulan adakah lebih besar daripada kumpulan lain. (expect poor students to leave in every group) but if > in one group than the other? One of the methods may appear more effective than the other. Paling sukar dikawal!!!!!! EDU 5900 AB RAHIM BAKAR

15. How to overcome? Berikan maklumat bahawa subjek yang terciri adalah serupa seperti subjek yang ada. How? Pretest score, ciri-ciri demografi atau lain-lain variable yang ada kaitan dengan kajian. Cuba halang keciciran atau minimize the loss, but how? Masa kajian sesuai atau tidak? (hampir perayaan, waktu tamat sekolahan) Subjek kajian sesuai atau tidak? (senior students selected too? May be too many of them 20-30%--control group)Effect of new diet on endurance Masa untuk menjawab soal-selidik panjang atau tidak? Soal-selidik senang difahami atau tidak? EDU 5900 AB RAHIM BAKAR

16. Mula di sini mula di sini untuk bintulu EDU 5900 AB RAHIM BAKAR

17. Threats to Internal Validity Perhaps one group was at a disadvantage because of their location.  The city may have been demolishing a building next to one of the schools in our study and there are constant distractions which interfere with our treatment. Location EDU 5900 AB RAHIM BAKAR

18. Threats to Internal Validity The testing instruments may not be scores similarly. Perhaps the person grading the posttest is fatigued and pays less attention to the last set of papers reviewed. It may be that those papers are from one of our groups and will received different scores than the earlier group's papers Instrumentation Instrument Decay EDU 5900 AB RAHIM BAKAR

19. Threats to Internal Validity The subjects of one group may react differently to the data collector than the other group. A male interviewing males and females about their attitudes toward a type of math instruction may not receive the same responses from females as a female interviewing females would. Data Collector Characteristics EDU 5900 AB RAHIM BAKAR

20. Threats to Internal Validity The person collecting data my favors one group, or some characteristic some subject possess, over another. A principal who favors strict classroom management may rate students' attention under different teaching conditions with a bias toward one of the teaching conditions. Data Collector Bias EDU 5900 AB RAHIM BAKAR

21. Distort the data in such a way to make certain outcomes such as support for the hypothesis. Allow more time to complete the test for some classes. Ask leading questions. Favoring one over the other (instructional methods) EDU 5900 AB RAHIM BAKAR

22. Report on absentism (tidak tanpa sebab tidak dikira tidak hadir dan tidak hadir dengan sebab dikira hadir). New definition of absentism. Menghasilkan kedatangan yang lebih baik daripada sebelumnya. What if the other school define differently? Then comparison is not valid. Suggest a possible answer during the interview. Awareness of which subjects are being experimented. EDU 5900 AB RAHIM BAKAR

23. How to overcome? Standardized all the procedures. Give training to data collectors Example of instrumentation threat: Grading an essay test for a few hours without stopping. EDU 5900 AB RAHIM BAKAR

24. Threats to Internal Validity Testing The act of taking a pretest or posttest may influence the results of the experiment. The pretest may have actually increased both groups' sensitivity and we find that our treatment groups didn't score any higher on a posttest given later than the control group did. If we hadn't given the pretest, we might have seen differences in the groups at the end of the study. EDU 5900 AB RAHIM BAKAR

25. Berikan Pretest –Matematik (Standard Test), Pelajar Dapat Markah Rendah. Kemudian Beri Treatment. Uji Dapat Markah Tinggi. Bolehkan Kita Katakan Ianya Disebabkan Oleh Intervention Berkenaan? Mungkin Tidak Boleh!!! PRETEST---MAKE STUDENTS MORE ALERT EDU 5900 AB RAHIM BAKAR

26. Threats to Internal Validity History Something may happen at the site during our study that influences the results. Perhaps a classmate dies in a car accident at the control site for a study teaching children bike safety. The control group may actually demonstrate more concern about bike safety than the treatment group. EDU 5900 AB RAHIM BAKAR

27. Apa yang berlaku semasa kajian boleh mempengaruhi hasil kajian berkenaan. Exposure to certain programs while participating in the study. Endurance test. While taking special diet, jog everyday. Democratic leadership practices improve commitment at the same time guru attend courses. EDU 5900 AB RAHIM BAKAR

28. Threats to Internal Validity There may be natural changes in the subjects that can account for the changes found in a study. A critical thinking unit may appear more effective if it taught during a time when children are developing abstract reasoning. Maturation EDU 5900 AB RAHIM BAKAR

29. Pengalaman dan natural Kepantasan berlari ada hubung kaitnya dengan kematangan fizikal (perubahan fizikal). maturation. Misalnya latihan lumba lari untuk pelajar darjah enam. Jadi kita tidak boleh sewenang-wenangnya katakan kalau berlatih boleh lari pantas. Mungkin yang tak belatih pun boleh lari Pantas. Keupayaan befikir pun ada kaitan dengan kematangan. Strength pun ada kaitan dengan kematangan. EDU 5900 AB RAHIM BAKAR

30. Threats to Internal Validity Hawthorne Effect The subjects may respond differently just because they are being studied. The name comes from a classic study in which researchers were studying the effect of lighting on worker productivity. As the intensity of the factory lights increased, so did the worker productivity. One researcher suggested that they reverse the treatment and lower the lights. The productivity of the workers continued to increase. It appears that being observed by the researchers was increasing productivity, not the intensity of the lights. EDU 5900 AB RAHIM BAKAR

31. Threats to Internal Validity One group may view that it is in competition with the other group and may work harder than they would under normal circumstances. This generally is applied to the control group "taking on" the treatment group. The terms refers to the classic story of John Henry laying railroad track. John Henry Effect EDU 5900 AB RAHIM BAKAR

32. Threats to Internal Validity The control group may become discouraged because it is not receiving the special attention that is given to the treatment group. They may perform lower than usual because of this. Resentful Demoralization of the Control Group EDU 5900 AB RAHIM BAKAR

33. Threats to Internal Validity Regression (Statistical Regression) -- A class that scores particularly low can be expected to score slightly higher just by chance. Likewise, a class that scores particularly high, will have a tendency to score slightly lower by chance. The change in these scores may have nothing to do with the treatment. EDU 5900 AB RAHIM BAKAR

34. Apa sebenarnya regrasi ke arah min? Berlaku apabila skor rendah semasa pre-test akan bergerak ke arah min skor post-test. Begitu juga skor tinggi semasa pre-test akan bergerak ke arah min semasa post-test. Tidak berlaku perbezaan skor yang di sebebkan oleh rawatan yang diberikan When you are at the bottom you have no where to go. When you are at the top you have no where to go. EDU 5900 AB RAHIM BAKAR

35. Cook and Campbell (1979, pp52-53) explained regression toward the as a phenomenon that: 1) operates to increase obtained pretest-posttest gain scores among the low pretest scores since this group's pretest scores are more likely to have been depressed by error; 2) operates to decrease the obtained change in scores among persons with high pretest scores since their pretest scores are likely to have been inflated by error; and 3) does not affect obtained scores among scorers at the center of the pretest distribution since the group is likely to contain as many units whose pretest scores are inflated by error as units whose pretest scores are deflated by it. EDU 5900 AB RAHIM BAKAR

36. Bila berlaku? Apabila subjek dipilih berasaskan skor extreme dalam pra-uji. Examples of this are students with the worst math ability, workers with the lowest morale or patients with the most severe symptoms. Regression toward the mean is expected in these cases where there is non-random sampling or assignment in experiments. Penyelesaian: EDU 5900 AB RAHIM BAKAR

37. Threats to Internal Validity The treatment may not be implemented as intended. A study where teachers are asked to use student modeling techniques may not show positive results, not because modeling techniques don't work, but because the teacher didn't implement them or didn't implement them as they were designed. Implementation EDU 5900 AB RAHIM BAKAR

38. Threats to Internal Validity Compensatory Equalization of Treatment Someone may feel sorry for the control group because they are not receiving much attention and give them special treatment. For example, a researcher could be studying the effect of laptop computers on students' attitudes toward math. The teacher feels sorry for the class that doesn't have computers and sponsors a popcorn party during math class. The control group begins to develop a more positive attitude about mathematics. EDU 5900 AB RAHIM BAKAR

39. Threats to Internal Validity Experimental Treatment Diffusion Sometimes the control group actually implements the treatment. If two different techniques are being tested in two different third grades in the same building, the teachers may share what they are doing. Unconsciously, the control may use of the techniques she or he learned from the treatment teacher. EDU 5900 AB RAHIM BAKAR

40. Control of Extraneous Variables (EV) • eliminate the variable(i.e., if you are worried about gender, only include one gender in the study). • randomization(i.e., if you randomly assign subjects to groups, the extraneous variable should be equally distributed among the groups) • build it into the design – make it a moderator variable (i.e., if you are worried about gender, build it into the analysis [2-way ANOVA]—we’ll learn about this later) • match subjects(i.e., match the characteristics of subjects and put one of each matched pair in each group) • statistically equate groups(i.e., use ANCOVA [Analysis of Covariance] to analyze the data with the extraneous variable used as a covariate—we’ll learn about this later) EDU 5900 AB RAHIM BAKAR

41. Once the researchers are confident that the outcome (dependent variable) of the experiment they are designing is the result of their treatment (independent variable) [internal validity], they determine for which people or situations the results of their study apply [external validity]. EDU 5900 AB RAHIM BAKAR

42. External Validity: • Are the results of the study generalizable to other populations and settings? External validity comes in two forms: population and ecological. EDU 5900 AB RAHIM BAKAR

43. KESAHAN LUARAN: SEJAUHMANA HASIL KAJIAN BOLEH DI GENERALIZE KEPADA POPULASI DALAM SEMUA KEADAAN. ADAKAH HASIL KAJIAN BENAR-BENAR MENGGAMBARKAN POPULASI? Apabila kajian dijalankan hasilnya dijangka dapat di aplikasi kepada keseluruhan populasi kajian di tempat mana sekali pun populasi itu berada. Tetapi perlu diketahui bahawa hasil kajian kita tidak semestinya boleh digenralize kepada semua populasi dalam semua keadaan. Keupayaan kita generalize hasil kajian banyak bergantung kepada sampel dan pesampelan yang dilakukan. EDU 5900 AB RAHIM BAKAR

44. Threats to External Validity (Population) Population Validity is the extent to which the results of a study can be generalized from the specific sample that was studied to a larger group of subjects. It involves... • ...the extent to which one can generalize from the study sample to a defined population--If the sample is drawn from an accessible population, rather than the target population, generalizing the research results from the accessible population to the target population is risky. • ...the extent to which personological variables interact with treatment effects--If the study is an experiment, it may be possible that different results might be found with students at different grades (a personological variable). EDU 5900 AB RAHIM BAKAR

45. CUBA MENGGUNAKAN HASIL KAJIAN UNTUK MENGATAKAN BAHAWA HASIL KAJIANNYA JUGA ADALAH BENAR UNTUK POPULASI/POPULASI MEMPUNYAI CIRI YANG SERUPA SEPERTI SAMPEL. EDU 5900 AB RAHIM BAKAR

46. POPULASI/POPULASI MEMPUNYAI CIRI YANG SERUPA SEPERTI SAMPEL. SO, HOW LARGE (SIZE) IS THE SAMPLE? HOW REPRESENTATIVE IS THE SAMPEL? WHAT IS MEANT BY REPRESENTATIVE? REP. – PEMBOLEHUBAH YANG BERKAITAN YANG DIJANGKA BOLEH MENENTUKAN HASIL KAJIAN EDU 5900 AB RAHIM BAKAR

47. BAGAIMANA TREATMENT (KAEDAH MENGAJAR) DIBERIKAN? ADAKAH SECARA RAWAK? ADAKAH ORANG YANG MEMBERI TREATMENT JUGA DIPILIH SECARA RAWAK? Adakah ia juga represeantive? Dipilih secara rawak? ADAKAH DATA DIPEROLEH DARIPADA SEMUA UNIT PESAMPELAN? APAKAH LANGKAH YANG DIAMBIL UNTUK MEMPASTIKAN RESPONDEN ADALAH SERUPA SEPERTI NON-RESPONDENT? EDU 5900 AB RAHIM BAKAR

48. BAGAIMANA JIKA PERSAMPELAN RAWAK TIDAK DAPAT DILAKSANAKAN? [Kita akan sentuh mengenai sampel dan persampelan sekejap lagi] TERANGKAN SEPENUHNYA SIAPAKAH SAMPEL KAJIAN (JANTINA, BANGSA, KEUPAYAAN, DAN SEBAGAINYA) REPLICATE KAJIAN BEKENAAN- ULANG KAJIAN MENGGUNAKAN SAMPEL YANG LAIN DALAM SITUASI YANG LAIN (SES, GEOGRAPHY,ABILITY DAN SEBAGAINYA) EDU 5900 AB RAHIM BAKAR

49. Threats to External Validity (Ecological) Ecological Validityis the extent to which the results of an experiment can be generalized from the set of environmental conditions created by the researcher to other environmental conditions (settings and conditions). There are 10 common threats to external validity. EDU 5900 AB RAHIM BAKAR

50. SEJAUHMANA HASIL KAJIAN ADALAH BENAR DALAM SETTING YANG LAIN. MAKSUDNYA JIKA DIULANG KAJIAN ITU DALAM SETTING YANG LAIN adakah HASIL YANG SERUPA AKAN DIPEROLEH. jelaskan dengan sejelas-jelasnya tentang SETTING KAJIAN berkenaan (SPACIOUS VS CROWDED CLASSROOM). HASIL KAJIAN YANG DIPEROLEH DARIPADA URBAN AREAS SUKAR UNTUK DIGENERALIZE KEPADA RURAL AREAS. MISALNYA KOMITMEN GURU TERHADAP PROFESION PERGURUAN ADALAH TINGGI TETAPI SUBJEK KAJIAN ADALAH DARIPADA BANDAR (KEMUDAHAN CUKUP MISALNYA) MAKA KITA TIDAK BOLEH MENGANDAIKAN IANYA JUGA SERUPA DENGAN GURU DILUAR BANDAR So How Close Is The Experimental Setting To The Natural Setting? EDU 5900 AB RAHIM BAKAR