Experimentation

1. Experimentation INFO4990 – Week 6 INFO4990 Information Technology Research Methods (July, 2004)

2. Agenda • Experimentation in Computer Science and information systems research • Basic experimentation concepts • Some widely used experimental design in CS and IS field • Analyze data from experiment study INFO4990 Information Technology Research Methods (July, 2004)

3. History • Experiment in natural science • systematic acquisition of new knowledge, testing theory about nature • Agriculture • Chemistry • … • Experimentation in social, psychology and economic studies • Study people’s behavior • E.g., fairness study INFO4990 Information Technology Research Methods (July, 2004)

4. Experiment in computer science research • Derived from natural science experimentation • Computer systems performance analysis • Hardware • Software • Algorithm • Network INFO4990 Information Technology Research Methods (July, 2004)

5. Experimentation in Information System research • Derived from social and economic experimentation • Subject under study is usually human • Human behavior with regard to information system • Hyperlink transferred trustiness • Which subject is most suitable for distance learning INFO4990 Information Technology Research Methods (July, 2004)

6. Purpose of experiment • Discover and confirm causal relationship • Examine the possible influences that one factor or condition may have on another factor or condition INFO4990 Information Technology Research Methods (July, 2004)

7. Basic experimentation concepts • Independent variable • Cause • Research “measure” (manipulate) independent variable by creating a condition or situation • Manipulation of independent variable create different treatments. • Event manipulation • Affecting the independent variable by altering the events that subjects experience • Presence versus absence • Instructional manipulation • Varying the independent variable by giving different sets of instructions to the subjects INFO4990 Information Technology Research Methods (July, 2004)

8. Basic experimentation concepts (cont) • Effect (outcome) • Physical conditions, behaviors, attitudes, feelings, or beliefs of subjects that change in response to a treatment. • How to measure • IS research: various data collection methods • Questionnaire, interviews, observation, test • CS research: Metrics in the field • Performance time, rate, error rate, time to failure and duration INFO4990 Information Technology Research Methods (July, 2004)

9. The importance of control • Internal validity -- The extent to which we can accurately state that the independent variable produced the observed effect INFO4990 Information Technology Research Methods (July, 2004)

10. Experiment cases • A marketing researcher wants to study how humor in television commercials affects sales. To do so, the researcher studies the effectiveness of two commercials that have been developed for a new soft drink called Zowie. One commercial, in which a well-known but serious television actor describes how Zowie has a zingy and a refreshing taste, airs during the months of March, April and May. The other commercial, a humorous scenario in which several teenagers throw Zowie at on another on a hot summer day, airs during the months of June, July, and the August. The researcher finds that in June through August, Zowie sales are almost double what they were in the preceding three months. “Humor boost sales,” the research concludes. • Many alternative explanations INFO4990 Information Technology Research Methods (July, 2004)

11. Strategies to achieve control • Keep some things constant • What are variables that need to be held constant in most experiments? • Include a control group • Treatment group (experimental group) • Between-subjects design • Randomly assign people to groups • Use matched pairs • Matched-subject design INFO4990 Information Technology Research Methods (July, 2004)

12. 7 3 7 3 10 3 3 5 1 8 2 7 10 7 1 6 4 10 9 2 6 8 4 9 5 2 2 2 8 8 5 9 5 9 4 1 6 Random assignment Randomly assign one member of each pair to each group 1 4 10 6 treatment control DV DV Between and matched-subjects design INFO4990 Information Technology Research Methods (July, 2004)

13. Steps in conducting an experiment • Identify the relevant variables • State hypotheses • Decide on an experimental design • Decide the way to manipulate independent variables • Develop a valid and reliable measure for dependent variable • Pilot testing the treatment and dependent variable measures • Recruit subjects (or locate cases) • Assign subject to groups • Introduce treatment to treatment groups • Gather data for measure of the dependent variables • Hypotheses testing INFO4990 Information Technology Research Methods (July, 2004)

14. Experimental design • One shot case study • True experimental design • Factorial design • Block design INFO4990 Information Technology Research Methods (July, 2004)

15. Vertical alignment shows two Pretests are measured at same time Classic true experimental design • pretest-posttest • Treatment Versus control group • Randomized • Experimental design http://trochim.human.cornell.edu/kb/desintro.htm INFO4990 Information Technology Research Methods (July, 2004)

16. Factorial design • Two or more independent variables are manipulated in a single experiment • They are referred to as factors • The major purpose of the research is to explore their effects jointly • Factorial design produce efficient experiments, each observation supplies information about all of the factors INFO4990 Information Technology Research Methods (July, 2004)

17. A simple example • Investigate an education program with a variety of variations to find out the best combination • Amount of time receiving instruction • 1 hour per week vs. 4 hour per week • Settings • In-class vs. pull out • 2 X 2 factorial design • Number of numbers tells how many factors • Number values tell how many levels • The result of multiplying tells how many treatment groups that we have in a factorial design INFO4990 Information Technology Research Methods (July, 2004)

18. Factorial designs in computer system performance analysis • Personal workstation design • Processor: 68000, Z80, 8086 • Memory size: 512K 2M or 8M bytes • Number of disks: one, two or three • Workload: Secretarial, managerial or scientific • User education: high school, college, post-graduate level • Dependent variable • Throughput, response time INFO4990 Information Technology Research Methods (July, 2004)

19. 22 factorial design • Two factors, each at two levels • Example: workstation design • Factor 1: memory size • Factor 2: cache size • DV: performance in MIPS INFO4990 Information Technology Research Methods (July, 2004)

20. 2K factorial design • K factors, each at two level • 2K experiments • 23 design example • In designing a personal workstation, the three factors needed to be studied are: cache size, memory size and number of processors INFO4990 Information Technology Research Methods (July, 2004)

21. Full and fractional factorial design • Full factorial design • Study all combinations • Can find effect of all factors • Fractional (incomplete) factorial design • Leave some treatment groups empty • Less information • May not get all interactions • No problem if interaction is negligible INFO4990 Information Technology Research Methods (July, 2004)

22. 2 factors full factorial design • Used where there are two factors that are carefully controlled • Examples in computer system performance analysis • To compare several processors using several workload • To determine two configuration parameters such as cache and memory size INFO4990 Information Technology Research Methods (July, 2004)

23. 2 factors full factorial design (cont) • Example: cache comparison INFO4990 Information Technology Research Methods (July, 2004)

24. Field and controlled laboratory experiment • Field experiment • Experiments conducted in real-life or field settings • Researcher has less control over the experimental condition • Greater external validity but lower internal validity • Controlled laboratory experiment • Conducted under controlled conditions of a laboratory • Greater internal validity but lower external validity • Practical consideration • Planning and pilot testing • Instruction to subjects • Post experiment interview INFO4990 Information Technology Research Methods (July, 2004)

25. Example of field and controlled laboratory experiments • Field experiment • The case in slide 10 • A controlled laboratory version • Ask two group of subject (students) to view the tape of two different Ads (event manipulation). • Use questionnaire to collect their intentions to buy the product. • Compare the response from the two groups INFO4990 Information Technology Research Methods (July, 2004)

26. Analyzing data from between subject design • Problem • You want to measure the acquisition of mathematical skills by distance learning and traditional classroom learning. The study involves the comparison of 20 students, ten taught in classroom and ten taught by distance learning program. The final test scores were collected as dependent variable. INFO4990 Information Technology Research Methods (July, 2004)

27. Why can’t we just compare the means • The difference between the means is the same in all three. • They tell very different stories • When we are looking at the differences between scores for two groups, we have to judge the difference between their means relative to the spread of variability of their scores INFO4990 Information Technology Research Methods (July, 2004)

28. T-test • t-test • Assesses whether the means of two groups are statistically different from each other • Sample size is small • Approximately normal distribution of the measure in the two groups is assumed INFO4990 Information Technology Research Methods (July, 2004)

29. Perform t-test INFO4990 Information Technology Research Methods (July, 2004)

30. Interpret result • Set a significance level • Degree of freedom • N1+N2 - 2 • Compare t-value with critical value from t-distribution to see if it is larger enough to be significant INFO4990 Information Technology Research Methods (July, 2004)

31. Analyzing data from matched subject design • Problem • You want to compare the hit rate of a two cache algorithms. The simulated cache algorithms are running on 5 benchmarks and the hit rate were recorded INFO4990 Information Technology Research Methods (July, 2004)

32. Suitable test: Paired t-test • Calculation of t-value • Degree of freedom • N-1 INFO4990 Information Technology Research Methods (July, 2004)

33. Analyzing data from factorial design • Problem • The memory-cache experiments were repeated three times each. The result is shown right • What we want to find out • Which factor contribute most to the performance • What’s the joint effect of the two factors INFO4990 Information Technology Research Methods (July, 2004)

34. Suitable test: ANOVA • 2 way ANOVA (Analysis of Variance) • F-value • Between-sample variation/within-sample variation INFO4990 Information Technology Research Methods (July, 2004)

35. Statistical package • Excel • SPSS • SAS INFO4990 Information Technology Research Methods (July, 2004)

36. References • Paul D. Leedy and Jeanne Ellis Ormrod << Practical Research: Planning and Design>> 7th edition • Robert.B.Burns <<Introduction to Research Methods>> 4th edition • Raj Jain <<The art of computer system performance analysis by >> • www.socialresearchmethods.net • http://www.statsoft.com/textbook/stathome.html INFO4990 Information Technology Research Methods (July, 2004)