Today:

1. Today: • Assignment 2 misconceptions • Our process • True Experiments: Single-Factor Design • Assignment 3 Q&A • Mid-term: format, coverage

2. Assignment 2 problems • Still issues with IV/DV identification • Not enough detail to really answer the questions • Not fully operationalizing the concept of the DV into the actual data that will be measured • Discrete/continuous (for the measurement) • Scaling types • What are descriptive statistics • Which descriptive statistics are appropriate for the different types of data • Estimation of reliability/validity of measurement (not the experiment)

3. Where we are in the process • Have defined broad research question and identified hypotheses or very refined research questions that must be evaluated to answer the question (Ass 1) • Have identified the variable that need to be manipulated/studied in order to evaluate the hypotheses and have thought about what kind of data will need to be measured (Ass 2)

4. Assignment 3 • Start thinking about threats to validity – what will you have to justify in order to convince readers that you have a valid study that addresses the research question • Note: as of yet, you don’t have a study designed. But you should be thinking about possible study designs that might be appropriate • Ass. 2 was descriptive stats (individual variables) – in Ass. 3, you need to think about which comparisons you need to make • Different measurements, single variable? • Satisfaction before/after treatment? • Correlations between different variables? • Satisfaction as it relates to a participant’s age? • Focus on describing the data – what tables/graphs would you use to show the relationships

5. Next: • Assignment 4: Hypothesis testing • Starting to think about statistical evaluation of the hypotheses: • Stating alternate and null hypotheses • Is each hypothesis one tailed or two tailed? • If there was a type I error, what would be the conclusion statement based on the error? What should it be if you knew it was an error? • If you had a type II error, what would be the conclusion statement based on the error? What should it be if you knew it was a error?

6. Next… • Assignment 5: Control • By now, you should be able to focus in on your study design • For each hypothesis, identify: • Where you will get the data • How you will rule out alternate explanations for the results (give an alternate & show why it’s not viable) • The limitations to your study

7. Ongoing action items (1) • Look at related work: • What papers will you cite when writing your research paper • Papers to motivate the problem • Papers that have used similar methodologies to answer similar types of questions • Papers that establish baselines in your area • Papers about methodologies • Papers that you should compare your results with (if you actually end up running the study)

8. Ongoing action items (2) • Think about what kind of study is needed to be able to answer the research questions/hypothesis that you have identified • How many factors are you looking at? • How much control do you need? • What are your biggest validity concerns that need to be addressed? • What is the usual approach in your discipline? • …

9. The research paper • Full details on Tuesday • Proposing a study to evaluate your research questions • The assignments are the setup to get you thinking about the details • In the paper, you will be writing them formally • In the presentation, you will be presenting them • Your ability to discuss alternatives and justify your chosen approach will be key to a good mark

10. Mid-term • 2:35-4:05, Friday, October 28th • 90 minutes (will aim to have most of you done in 60 minutes) • At 4:20 we will have in-class discussion of the mid-term answers • note: this is your chance to discuss what would be a good answer and I have not budgeted time to do this on an individual basis, so stick around. After this session, it should be self-evident why you receive the marks you get on your mid-term

11. Mid-term • 2:35-4:05, Friday, October 28th • 90 minutes (will aim to have most of you done in 60 minutes) • At 4:20 we will have in-class discussion of the mid-term answers • note: this is your chance to discuss what would be a good answer and I have not budgeted time to do this on an individual basis, so stick around. After this session, it should be self-evident why you receive the marks you get on your mid-term

12. What will be covered • Concepts covered in class (including the material from today) • Concepts covered in tutorials (up to the material covered on Monday, October 17) • Anything is fair game – if we talked about it, you may be evaluated on it

13. Question types • Multiple choice or T/F with explanation • Short answer • Define/contrast/explain/describe concepts covered • Analysis (apply your knowledge) • Given a research question or a study description, answer questions about the study design, validity, data analysis, etc.

14. True Experiments: Single-Factor Design Research Methods & Statistics Oct 21, 2011

15. True Experiment • Experimental control • Control as many potential threats to validity as possible • Random assignment of participants/data to conditions • Could be within-subjects or between-subjects

16. Control • True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects • Control over the who, what, when, where, how • Control of the who => random assignment to conditions • Only by chance can other variables be confounded with IV • Control of the what/when/where/how => control over the way the experiment is conducted

17. Quasi-Experiment • When you can’t achieve complete control • Lock of complete control over conditions • Subjects for different conditions come from potentially non-random pre-existing groups

18. Terminology • Factors: IVs of an experiment • Level: particular value of an IV • Condition: a group or treatment (technique) • e.g., Condition 1: old system, Condition 2: new system • Treatment: a condition of an experiment • Subject: participant (can also think more broadly of data sets that are ‘subjected’ to a treatment)

19. Factors to Treatments • At least 1 Factor (IV) has to vary to have an experiment • Effect of screen size and input technique on performance (speed, accuracy) • An IV must always have at least 2 levels • Condition refers to a particular way that subjects are treated • Between subject: experimental conditions are the same as the groups • Within subjects: only 1 group, that experiences every condition (can be many conditions in an experiment)

20. Experimental Design: spot the flaw • One-Group Post-Test-Only Design • Group of subjects are given a treatment • Then tested on the dependent variables • What’s the problem?

21. Experimental Design: spot the flaw • Post-Test-Only, non-equivalent control groups • Non-random allocation of subjects into groups • One group is given the treatment, one doesn’t receive it (different levels to each group) • Post-test: measure the DV • What’s the problem?

22. Experimental Design: spot the flaw • One-Group Pre-Test-Post-Test Design • Single group (within subjects) • Pre-test: measure the DVs • Give the treatment • Post-test: Re-measure the DVs • What’s the problem?

23. Good Experimental Design • Two-Group, Post-Test Design • Two conditions • Two groups: • Between subjects: random allocation • Treatment • Post-test: measure the DV • What’s really important?

24. Within-subjects • Similar to the one-group pre-test-post-test design • It solves the individual differences issues • But raises other problems: • Need to look at the impact of experiencing the two conditions • Will they get tired? Gain practice? Learn what is expected? • Need to control for order and sequence effects?

25. Order Effects • Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?) • Arises from warm-up, learning, fatigue, etc. • Effect can be averaged and removed if all possible orders are presented in the experiment and there has been random assignment to orders

26. Sequence effects • Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2) • Effects viewed may not be main effects of the IV, but interaction effects • Can be controlled by arranging each condition to follow every other condition equally often

27. Counterbalancing • Controlling order and sequence effects by arranging subjects to experience the various conditions (levels of the IV) in different orders • Self-directed learning: investigate the different counterbalancing methods • Randomization • Block Randomization • Reverse counter-balancing • Latin squares and Greco squares (when you can’t fully counterbalance) • http://www.experiment-resources.com/counterbalanced-measures-design.html

28. Discussion • Pros and cons of within versus between subjects designs

29. Assignment 3: Q&A • For five of the threats to validity covered in class, give an example of a possible threat to validity for your research question. Note: as you have not yet designed your experiment, this exercise is to identify possible threats under a possible experimental design. You can use different possible experimental designs for each threat. • What comparisons should you make to explore the answers to your research question? That is, which of the variables you identified in Assignment 2 should be analyzed individually and which variables should be compared against each other (which do you think might have interactions)? For each variable, specify what tabular and graphical methods you would use to describe the data.