1 / 86

Useful Statistical Tools

Useful Statistical Tools. February 19, 2010. Today’s Class. Aphorisms Useful Statistical Tools Probing Question Assignments Surveys. Aphorisms. “Get close enough to know the task, but stay far enough to see the patterns.” "Humor happens, embrace it.“

kimberly
Download Presentation

Useful Statistical Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Useful Statistical Tools February 19, 2010

  2. Today’s Class • Aphorisms • Useful Statistical Tools • Probing Question • Assignments • Surveys

  3. Aphorisms “Get close enough to know the task, but stay far enough to see the patterns.” "Humor happens, embrace it.“ "Much like improv, prom night, and getting into fights, the key to good contextual inquiry is to always say yes.“ "Learn as though you would never be able to master it; hold it as though you would be in fear of losing it.“ "Until you learn to interpret openly, you open yourself to mis-interpretation.“ "To know an answer, you must ask a question. To know a truth, you must contextually inquire the right question.” "Your participant does all the hard stuff. All you have to do is talk about it and check your work“ "You cannot learn if you already know, unless you first learn how to forget!“ "Listen to the people around you, including to those you know well -- but listen deeper.“ "Do, or do not. There is no try."

  4. Any guesses • From those who did not email in? • Juelaila has won the first cookie • There is one cookie remaining

  5. Aphorisms “Get close enough to know the task, but stay far enough to see the patterns.” "Humor happens, embrace it.“ "Much like improv, prom night, and getting into fights, the key to good contextual inquiry is to always say yes.“ "Learn as though you would never be able to master it; hold it as though you would be in fear of losing it.“ "Until you learn to interpret openly, you open yourself to mis-interpretation.“ "To know an answer, you must ask a question. To know a truth, you must contextually inquire the right question.” "Your participant does all the hard stuff. All you have to do is talk about it and check your work“ "You cannot learn if you already know, unless you first learn how to forget!“ "Listen to the people around you, including to those you know well -- but listen deeper.“ "Do, or do not. There is no try."

  6. Aphorisms “Get close enough to know the task, but stay far enough to see the patterns.” "Humor happens, embrace it.“ "Much like improv, prom night, and getting into fights, the key to good contextual inquiry is to always say yes.“ "Learn as though you would never be able to master it; hold it as though you would be in fear of losing it.“ "Until you learn to interpret openly, you open yourself to mis-interpretation.“ "To know an answer, you must ask a question. To know a truth, you must contextually inquire the right question.” "Your participant does all the hard stuff. All you have to do is talk about it and check your work“ "You cannot learn if you already know, unless you first learn how to forget!“ "Listen to the people around you, including to those you know well -- but listen deeper.“ "Do, or do not. There is no try."

  7. Cookies! • "Do, or do not. There is no try.“ • Juelaila answered first • "Until you learn to interpret openly, you open yourself to mis-interpretation.“ • No answers

  8. Let’s discuss • A few of these aphorisms • Do you think that they help us understand the idea and practice of contextual inquiry better?

  9. Your thoughts? • “Get close enough to know the task, but stay far enough to see the patterns.”

  10. Your thoughts? • "Much like improv, prom night, and getting into fights, the key to good contextual inquiry is to always say yes.“

  11. Your thoughts? • "Until you learn to interpret openly, you open yourself to mis-interpretation.“

  12. Your thoughts? • "Your participant does all the hard stuff. All you have to do is talk about it and check your work“

  13. Your thoughts? • "You cannot learn if you already know, unless you first learn how to forget!“

  14. Comments? Questions?

  15. Today’s Class • Aphorisms • Useful Statistical Tools • Probing Question • Assignments • Surveys

  16. Useful Statistical Tools • Power Analysis • Meta-Analysis • Imputation

  17. Power Analysis • A set of methods for determining • The probability that you will obtain a statistically significant result, assuming a true effect size and sample size of a certain magnitude

  18. Or • The reverse • Given a certain true effect size, and a desired probability of obtaining a statistically significant result, what sample size is needed?

  19. Why? When? • Why might a researcher want to do each type of power analysis? • When might a researcher want to do each type of power analysis?

  20. When used • Effect size + Power --> Sample Size • Usually used before running study to pick sample size • Effect size + Sample Size --> Power • Usually used after running study to explain to thesis committee why more subjects are needed

  21. Power analysis • Can be computed from • “Effect Size”/ Cohen’s d • (M1 – M2)/ (pooled SD, e.g. s) • r • Difference in two r values • And several other metrics

  22. Power analysis • Can be computed for • Single-group t-test • Two-group t-test • Paired t-test • F test • Sign test • Etc., etc., etc.

  23. Mathematical Details • Differ for different statistical tests and metrics • Possible to do this in online power calculators

  24. Sign Test Example(Courtesy of John McDonald)

  25. What is a good value for power? • Conventionally, power = 0.80 is treated as “good” • Kind of a magic number

  26. Comments? Questions?

  27. I need 3 volunteers

  28. Play with calculator • http://www.cs.uiowa.edu/~rlenth/Power/ • Two-sample t-test

  29. Volunteer #1 • If the true effect size is 0.5 s, how big a sample do you need to achieve Power = 0.8?

  30. Volunteer #2 • If the true effect size is 0.2 s, how big a sample do you need to achieve Power = 0.8?

  31. Volunteer #3 • If your control condition gains 20 points pre-post • And your experimental condition gains 40 points pre-post • And the pooled standard deviation is 30 points • And you have 20 students in each condition • What’s your statistical power?

  32. Comments? Questions?

  33. How can statistical power be increased? • Both in theory, and in real life

  34. How can statistical power be increased? • Increase sample size

  35. How can statistical power be increased? • Increase difference in means • Make your intervention better

  36. How can statistical power be increased? • Increase difference in means • Make your control condition worse • Some researchers make the mistake of picking a control condition that’s impossibly good • ScienceAssistments versus ScienceAssistments, with one less potential IV • This doesn’t mean you should fish for a control condition that is absurdly awful • DrScheme versusLearning programming through interpretive dance • Miley’s World versusLearning math through reading textbooks

  37. How can statistical power be increased? • Increase difference in means • Make your control condition worse • Some researchers make the mistake of picking a control condition that’s impossibly good • ScienceAssistments versus ScienceAssistments, with one less potential IV • This doesn’t mean you should fish for a control condition that is absurdly awful • DrScheme versusLearning programming through interpretive dance • Miley’s World versusLearning math through reading textbooks written in Danish

  38. How can statistical power be increased? • Reduce standard deviation • What methods have we discussed in class that could help us do this?

  39. How can statistical power be increased? • Reduce standard deviation • What methods have we discussed in class that could help us do this? • Stratification

  40. Comments? Questions?

  41. Meta-Analysis

  42. Meta-Analysis • Very important point, right up front • There is meta-analysis • And then there are the statistical techniques used in meta-analysis • Much broader in application than just classical meta-analysis!

  43. Meta-Analysis • In the classic sense, integrating across a set of previous studies, to attempt to find an overall effect size or significance of finding across all those studies

  44. Examples • Kulik & Kulik (1991) computer-aided instruction does 0.3 s better than traditional instruction • Cohen, Kulik, & Kulik (1982) found that expert tutors do 2.3 s better than traditional instruction; novice tutors only do 0.4 s better than traditional instruction

  45. Process of doing a meta-analysis • Find all the studies on topic of interest • Find measure of interest (effect size or statistical significance) • Integrate across studies

  46. Challenges • What might make it difficult to • Find all the studies on topic of interest • ?

  47. Challenges to Finding all Studies • Knowing what terminology to use in literature review – many phenomena have many names • Off-task behavior, Time-on-task, Percent On-Task, Attention • Gaming the system, Systematic Guessing, Hint Abuse, Help Abuse, Executive Help-Seeking, Letaxmaning, Off-Task Gaming Behavior, Player Transformation, Goal Structure Misalignment

  48. Challenges to Finding all Studies • “File-Drawer Effect” • Papers with null results get rejected by conference program committees and journal reviewers • Papers with null results don’t get submitted in the first place

More Related