1 / 41

Advanced Methods and Analysis for the Learning and Social Sciences

Advanced Methods and Analysis for the Learning and Social Sciences. PSY505 Spring term, 2012 February 22, 2012. Today’s Class. Meta-Analysis. Meta-Analysis. What is it?. Meta-Analysis. What is it usually used for?. Meta-Analysis. What are the key challenges?. Meta-Analysis.

simeon
Download Presentation

Advanced Methods and Analysis for the Learning and Social Sciences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Advanced Methods and Analysis for the Learning and Social Sciences PSY505Spring term, 2012 February 22, 2012

  2. Today’s Class • Meta-Analysis

  3. Meta-Analysis • What is it?

  4. Meta-Analysis • What is it usually used for?

  5. Meta-Analysis • What are the key challenges?

  6. Meta-Analysis • What are the key challenges? • Lack of detail in reports (p<0.05) • Inconsistent reports (r, p, d/s) • File-drawer problem • Construct-name mapping

  7. Combining Significance • Stouffer’s Z Z sqrt(K)

  8. Combining Significance • Stouffer’s Z sum(Z) sqrt(sum(Var(Z))2) • Var(Z)=1 because it’s the normal distribution

  9. What if you have p?

  10. What if you have t?

  11. What if you have r?

  12. Asgn. 5 • Problems 1-3

  13. Weighting • When do R&R recommend weighting Z’s? • Is it a good idea?

  14. Key assumption • Independence between studies • When might this assumption be violated? • If independence not met, there are other tests that can be used • See chapter

  15. Combining Effect Size • Case of linear correlation r

  16. Combining Effect Size • Convert r to Fisher z (not the same as Z!) • Using a table or function • Why? • As correlation approaches 1 or -1, the distribution of correlation becomes non-normal • The 95% confidence interval for a correlation of 0.9 might include 1.1, but correlation can’t be greater than 1 • Fisher z adjusts this to make all distributions normal, making it possible to integrate across correlations

  17. Combining Effect Size • Combine Fisher z sum(z) K

  18. Combining Effect Size • Convert Fisher z back to r

  19. Key assumption • Independence between studies • When might this assumption be violated? • If independence not met, there are other tests that can be used • See chapter

  20. What about d/s • Convert it to r • Then conduct meta-analysis on r • Different equations for this conversion depending on properties of the data set • For more info, see p. 239 of Cooper, H., Hedges, L.V. (1994) The Handbook of Research Synthesis.

  21. Cool thing • Same methods can be used to compare between significance values or correlations • Subtract values rather than summing them

  22. Example • What is the significance of each study? • What is the significance of the two studies? • What is the difference between the studies? • Z=1.9, Z=2.2 • Z=1.9, Z= -0.5

  23. Evaluating detector goodness • Let’s do problem 4 together

  24. Evaluating detector goodness • How do we get an A’ from this data? (ignoring non-independence)

  25. Evaluating detector goodness • How do we get SE(A’) from this data? (ignoring non-independence)

  26. Evaluating detector goodness • How do we compare A’ to chance? (ignoring non-independence)

  27. Evaluating detector goodness • Now, why was this the wrong thing to do? (ignoring non-independence)

  28. Evaluating detector goodness • Re-doing the procedure accounting for independence • Compute A’ for each student • Compute Z for each student • Use Stouffer’s Z to integrate across students

  29. Evaluating detector goodness • When is this method inappropriate/useless?

  30. Cleaning the Registers • At this point, every student should have handed in three assignments • If you haven’t, come talk to me after class • You now need to do 3 of Assignments 6-10

  31. Asgn. 6

  32. Next Class • Monday, February 27 • 3pm-5pm • AK232 • Regression and Regressors • Readings • Ramsey, F.L., Schafer, D.W. (1997) The Statistical Sleuth: A Course in Methods of Data Analysis. Sections 7.2-7.4, 9.2-9.3, 10.2-10.3 • Witten, I.H., Frank, E. (2005)Data Mining: Practical Machine Learning Tools and Techniques. Sections 4.6, 6.5. • Assignments Due: 6. Regression

  33. The End

  34. Bonus Slides • If there’s time

  35. BKT with Multiple Skills

  36. Conjunctive Model(Pardos et al., 2008) • The probability a student can answer an item with skills A and B is • P(CORR|A^B) = P(CORR|A) * P(CORR|B) • But how should credit or blame be assigned to the various skills?

  37. Koedinger et al.’s (2011)Conjunctive Model • Equations for 2 skills

  38. Koedinger et al.’s (2011)Conjunctive Model • Generalized equations

  39. Koedinger et al.’s (2011)Conjunctive Model • Handles case where multiple skills apply to an item better than classical BKT

  40. Other BKT Extensions? • Additional parameters? • Additional states?

  41. Many others • Compensatory Multiple Skills (Pardos et al., 2008) • Clustered Skills (Ritter et al., 2009)

More Related