1 / 33

Garry Young George Washington Institute of Public Policy December 2, 2009

Beyond Statistical Significance: Using Stata Post-Estimation Procedures to Examine Substantive Effects. Garry Young George Washington Institute of Public Policy December 2, 2009. What’s So Great About Statistical Significance?.

tanika
Download Presentation

Garry Young George Washington Institute of Public Policy December 2, 2009

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Beyond Statistical Significance: Using Stata Post-Estimation Procedures to Examine Substantive Effects Garry Young George Washington Institute of Public Policy December 2, 2009

  2. What’s So Great About Statistical Significance? • Statistical significance is crucially important in quantitative research • Tells you if a relationship likely occurred by chance (if analysis done correctly) • Tells you the direction (or sign) of the relationship • Can’t tell you the substantive significance or size of the relationship

  3. What’s So Great About Statistical Significance? • Can’t tell you the substantive significance or size of the relationship • Did the s.s. covariate increase Pr(Y=1) by a lot or little? • With large Ns statistical significance is easier to get and thus more prone to finding trivial relationships “significant”

  4. Substantive Effects in Research • Today journal reviewers, editors, and readers expect a consideration of substantive effects • authors often give it cursory treatment • Perhaps to hide trivial results • Perhaps because it can be computationally complex • Perhaps because we have no clear way to evaluate substantive significance • If a covariate increases Pr(Y=1) by 100% is that significant? • What if Pr(Y=1) without the covariate is .05 then the covariate doubles it to .10. Is that important?

  5. Computational Complexity • In OLS determining the substantive effect is easy: A one-unit change in X produces a b -unit change in Y, holding other variables constant. • Non-linear estimators (Poisson, logit, ordered probit, etc.) pose far more difficulty. • Today’s statistical packages – especially Stata -- make it easy

  6. Stata Options • Large number of post-estimating procedures in Stata for virtually all estimators • In Stata • . help postestimation commands • Extensive help • .search postestimation • Long list of available add-ons => Stata’s true strength as a program • S-Post • Clarify

  7. S-Post • Suite of post-estimation commands • Substantive effects • Diagnostics (e.g., fit statistics) • Developed by Scott Long & Jeremy Freese • J. Scott Long and Jeremy Freese, 2005, Regression Models for Categorical Outcomes Using Stata. Second Edition. College Station, TX: Stata Press. • For more on S-Post: http://www.indiana.edu/~jslsoc/web_spost/sp_install.htm

  8. Clarify • Suite of post-estimation software developed by Michael Tomz, Jason Wittenberg, and Gary King • There are different ways to install Clarify, here’s one:

  9. Installing Clarify: Step 1 On an internet-connected machine type: findit clarify

  10. Installing Clarify: Step 2 Then click

  11. Installing Clarify: Step 3 Then click

  12. What is Clarify? • Software that works within Stata • Uses Monte Carlo simulations to produce estimates of interest

  13. Available Estimators Negative binomial regression (nbreg) Seemingly unrelated regression (sureg) • OLS (reg) • logit (logit) • probit (probit) • Ordered logit (ologit) • Ordered probit (oprobit) • Multinomial logit (mlogit) • Poisson regression (poisson)

  14. Some Limitations • Hard to use with time-series estimators • Can’t handle TSCS estimators • E.g., xtreg, xtlogit, etc. • Can’t handle most types of survival analysis • E.g., stcox, streg • Stata 7 & earlier can do Weibull • Some diagnostics aren’t available • e.g., fitstat

  15. Workaround for Some Diagnostics • In many cases • run the regular model outside of Clarify • do the diagnostic • then run the model in Clarify to get your substantive effects.

  16. Take Fitstat as an example After running logit in Clarify, Fitstat returns an error

  17. Fitstat Example Part 2 Run regular logit Then fitstat. If you seriously still want to run this model then run it now in Clarify.

  18. The 3 Core Commands • estsimp • setx • simqi

  19. estsimp • estsimp prefaces your model • Instead of: logit Y X1 X2… • It’s: estsimplogit Y X1 X2… • This tells Stata to use Clarify to estimate a logit model and simulate it’s parameters • Most options normally available with the estimator are available within Clarify • E.g., estsimplogit Y X1 X2 X3 if year == t • There are a few estsimp specific options, e.g., number of simulations to run or to run multiply-imputed datasets (more later)

  20. setx • Use Setx to set the values of your explanatory variables. You have many options:

  21. Simqi • Simqi returns Pr(Y=) or the expected value of Y (depending on the estimator) • Here, too, are many options for adjusting how simqi runs and the type of output produced

  22. A Warning • Clarify derives its estimates from Monte Carlo simulations. • This means parameter estimates will vary slightly – usually very slightly. • Generally increasing the number of sims will negate differences • If you need exact replication you can set the random number seed to given number using the “set seed” command.

  23. An Ordered Probit example • Constituency-orientation of 173 MPs in single-member district seats in Australia, Canada, New Zealand, and the UK (Heitshusen, Young, and Wood 2005). • D.V: Constituency Orientation: High (3), Medium (2), and Low (1) • RHS variables: electoral safety, portfolio, years in office, travel time to parliament, country dummies.

  24. Oprobit Estimate of Constituency Orientation

  25. Substantive Effects • What if all Xs are at mean values? . setx mean . simqi Quantity of Interest | Mean Std. Err. [95% Conf. Interval] ---------------------------+-------------------------------------------------- Pr(conprior=1) | .1782677 .0309813 .1224842 .2405272 Pr(conprior=2) | .2802904 .0381232 .2080904 .3550502 Pr(conprior=3) | .5414419 .0418051 .4566825 .6183931

  26. Marginal MPs • MP at mean values except for safety. Setx is still at mean in memory so: . setx margin min . simqi Quantity of Interest | Mean Std. Err. [95% Conf. Interval] ---------------------------+-------------------------------------------------- Pr(conprior=1) | .0881035 .0303765 .0387724 .1638489 Pr(conprior=2) | .2047718 .0385767 .1329525 .2831817 Pr(conprior=3) | .7071246 .058064 .5838024 .8131391

  27. Safe MPs . setx margin max . simqi Quantity of Interest | Mean Std. Err. [95% Conf. Interval] ---------------------------+-------------------------------------------------- Pr(conprior=1) | .475743 .1037975 .2744931 .6825318 Pr(conprior=2) | .2925022 .0459677 .1975205 .3819004 Pr(conprior=3) | .2317548 .0821582 .0960326 .4181555

  28. How about the same thing as a first difference? . setx mean . simqi, fd(pr) changex(margin min max) First Difference: margin min max Quantity of Interest | Mean Std. Err. [95% Conf. Interval] ---------------------------+-------------------------------------------------- dPr(conprior = 1) | .3876395 .119811 .1415692 .6151865 dPr(conprior = 2) | .0877304 .0322312 .0266251 .1543707 dPr(conprior = 3) | -.4753699 .1214275 -.6935345 -.2141026

  29. Extensions • Lots you can do with simqi • Save predicted values and graph them, do first differences, etc. See Tomz, Wittenberg, and King (2001) or Clarify help in Stata for details.

  30. Substantive Significance? • What’s up with those confidence intervals? . setx margin max . simqi Quantity of Interest | Mean Std. Err. [95% Conf. Interval] ---------------------------+-------------------------------------------------- Pr(conprior=1) | .475743 .1037975 .2744931 .6825318 Pr(conprior=2) | .2925022 .0459677 .1975205 .3819004 Pr(conprior=3) | .2317548 .0821582 .0960326 .4181555

  31. It tells about range and certainty • Take the statement from King, Tomz, and Wittenberg (2000): “Other things being equal, an additional year of education would increase your annual income by $1,500 on average, plus or minus $500.” • Contrast with: “Other things being equal, an additional year of education would increase your annual income by $1,500.” • Or: “There is a statistically significant relationship between education and income.”

  32. Multiple Imputation • Clarify can work with Amelia (King et al 2001). • Amelia is another that works with Stata. It’s a multiple imputation program for addressing missing data. • I believe it will also work with Stata’s new multiple imputation procedure (mi) but I’ve not tried it.

  33. References • Heitshusen, Valerie, Garry Young, and David Wood. 2005. “Electoral Context and MP Constituency Focus in Australia, Canada, Ireland, New Zealand, and the United Kingdom.” American Journal of Political Science 49: 32-45. • King, Gary, James Honaker, Anne Joseph, and Kenneth Scheve. 2001. “Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation, “ American Political Science Review 95: 49-69. • King, Gary, Michael Tomz, and Jason Wittenberg. 2000. “Making the Most of Statistical Analyses: Improving Interpretation and Presentation,” American Journal of Political Science 44: 341-355. • Long, J. Scott and Jeremy Freese, 2005, Regression Models for Categorical Outcomes Using Stata. Second Edition. College Station, TX: Stata Press. • Tomz, Michael, Jason Wittenberg, and Gary King. 2001. “Clarify: Software for Interpreting and Presenting Statistical Results.” Manuscript, Stanford University. http://gking.harvard.edu/clarify/clarify.pdf.

More Related