1 / 35

Environmental Policy Evaluation

Environmental Policy Evaluation. MADE Program 30/10/2009. Policy Evaluation. Oversight agencies increasingly concerned about effectiveness of policies (both mandatory and voluntary)

wade-mcintosh
Download Presentation

Environmental Policy Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Environmental Policy Evaluation MADE Program 30/10/2009

  2. Policy Evaluation • Oversight agencies increasingly concerned about effectiveness of policies (both mandatory and voluntary) • Voluntary programs/International Agreements (ISO 9001/Kyoto Protocol) are greatly increasing in number and size • Need for rigorous, scientfic evaluations of policies

  3. Treatment • Policy is known as the treatment • Observations of unaffected entities is called the control • An effective policy will alter the outcome of those treated (given the policy) relative to the control

  4. Policy Story • It is worthwhile to learn the policy background before starting an evaluation • Sometimes the proper outcome variable to measure is not clear • Be willing to listen to those involved in the policy but remember their interests are different than yours • No Stockholm Syndrome

  5. Ideal Story • You find data that: • Starts before policy went into affect • Continues after policy went into affect • Covers those affected by policy and those unaffected • The policy • Randomly assigned those involved • Staggered its starting date

  6. Issue to Control for • The big issues to control for are: • Pre-policy behavior/trend • Behavior of those unaffected by policy • Selection of those affected by policy • Other variables that affect outcome • Ideally the policy will have a staggered start so that the analysis is using both time and space variation to determine policies affect

  7. Panel Data • Usually evaluations occur using panel data methods • Panel data is when you follow the same entity over time • Methods often used are fixed effects, random effects, dynamic panel, logit

  8. Voluntary Programs • Generally between government agency and a group of firms • Firms join the program (become a partner) and pledge to make specified environmental improvement • Partners potentially receive benefit in form of confidential information

  9. Standard Analysis • Conventional interpretation of partnership programs is that information given to partners bring exclusive benefits • Selection bias likely to occur; needs to be controlled for • Typical View - Program effective if partners had statistically better environmental outcome than non-partners

  10. Estimation • Difference in Difference Model • Post Policy Dummies (T) • Partner Dummy (P) • Partner & Policy Dummy Interaction (TP) • Other controls that alter dependent variable (X, S, N) • β9 is the relevant variable

  11. Diff-in-Diff Variable • β9 tells us whether the group affected by the policy acted differently than the group unaffected, once the policy went into affect • If it is positive & statistically significant, the policy is associated with an increase in dependent variable • Vice Versa for negative coefficients

  12. Other Interpretations • Policy variable (β8) tells us whether the dependent variable changed when policy went into affect • Partner variable (β7) tells us whether partners are inherently different with respect to dependent variable

  13. Example: C2P2 program • Coal Combustion Partnership Program (C2P2) started in 2001, part of Resource Conservation Challenge • promotes beneficial uses of Coal Combustion Products (CCP) as opposed to lanfilling them • CCP can be reused in concrete, asphalt, & more • About 170 Partners • Work with industry groups & other federal agencies to promote education about CCP reuse • Outreach/networking/acknowledgement for CCP suppliers and demanders • Document reuse by individual facilities, companies and government organizations

  14. Question • Is C2P2 associated with an increase in CCP re-use? • Data on coal power plants in US for 96-05 (C2P2 started in 2001) • Dependent Variable: Fly Ash Re-Use Rates • the amount of fly-ash sold divided by the total fly-ash generated

  15. Rate of Fly Ash Reuse

  16. Raw Data • Be careful with Raw Data like this • Other co-variates not controlled for • Policy makers tend to like these pictures even though they are not telling the whole story

  17. Estimation • Random Effects Model • Dependent Variable: • Ratio of Fly Ash Reused to Total Fly Ash Generated • Controls: • Pre, Early, & Late C2P2 Time Dummies • Partner Variable & C2P2 Time Interactions • Total Coal Burnt • Ash Content of Coal • Price of Aggregates/Cement in State • Level of Cement Imports • Region Dummies • NOx Control Equipment Present

  18. Evaluation Results

  19. Other Issues • Treatment Spillovers • Selection Bias • Error Specification

  20. Treatment Spillovers • Programs provide information that spills-over to non-partners; programs may be public good in nature • Technical information-spreading is not limited to partners, thus both partners and non-partners can take advantage of program • One country agreeing to reduce consumption of a good could lead to increases in price for other countries

  21. Evaluation Interpretation • If there are treatment spillovers, • Partners and non-partners do not need to be statistically different for program to be effective • Requires evidence that spillovers exist • Requires statistically better environmental outcomes for both groups after program initiation

  22. Evidence of Spillovers: C2P2 • Non-partner data analyzed for evidence of treatment spillover by comparing non-partners in states w/ many partners to non-partners in states w/ few partners

  23. Example: C2P2

  24. Selection Bias • If entities are not randomly assigned to treatment or control, selection bias could be a problem • If treatment is mandatory, selection bias is less likely to be a problem • If treatment is voluntary, then selection bias is likely to be a problem

  25. Endogeneity • Another way to describe selection bias is to say that the choice of enetering the voluntary progam is endogenous • This generally means that the choice of entering the program affects the outcome measure, rather than the outcome measure affecting the choice of entering the program.

  26. Direction of Bias • One big problem is that the direction of bias could be either positive or negative • If entities join program because it is easy to acheive goals set, the bias is positive • If entities join program because they really need the help, the bias is negative.

  27. Example-Positive • Norway joins the Oslo Protocol because it has large natural gas reserve, making reducing sulfur emissions relatively cheap • If evaluate the effect of the Protocol on Norway (relative to Poland for example), it will look like the Protocol had a large effect when in reality the change in behavior is driven by the switch to natural gas.

  28. Example-Negative • Poland joins the Sophia Protocol even though it relies heavily on coal for electricity production • If we evaluate the effect of the Protocol on Poland (relative to Sweden for example) it will look like the Protocol is having no effect when in reality it is the lack of alternatives for electricity that is driving the result.

  29. Correcting for Endogeneity • Generally known at Instrumental Variables • Find other variables that predict the variable that is endogenous • In the case of selection bias into the policy treatment, this generally means variables that are associated with the benefits or costs of recieveing the treatment

  30. Partner Choice Example: C2P2 • Dependent Variable: • Partner Designation • Controls: • Investment in Solid Waste Disposal 96-00 • State w/ CCP re-use authorized • State w/o CCP Permit Requirement • Utility Size • Avg Fly and Bottom Ash Reuse 96-00 • Avg Price of Agg. & Cement in State 96-00

  31. Example : C2P2

  32. Error Specification • Generally in panel methods, you would like to cluster errors by largest grouping entity possible • These could be industry classification, firm (as opposed to plant), country, etc • This controls for unobserved correlation better than a narrow grouping

  33. Other Options • Bootstrapping of error terms is a catch all • Code is quite easy in Stata to code • Many other potential fixes available, Stata help very good for this

  34. Policy Evaluation • Important to: • Consider pre-policy trends • Consider control group’s actions • Other co-variates • How entities enter the policy • How the treatment affects entities • With these issues covered, you have done as much as is possible as a data user

  35. Questions? • About Data • About Methods • About Examples • About Information on Policies • About dealing with policymakers • …

More Related