Assessing Climate Forecast Impacts

1. Assessing Climate Forecast Impacts Advancing Ex Post Methodologies Mark W. Rosegrant Siwa Msangi Liangzhi You

2. The Growing Importance of Climate Forecast Information • Increasing frequency of extreme weather events and changing global trends in climate characteristics • The vulnerability of increasingly complex global economic and food systems to environmental factors • In the face of increasing uncertainty, policy makers are demanding better information

3. How To Measure Forecast Value? • Growing body of literature examining the economic value of forecast information • Theoretical underpinnings are grounded in the theory of decision-making under uncertainty • The majority of this literature employs ex ante methods

4. Ex Ante or Ex Post ? Realized Climate Outcome Receive Forecast Signal B (update beliefs) (model possible response) A (prior beliefs) C (observe outcome of event) (also observe agent’s actions) Ex Ante Ex Post Modeled behavior Simulated Benefit Measured behavior Realized Benefit (simulated counter-factual)

5. Measuring Forecast Value • Within an Ex Ante framework, behavior is modeled and response is simulated to evaluate the net benefits of forecast • Within an Ex Post framework actual behavior is observed and underlying structural relationships driving response must be inferred to estimate the net benefits (comparing to without forecast information)

6. Overview of the Presentation • Look at Traditional Impact Assessment • Discuss the Challenges of Ex Post Evaluation • Look at some promising directions • Draw conclusions and recommendations for advancing research in this area

7. Ex Post Assessment in Ag Research • Long history of application in empirical literature • Looks mostly at benefits of new technology • Evaluates net benefits with and without innovation • Employs a variety of empirical methods

8. Two Approaches in Ex Post Assessment • One approach tries to econometrically measure the impact of Ag Research on productivity or production costs with reduced form relationships • Another approach uses consumer welfare theory to relate technology improvements to benefits received by consumers and producers of the agricultural goods within the economy

9. Econometric Methods • Treating technology as an input, estimate production function, cost function or total factor productivity (TFP) Ag. Output Conventional Inputs (i.e. land) Technical knowledge (i.e.R&D investment) Unconventional Inputs (i.e. infrastructure) Uncontrollable factors (i.e. weather)

10. Econometric Methods • Estimated research coefficients are then used to calculate the value of additional output attributable to the lagged research expenditures ( marginal rate of return to the research investment) • Growth Accounting: contributions by the components in the above equation to the rate of growth of aggregated output

11. Basic Economic Surplus Model • Technology-induced supply shift (S0 to S1) • Total benefits (consumer and producer benefits) are Area of I0I1ab • Basic model can be extended to incorporate multi-markets, to accommodate spillover, to adjust for market distortions etc. e

12. Lessons to Draw from the Literature • Scale (project, program, institution or the whole system) • Attribution (proper accounting for benefits and costs) • Selection bias (random sampling or “cherry-picking”) • Time lags (long lag between R&D investment and final impact)

13. Lessons to Draw from the Literature • Econometric methods rely on good-quality time series/panel data. More appropriate for entire research system rather than individual projects. • Economic surplus method requires limited data and flexible. It is widely used.

14. Challenges of Ex Post Assessment • Harder to take a ‘descriptive’ rather than ‘prescriptive’ approach • Non-excludable nature of climate information makes valuation harder • Must infer relevance of forecast from observed behavior which could be driven by a variety of factors • Dis-entangling the underlying structural relationships is non-trivial

15. 1. Define decision alternatives and determine that the decision is weather-information-sensitive 2. Identify user’s goals 3. Identify all decision-relevant information available to user 4. Develop a model describing the relationship between available information and the decision 5. Evaluate the model. Does it adequately describe the user’s behavior? 6. Use the model to determine the impact of forecast on criteria Undertaking a Descriptive Analysis

16. Economic Valuation Methods

17. Direct Valuation Methods • An Ex Ante approach would rely on questioning the information ‘consumer’ directly • An Ex Post approach relies on the revelation of this value through behavior directly tied to climate shocks • Making the structural link between the climate shock and observed action is the challenge

18. Ex Post Approaches to Direct Valuation • Direct Valuation can be done using reduced-form relationships that derive statistical relationships between forecast information to behavior • Surplus values can also be computed • An alternative is to look more closely at the underlying structural determinants of behavior and estimate models that can link those to climate information

19. Reduced-Form Methods • An econometric Ex Post analysis relies on the statistical inference of forecast value from observables in both the physical and economic environment (e.g. land value and climate characteristics) • Controlling for non-forecast related factors that affect observed reaction to climate shocks (and information) is the principal challenge

20. Possible Reduced-Form Model • Relationship tying Farm profits (P) to climate information (K) and other on-farm characteristics Conventional Inputs (i.e. land) Climate Information (knowledge sources) Unconventional Inputs (i.e. infrastructure) Uncontrollable factors (i.e. weather)

21. Controlling for Behavioral Factors • Since the on-farm input levels are endogenous, they are instrumented • The ability of the farmer to adjust also should be accounted for Exogenous factors (environmental, etc. ) On-farm characteristics (i.e. credit/labor constraints, farmer experience) Error term

22. Structural Estimation Approach • Is better able to connect the underlying drivers of response to climate information and environmental shocks than a ‘reduced-form’ approach • Better able to represent the constraints to agent behavior • Comes at a great computational cost as behavioral as well as environmental relationships must be estimated

23. Promising Trends • Despite relatively thin literature covering ex post methods, recent examples of innovative applications have surfaced • Cover a variety of settings, from pastoral management to crop production • Some methods empirical while others are experimental

24. A Few Case Studies • Solow et al. (1998) estimate net welfare from the use of ENSO-based climate information to range between $240 and $320 million annually for the U.S. agriculture sector alone • Bradford and Kelajian (1978) looked at the benefit-to-cost ratio of reducing sampling error in government-collected crop and livestock statistics

25. Livestock Management • Luseno et al. (2003) look at pastoralists in Northern Kenya/Southern Ethiopia • Compare pastoralists own perceptions with climate forecasts and observe how it influences their beliefs • They note the importance of taking flexibility into account, when measuring forecast value, to avoid underestimation

26. Experimental Approach • Sonka et al. (1988) use agribusiness example to show how decision experiments can be used to test response to varying levels of climate information • By using a controlled setting they are able to measure the impact of information more accurately and directly observe agent behavior

27. Comments and Critique • Despite the novelty of experimental methods, one may not be entirely sure that ‘real world’ behavior is being observed • Much depends on the design of the experiment and the ‘framing’ of the problem • But much insight can still be gained, and methods are apt to keep improving

28. Conclusions • Surplus-based methods might be applicable if designed properly • Structural estimation approaches give the most detail of the underlying relationships, but are the most challenging to apply, hard to generalize • Experimental techniques have increasing appeal and potential utility

29. Conclusions • Econometric methods would be valuable if reliable time series/panel data are available • Need to design panel or cross-sectional data collection efforts for evaluating climate forecast information (i.e. ASTI in agricultural R&D). Needs collaborative effort and long-term commitment.