EVALUATION OF US DISPUTE RESOLUTION PROGRAMS FOR ENVIRONMENTAL AND COMPLEX PUBLIC POLICY DISPUTES

1. Andy Rowe GHK International ghkint.us@ghkint.com www.ghkint.com EVALUATION OF US DISPUTE RESOLUTION PROGRAMS FOR ENVIRONMENTAL AND COMPLEX PUBLIC POLICY DISPUTES

2. DISCLAIMER This presentation has been prepared by the author and represents only my views. The agencies participating in or contracting the work on which this presentation is based do not necessarily subscribe to any or all of what is contained in the presentation. Andy Rowe

3. CHEAPER FASTER BETTER? • There appears to be more belief than evidence underlying this claim • This is really a statement about program effectiveness: “Collaborative processes are more effective than a reasonable or likely alternative process” • The evaluation system being used by important U.S. federal and state agencies is starting to provide evidence for this claim of effectiveness

4. BACKGROUND • The evaluation system specifically addresses environmental and complex public policy disputes • Features of the system: • Takes a results-based accountability / performance management approach • Takes an improvement orientation • Addresses expected internal and external performance reporting requirements (state and federal) • Self administered • Has a high level of face validity for both ADR and evaluation professions • Is proving to have a high level of validity and reliability • Focus on utilization. Results will: • Be easy to interpret • Be useful for improving performance • Be easily comprehended by ADR professionals

5. APPLICATIONS • U.S. Federal agencies • USEPA Conflict Prevention and Resolution Center • US Institute for Environmental Conflict Resolution • Other federal environmental agencies through contact to USIECR • U.S. State agencies • Oregon Dispute Resolution Commission • Florida Conflict Resolution Consortium • Other applications • Base for Maryland Alternative Dispute Resolution • Applied in pilot project of 9th Circuit

6. A PERFORMANCE MANAGEMENT APPROACH • Primary goal is to use evaluation as a mechanism to focus organizations on results and achieve success • While satisfying evaluation goals and standards • Address possible Government Performance Results Act (GPRA) requirements • John A. Koskinen’s three GPRA questions are what the evaluation attempts to address (John McLaughlin version): • What is your program / agency supposed to achieve? • How will you know when you have achieved this? • How are you doing now?

7. IMPLEMENTATION • Questionnaires to parties, practitioners and project manager at conclusion of the process and a follow-up to parties at a later date • Mail, web and email administration options • Response rates are high (will exceed 70% in Oregon pilot) and 100% of practitioners (included their contracts) • Information was systematically tested and has proved valid and reliable • Meets performance standards for the design • Managers are “can answer” to “fluent” for first two questions • Resource requirements = 10 hours per case for admin, analysis & reporting • Culture of the organizations shifts towards results focus • Internal stakeholders very satisfied, external not yet known

8. OUTCOME CHARTS ARE THE KEY TO THE DESIGN • Similar to logic models / logframes but much more focused on results • Outcome charts are best developed collaboratively with the program managers / staff • Illustrations that follow are from the Public Policy Program of the Oregon Dispute Resolution Commission • This is an ADR collaborative process – similar to applications in other agencies

9. OREGON PUBLIC POLICY DISPUTE RESOLUTION PROGRAM OUTCOMES Decreased cost of resolving disputes Increased efficiency of agencies through use of ADR Increase agency and public satisfaction with process and results of ADR ADR in State Government Case Management Dispute Systems Designs Oregon state agencies have practices and systems that ensure ADR is available Practice of ADR supports agency mission, and receives positive feedback ADR is successful (time and resources spent in disagreement and conflict are now redirected to more constructive purposes) Dispute resolution systems are improved Mediator Roster Good agreements Good Process Roster provides expedited access to qualified and appropriate providers

10. EXTRACTED OUTCOMES FOR OREGON CASE MANAGEMENT ADR successful (resources spent in disagreement are now used more constructively) Good outcomes from process Good agreements Party interactions constructive All parties involved in an ADR process are satisfied that the process was fair and open. ADR more effective (better benefits for resources expended) than other options for this dispute Durable and implementable agreement reached using ADR Agreements reached with ADR are complete - there are no hard issues left or deferred. Right parties continue to be engaged, new parties added as required. Use of ADR helps adversarial parties collaborate Use of ADR narrows disagreements Stakeholder capacity to use ADR is improved through experience with this case ADR is used where it is the best approach for this case Design of the ADR process is appropriate for the dispute and needs of parties ADR used appropriately Government decision making is improved through use of the process Appropriate mediator leads ADR Right parties engaged in process Non-ADR processes used where the best approach

11. Simplifying Complex Practices • Outcome charts seek to display the responses to the first of the GPRA questions: • What is my agency/unit trying to achieve? • The complexity of any one element of ADR practice is difficult to articulate in a single outcome but this complexity is required to respond to the second GPRA question: • How will we recognize this? • The mechanism I am currently using to simplify this is nested outcomes – mini outcome charts under individual outcomes • Even the nested outcomes can be at too high a level of generalization to be able to respond to the third GPRA questions • How are we doing now? • This is either addressed through further nesting or directly by the questions we ask

12. At not time did one of the parties dominate to the detriment of the process or others The neutral helped us manage our time well. The neutral made sure that the concerns of all parties were heard. The parties followed the direction of the neutral. The neutral made sure that the concerns of all parties were addressed. The neutral ensured that all parties were fully engaged in the process. When things got tense the neutral was always able to find ways to move ahead constructively ILLUSTRATION OF CHARTED AND NESTED OUTCOMES Nested Outcomes Expressed as questions Nested Outcomes Charted Outcome Appropriate neutral – parties satisfied with neutral Appropriate neutral – neutral understands issues Appropriate Neutral Guides the Process Appropriate neutral – manages process well Appropriate neutral – available for process Appropriate neutral – parties understand process

15. OTHER ELEMENTS IN THE EVALUATION SYSTEM • Collaborative Processes • Program Intake • ECR/ADR/Collaborative Processes (longer and shorter forms • Conflict Assessment / Collaborative Assessment / Situation Assessment • Meetings: information gathering / public input • Dispute System Design • Other ADR Services • Training • Roster program • Contract for neutral services

16. THE EVALUATION DESIGN IS SUCCESSFUL • Have established a program theory for ADR in environmental and complex public policy disputes • Approach is being implemented by a growing number of agencies • Positive peer external reviews (practice and evaluation) • Information is helping program improve • Information is addressing internal and external accountability requirements

17. MAIN REMAINING CHALLENGES • Incremental Benefits • Inherently difficult because of nature of many ADR cases • In some settings this can be addressed without difficulty • Effectiveness • Inherently difficult because of nature of most or perhaps all ADR cases • Lack of useful applied research in this area • Both incremental benefits and effectiveness are highly charged concepts in the field • Many strong believers have vigorously argued the merits of ADR based on intuitive appeal, inappropriate straw dogs, limited conceptual rigor and poor evidence • These are also linked to an even more charged debate between proponents of process versus proponents of agreements • There is a compelling logical and some empirical evidence that ADR will prove to deliver incremental benefit and be an effective approach when used appropriately

18. CHALLENGE 1: INCREMENTAL BENEFITS • Disputes tend to be heterogeneous • Arises from some or all of: characteristics of the dispute, parties, practitioner, general setting in time and place • Very challenging to find a counterfactual • Counterfactuals will always need to be unique to the case • Disputes often engage a variety of resolution options over their life • Litigation is not the only alternative • When litigation is the likely alternative we can get some reasonable information • For many disputes the likely alternatives are: do nothing, legislation or administrative rule making • Addressing the challenge • The strength of the program theory, fact that claims made by the program theory are not exceptional  “likely to” should suffice as an evaluation judgment of incremental benefits

19. CHALLENGE 2: EFFECTIVENESS • Costs and benefits occur in three phases: • Prior to the dispute resolution process. Many costs have already occurred such as fact finding, establishing the technical, legal and economic claims, identifying rights • During the dispute resolution process. Main costs are time of parties, attorneys, practitioner, travel. Ability to pay can vary dramatically (e.g. dispute involving a major utility, environmental groups and individual small farmers) • After the dispute resolution process. The hypothesis is that a good agreement will significantly reduce costs • Shelf life of this information is relatively short • The classes and characteristics of costs and benefits have not been enumerated, nor weights or likely ranges

20. ADDRESSING THE CHALLENGES • Incremental Benefits • Use litigation as an alternative where feasible • Undertake GOOD case studies to create illustrative comparators that can be referred to • Use “likely to” approach in the interim • Effectiveness • Case studies to establish the categories, ranges and coefficients • Include limited number of cost questions in the meantime • Manage the dance of “doing no harm” by not collecting bad cost data in the interim

21. MY CLAIM • Getting the organization to focus on results generates immediate and significant benefits • Outcome chart  program intent is clear = improved evaluability • 3 GPRA questions  requirement for monitoring outcomes = performance information • Worst case  Improved evaluation • Utilization enhanced • Can report on outcomes • Credible information for program improvement • Better case  Organizations adopt an evaluative and performance culture

22. CONTACT INFORMATION Andy Rowe GHK International 2707 Stratford Rd. Columbia SC 29204 (803) 782-2182 voice (803) 782-3618 fax ghkint.us@ghkint.com www.ghkint.com