Independent Evaluation: Insights from Public Accounting

1. Independent Evaluation: Insights from Public Accounting Abigail Brown (Harvard) Jacob Alex Klerman (Abt Associates) Presentation for Improving the Quality of Public Services, Moscow, June 2011

2. This Paper/Presentation … • Reviews the history, theory, and experience in “independent audits” for insights about “independent evaluations” • Our thesis: In both fields • The disjunction between who oversees the audit/evaluation and the ultimate consumers of the results • Leads to problematic incentives for the auditor/evaluator and bad outcomes • But that recognizing the “disjunction” and the experience in auditing suggest several feasible changes that would improve the situation Brown & Klerman Brown & Klerman May 9, 2011 2

3. Outline The Auditing Problem(s) Some Auditing “Solutions” The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 3

4. Provide the capital Owners and Managers Owners • Managers run the firm day to day Managers Brown & Klerman

5. Owners are too busy to “watch well” But need to “watch” Malfeasance Insufficient effort Hire/Fire Incentive compensation (bonuses, stock options) But Who Watches the Mangers? Owners Managers Brown & Klerman

6. Paid by owners(corporate funds) Enter “Independent Auditor” Owners • Origins in 1840s England • Required by 1934 (U.S.) Securities Exchange Act Auditor To watch (and report on) managers Managers Brown & Klerman

7. But Why Should this Work? • Managers • Choose auditors—this time and next time—and how much to pay them • Oversee auditor’s contract, including scope • Can give auditors other (non-auditing/consulting) work • So why should auditors report truthfully to owners? • Why not just report what managers want reported? Paid by owners(corporate funds) Owners Auditor To watch (and report on) managers Managers Brown & Klerman

8. Outline The Auditing Problem(s) Some Auditing “Solutions” The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 8

9. Professionalism Reputation Liability Owner Oversight Four Partial Solutions Brown & Klerman

10. Professionalism Reputation Liability Owner Oversight A distinguishing mark of a profession is acceptance of its responsibility to the public. … In discharging their professional responsibilities, members may encounter conflicting pressures from among each of those groups. In resolving those conflicts, members should act with integrity, guided by the precept that when members fulfill their responsibility to the public, clients' and employers' interests are best served. .. Those who rely on certified public accountants expect them to discharge their responsibilities with integrity, objectivity, due professional care, and a genuine interest in serving the public. AICPA Code of Conduct Brown & Klerman

11. Professionalism Reputation Liability Owner Oversight Suppose there are two types of auditors— “independent” and “not independent” The auditor selected by management is publicly observable What would owners conclude if management chose a “not independent” auditor? So management has to choose an “independent auditor” A Simple Equilibrium Argument Brown & Klerman

12. Professionalism Reputation Liability Owner Oversight Auditors can be sued for errors Have been large payouts Should be a strong financial incentive for “independence” But, liability requires Clear standards for malfeasance—which the profession has tried hard to avoid Strong liability—which the profession has successfully weakened Sue Them … Brown & Klerman

13. Professionalism Reputation Liability Owner Oversight If the problem is too little owner (too much management) involvement in the audit, fix that directly! SOX requires that an “Independent Audit Committee of BoD/Board of Directors” (i.e., excluding management) Hire the auditor Oversee the audit Approve other business with the auditor (to prevent side payments) SOX/Sarbanes-Oxley Act of 2002 Brown & Klerman

14. Outline The Auditing Problem(s) Some Auditing “Solutions” The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 14

15. Evaluation is Like Auditing What are the implications of experience in “independent auditing” for “independent evaluation”? Brown & Klerman Brown & Klerman May 9, 2011 15

16. Professionalism Reputation Liability Owner Oversight Applying these Strategies to Evaluation Brown & Klerman

17. Professionalism Reputation Liability Owner Oversight Also problematic in evaluation Evaluation is also a business Stakes are high: own employment and bonus Perhaps professionalism more effective in evaluation Public release of results and methods Results published in academic journals Results reviewed and used by academics Same Problems as in Auditing Brown & Klerman

18. Professionalism Reputation Liability Owner Oversight To some extent But clients also seem to value (perhaps value more) a reputation for “Client service” Not revealing client’s secrets Not publishing without permission Does Reputation Matter? Brown & Klerman

19. Professionalism Reputation Liability Owner Oversight There does not seem to be any statutory basis on which to sue evaluators for failure to be “independent” Who would sue? For what? What are the damages? Ideas for making this work? Liability Does Not Seem Operative Brown & Klerman

20. Professionalism Reputation Liability Owner Oversight Currently, agency Selects the evaluator Oversees the evaluation Reviews the report and any presentations Up to and including (indefinitely) preventing publication and presentation For evaluation seems easier to change By analogy with SOX Audit Committee reforms “0wner”Oversight Seems Promising Brown & Klerman

21. Professionalism Reputation Liability Owner Oversight Expand membership of TWG/Technical Working Group to include Congress (e.g., CBO, GAO) Higher executive (e.g., OMB, agency’s Inspector General, agency’s PA&E) General policy community (academics, think tanks from both sides) Change role of TWG Select the evaluator Oversee the evaluation Review the report and any presentations Decide what and when to release reports By Analogy with SOX, Audit Committee … Brown & Klerman

22. Professionalism Reputation Liability Owner Oversight Agency comments on, but need not approve, materials going to TWG Remove restrictions on evaluator contact with the TWG Encourage reporting of attempts by agency to subvert independence of the evaluation Encourage Independence by … Brown & Klerman

23. Outline The Auditing Problem(s) Some Auditing “Solutions” The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 23

24. “Owner Oversight” in Practice • Explicitly specify which tasks are “independent evaluations” • Sometimes in statute—specific evaluations, certain “evaluation funds” • Sometimes by programs wanting strong evidence of effectiveness • Define “independent evaluation” in the FAR/Federal Acquisition Regulations; i.e., • A TWG that includes key consumers, and • Rules for how the TWG would operate • Limit other business contact between “independent evaluators” and agencies (to prevent “side payments”) Brown & Klerman Brown & Klerman May 9, 2011 24

25. Plus, Facilitate Market for Reputation • Publicize deviations attempts to subvert independence • By the agency, and • By the evaluator • Distinguish “independent evaluations” from others • In making funding decisions (e.g., official policy reviews by NAS/IOM) • At conference presentations and at journal publication Brown & Klerman Brown & Klerman May 9, 2011 25

26. Outline Introduction The Auditing Problem(s) Some Auditing “Solutions” The Analogy to Evaluation Discussion Brown & Klerman Brown & Klerman May 9, 2011 26

27. Independent Evaluation: Insights from Public Accounting Abigail Brown (Harvard) Jacob Alex Klerman (Abt Associates) Presentation for Improving the Quality of Public Services, Moscow, June 2011

28. Evaluation is Like Auditing Brown & Klerman Brown & Klerman May 9, 2011 28

29. Provide the capital Owners and Managers Owners For most of human history, these were the same. • Managers run the firm day to day Managers Brown & Klerman

30. Separation of “owners” (i.e., stockholders) and “control” (i.e., management) is a cornerstone of the modern corporation Allowing: Poor entrepreneurs to start companies/implement their ideas Owners to invest all of their money and to diversify Owners and Managers Owners Managers Brown & Klerman

31. And the Issues Are Subtle • Pure fraud is clear, but • How much to spend looking? • Where to look? • Management often has a desired “spin”; owners may want something else • GE’s earnings smoothing • Lehman’s “Repo 105” • Good accounting requires judgment • When judgment is required, which way does the accountant lean? Paid by owners(corporate funds) Owners Auditor To watch (and report on) managers Managers Brown & Klerman

32. Professionalism Reputation Liability Owner Oversight Their own private (financial) interests are large Continued employment, repeat audit business, consulting business Auditing’s professional ethos emphasizes confidentiality and client (i.e., management) service Not independence (and serving owners) But … Brown & Klerman

33. Professionalism Reputation Liability Owner Oversight Now suppose management suggests “creative accounting” Perhaps with some “side payment” to the auditor (future auditing, consulting work, etc.) Auditor weighs Side payment vs. Loss of reputation as an “independent auditor” if side payment is discovered Deviation Should be Costly Brown & Klerman

34. Professionalism Reputation Liability Owner Oversight Yes: Arthur Anderson collapsed after criminal conviction in relation to Enron scandal No: Other accounting scandals did not cause similar collapse (Arthur Anderson was the exception) Some evidence of collusion among firms (rather than competition on “independence”) Too few to fail (currently only four large accounting firms) Does this “Reputation” Work? Brown & Klerman

35. Professionalism Reputation Liability Owner Oversight Duh! Wasn’t this obvious? This was “best practice” before SOX; but inconsistently applied Why not? And what does that suggest? Management often controls the BoD Management often “suggests” BoD members Who are not independent—relatives, interlocking BoD, other business relationships And board membership is lucrative, so members want to stay in management’s good graces Will SOX Work? Brown & Klerman

36. Evaluation is Like Auditing Brown & Klerman Brown & Klerman May 9, 2011 36