Federal Policy and Program Effectiveness Information: who can deliver?

1. Federal Policy and Program Effectiveness Information: who can deliver? A presentation to the Canadian Evaluation Society June 3, 2003 Vancouver Prepared by Michael Obrecht Canadian Institutes for Health Research

2. Key Points of the Presentation • There is lack of clarity world-wide about the nature of effectiveness information • Canadian federal departments and agencies tend to focus on the effectiveness of relatively narrow program entities • An arms-length yet participatory effectiveness assessment process should be considered

3. Background: points advanced last year • No major arms-length determination of effectiveness of Canadian federal programming • Auditor General’s mandate requires a focus on economy and efficiency -AG looks for presence of systems for measuring effectiveness but does not assess their products • Waste and inefficiency are captured, but not necessarily relative effectiveness

4. Effectiveness can be missed when Economy and Efficiency are the focus Economy and efficiency good bad good Effectiveness bad

5. Background (continued) • AG has independence, authority, power and resources • But even if mandated for effectiveness assessment may be handicapped by: • Fault-finding orientation • Focus on spending rather than outcomes • Top-down, non-participative approach • Positivistic view of government business

6. Background (concluded) • Last year (2002) we advanced the idea of a federal Evaluator General focused on generating effectiveness information on major policies and programs • Have since then further investigated the relationship between evaluation and audit in theory and in practice in governments around the world

7. Point 1:Worldwide lack of clarity about the nature of effectiveness information • In many jurisdictions, ongoing performance information and long-term effectiveness information are not clearly differentiated

8. Differentiating Effectiveness and Performance Information

9. Examples of Focus on Performance Indicators • United States: Government Performance and Results Act (GPRA) • Canada: Improved Estimates Process – Annual Performance Reporting (DPR)

10. Drawbacks of Focus on Performance Measures • One-way information flow • Selective information • Questionable targets • Devalued process • General shallowness/ unsuitability for decision-support

11. Unanswered Questions • What kind of performance information do Parliamentarians want? • Is ongoing performance information enough? • Are there federal jurisdictions in which effectiveness information is generated and used?

12. Point 2:Internal evaluations by departments and agencies are of limited use externally • As spenders of funds, departments and agencies have some motivation to understand the effectiveness of their discrete program mechanisms or activities • As receivers of funds, they have little motivation to evaluate major programs or policies that affect their funding

13. Unanswered Question • To what extent are cross-cutting evaluations of horizontal programs providing Parliament with decision-relevant effectiveness information?

14. Point 3:A participatory yet arms length effectiveness assessment process should be considered • Effectiveness information is value-laden • Stakeholders value systems must be taken into account when generating effectiveness information if it is to be useful • Public representatives and public interest groups must be involved in the design of effectiveness studies on programs or policies in which they are interested

15. Arms-length Management Required • Managing multi-stakeholder effectiveness studies requires • Dispassionate interest – objectivity • High credibility • Deep commitment – resources • Should it be the Auditor General?

16. AG Management of Effectiveness Studies – the points in favour • High credibility • Arms-length from government • World-wide trend of increased interest of auditors in effectiveness assessment • Recruitment of senior evaluation experts by AG • Intensive training in evaluative methodologies is feasible – example: US General Accounting Office (GAO) • Long history of federal audit function

17. AG Management of Effectiveness Studies – the points against • Past focus on fault-finding – and current power resides in fault-finding approach • Authoritarian rather than participative • Pervading positivist view • Parliament has in the past decided against mandating AG for effectiveness assessment • Federal separation of audit and evaluation

18. Idea of an Evaluator General for Canada • Still alive - no clear world evidence of in-depth effectiveness information from federal audit functions • Innovative and would fit the Canadian situation • Would provide balance and oversight • Best empowered through legislation