Inspection, Evaluation, and Performance Auditing

1. Inspection, Evaluation, and Performance Auditing Presented By George F. Grob Center for Public Program Evaluation To the Association of Inspectors General New Orleans November 2013

2. Essentials of Evaluations and Inspections • What are they? • Comparison to Audits • Other Related Professional Disciplines • Evaluation Types and Approaches • Analytic Frameworks • Possible Organizational Locations • Managing an Evaluation Office • Quality Standards

3. Center for Public Program Evaluation What Is an Evaluation

4. Evaluation—What Is That? Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. American Evaluation Association

5. Examples of Government Sponsored Evaluations • Early implementation of emergency response systems after 9-11 • Effects of programmatic earmarks for international HIV/AIDS relief programs • Case study of Freddie Mac’s loss of $1.2 billion from Lehman Brothers bankruptcy • Frequency of and criteria for home visits to foster care families • Performance standards for homeless youth centers

6. More Examples • Unlicensed fire sprinkler contractors performing inspections in Chicago • Potential use of a fiscal intermediary by the Chicago department of public health • Assessment of New Orleans' System of City Courts and Performance Review of New Orleans Traffic Court • Evaluation of the City of New Orleans Delinquent Property Tax Collection Program • Promising approaches for State and local government enforcement of prohibition on sale of cigarettes to youth • Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs

7. Websites of OIG Evaluation Reports Federal Agencies City Government U.S. Congress • Department of Health and Human Services OIG Office of Evaluation and Inspections • Department of Commerce OIG Audits and Evaluations • Department of State OIG Audits and Program Evaluations OIG Inspections • Department of Veterans Affairs OIG Office of Audits and Evaluations Oversight Reports • Department of Housing and Urban Development OIG Evaluations and Inspections • Federal Housing Finance Agency OIG Audits and Evaluations • City of Chicago Audits, Reviews, Advisories • City of New Orleans Audits, Inspections, and Evaluations • Government Accountability Office (GAO) Reports and Testimonies Audits, Inspections, and Evaluations

8. Center for Public Program Evaluation Evaluation Standards

9. Government and Professional Standards for Evaluations and Inspections • Association of Inspectors General • Council of the Inspectors General on Integrity and Efficiency • Government Accountability Office • Institute for Internal Auditors • American Evaluation Association

10. Association of Inspectors GeneralQUALITY STANDARDS FOR INSPECTIONS, EVALUATIONS, AND REVIEWS • Qualitative Standards • Quality Control • Planning • Data Collection and Analysis • Evidence • Timeliness • Fraud and Other Illegal Acts • Reporting • Confidentiality • Follow-Up • General Standards • Staff Qualifications • Independence • Due Professional Care http://inspectorsgeneral.org/files/2012/06/IGStandards_revised_july2012.pdf

11. CIGIE Standards for Inspection and Evaluation • Timeliness • Fraud, Illegal Acts, Abuse • Reporting • Follow-Up • Performance Measurement • Working Relationships and Communications • Competency • Independence • Quality control • Planning • Data Collection and Analysis • Evidence • Records maintenance http://www.ignet.gov/pande/standards1.html

12. GAO Standards for Performance Audits • Chapter 3: General Standards • Independence • Professional Judgment • Competence • Quality Control • Chapter 7: Field Work • Planning • Supervision • Evidence • Documentation • Chapter 8: Reporting • Form • Contents • Quality Elements • Report Issuance and Distribution http://www.gao.gov/govaud/yb2003.pdf

13. AEA Guiding Principles for Evaluators • Systematic Inquiry • Competence • Integrity/Honesty • Respect for People • Responsibilities for General & Public Welfare http://www.eval.org/p/cm/ld/fid=51

14. Institute of Internal Auditors • Standards for the Professional Practice of Internal Auditing Discusses evaluation of risk and controls, and requires audit results to be based on analysis and evaluation. No “Evaluation” standard as such https://na.theiia.org/standards-guidance/mandatory-guidance/Pages/Standards.aspx

15. Inspections, Evaluations, and Reviews—AIG Definition • OIGs may have responsibilities that go beyond audits and investigations. • This additional work includes inspecting, evaluating, reviewing, studying, and/or analyzing government operations and programs for the purposes of providing information for decision-making, and of making recommendations to improve programs, policies, or procedures. • The objectives of these processes include: providing a source of factual and analytical information, monitoring compliance, measuring performance, and assessing the efficiency and effectiveness of operations. Source: Association of Inspectors General

16. Inspections and Evaluations—Federal IG Definition Systematic and independent assessments of the design, implementation, and/or results of an Agency’s operations, programs, or policies. They can be used to determine efficiency, effectiveness, impact, and/or sustainability of agency operations, programs, or polices. Source: Council of the Inspectors General on Integrity and Efficiency

17. Inspections and Evaluations—Federal IG Definition • They provide information that is timely, credible, and useful for agency managers, policymakers, and others. • providing factual and analytical information; • measuring performance; • determining compliance with applicable law, regulation, and/or policy • identifying savings and funds put to better use; • sharing best practices or promising approaches; • assessing allegations of fraud, waste, abuse, and mismanagement. Source: Council of the Inspectors General on Integrity and Efficiency

18. Center for Public Program Evaluation The Proving Professions

19. Auditors and Evaluators Audit Evaluation

20. Auditors and Evaluators Audit Evaluation

21. Auditors and Evaluators Audit Evaluation Performance Audit Performance Evaluation Financial Audits Attestation Audits Non-Audit Services Issue Analysis Inspections Reviews A Multitude of Analysis Types

22. Evaluation Inspection Audit Social Science Economics Operations Research Analysis • Policy Analysis • Public Administration • Program Specialties • Health Care • Housing • Defense • Trade • International Assistance Other Professional Disciplines Evaluators/Inspectors Come From Many Professions

23. Legal Analysis Applied Social Research Public Administration Inspections Statistics Evaluation Economics Operations Research Psychology

24. Public Administration Legal Analysis Applied Social Research Evaluation Inspections Statistics Psychology Economics Operations Research

25. What About Investigators? Evaluators Investigators ?

26. Useful Interplay Investigators Fraud leads Fraud trends Tech assist Evaluators

27. Center for Public Program Evaluation Evaluation Types and Approaches

28. Evaluation Types and Approaches • There are many legitimate and useful forms of evaluations • Each has its purpose. The methodological approach must be adapted to it • Generally accepted professional guidelines are available to guide the work.

29. Center for Public Program Evaluation As A Program Matures The Evaluation Questions Change How is the implementation going? Any Issues? Are service providers in compliance with program rules? Are the required services being delivered? Have conditions improved as intended? Are some service providers doing better than others? How so? Are impacted citizens or businesses satisfied with program results? How do the services stack up to goals and standards?

30. Center for Public Program Evaluation And So Do the Analytic Approaches and Methods Used Some inspections and evaluations need to be done fast Others need more time Some require more resources than others All must be done professionally Following are some evaluation and inspections types commonly used today.

31. Center for Public Program Evaluation Inspection/Evaluation Types • Compliance Review • Performance Review • Landscape Survey • Early Implementation Reviews • Impact Assessment • Report Card • Process Review

32. Center for Public Program Evaluation More Inspection/Evaluation Types • “Just the Facts” Review • Report from “On the Ground” • “Try it out” Reviews • Client Satisfaction Surveys • Before and After Assessment • Control System Survey • Program Logic Model

33. Center for Public Program Evaluation Analytic Frameworks

34. Center for Public Program Evaluation The Standard IG Analytic Model • FINDINGS -- Criteria -- Condition -- Effect -- Cause • RECOMMENDATIONS

35. Center for Public Program Evaluation What the Federal IG Blue Book Says “Findings often have been regarded as containing the elements of criteria, condition, effect, and, when problems are found, cause. However, the elements needed for a finding depend entirely on the objectives of the inspection. Thus, a finding or set of findings is complete to the extent that the inspection objectives are satisfied and the report clearly relates those objectives to the applicable elements of a finding.”

36. Two Analytic Frameworks Performance Audit Performance Evaluation • Findings • Criteria • Condition • Effect • Cause • Evidence • Physical • Documentary • Testimonial • Analytic • Recommendations • Objective • Questions to Be Answered • Analytic Approach • Mixed Methods • Qualitative • Quantitative • Findings • Recommendations

37. Center for Public Program Evaluation METHODOLOGIES

38. Center for Public Program Evaluation The Fundamentals • The Correct Methods to Use Are the Ones Most Compatible With the Purpose of the Study • Always Use More Than One Method • At least one quantitative • At least one qualitative

39. Center for Public Program Evaluation Quantitative and Qualitative Methods • Quantitative methods involve things you can count • Even if they are subjective answers to a closed ended questionnaire • Qualitative methods are applied to the interpretation of statements • Especially from interviews • But also from observations and in documents • Reviews are far more compelling if both methods are used • Otherwise people will say: “Numbers always lie” or “They always say that”

40. Center for Public Program Evaluation The Most Common Methods Used by Evaluators • Background Search • On-Site Inspections • Review of Administrative Files • Sampling • Data Collection Instruments • Surveys • Focus Groups

41. Center for Public Program Evaluation Some Exercises

42. You Be the Evaluator • Early implementation of emergency response systems after 9-11 • Frequency of and criteria for home visits to foster care families • Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs

43. Center for Public Program Evaluation The Quick Start 1. State the purpose 2. Ask two or three key questions 3. Choose the methods for each question • Quantitative • Qualitative 4. Conduct the Field Work 5. Analyze the data 6. Construct Findings and Recommendations 7. Draft the report

44. Discussion 1. Would your approach work as either an audit or evaluation? 2. Who will use the findings and recommendations? 3. How will you know that the recommendations have been acted upon? 4. What can you expect will be the result?

45. Center for Public Program Evaluation Organizing and Managing the Evaluation Function

46. Possible Organizational Location Investigation Inspector General Evaluation Audit Evaluation Evaluation Evaluation

47. Managing an Evaluation Office Work Planning Operating Procedures Quality Control Quality Assurance Professional Development These are generally similar to audit practices Take Them Seriously !

48. Training Priorities for Evaluators Highest Priority Other Worthwhile Skills • General Evaluation Concepts • Writing to the Point • Survey Design • Interviewing • Descriptive Statistics • Sampling • Spread Sheets • Graphs • Focus Groups • Data Base Development • Randomized Control Trials • Team Leadership • Programmatic Knowledge

49. Work Planning • Combine Audit and Evaluation • Focus on risk and return on investment • Use a Tactical Approach • To systematically fill gaps • To develop staff capacity • To keep up the attention on important areas

50. Center for Public Program Evaluation A strategic plan might consist of several families of program studies, some in the early stages, some further along, even in final stages Program Area A Program Area B Program Area C Program Area D