April 18, 2012 AGA Training Audio Conference Albert J. Lee, Ph.D., CEO, Summit Consulting LLC

1. Find Strength in Numbers: Sampling Techniques to Improve Financial Audit, Control, & Program Performance April 18, 2012 AGA Training Audio Conference Albert J. Lee, Ph.D., CEO, Summit Consulting LLC Denise Wu, CGFM, CPA, Partner, CliftonLarsonAllen LLP

2. About CliftonLarsonAllen • CliftonLarsonAllen (CLA) is the union of Clifton Gunderson and LarsonAllen • This merger brings over 100 years of combined experience • The nation’s newest top ten public accounting and consulting firm • More than 3,600 professionals operating from more than 90 offices • CLA’s Government niche (Federal, state and local governments) is a significant part of the company’s firm-wide public sector practice

3. About CliftonLarsonAllen (cont’d) • Assurance and related IT services include a sampling of the following: • CFO and Non-CFO Act audits and advisory services • State and Local Government Financial and Compliance Audits • Internal control/FMFIA reviews/other financial management advisory services • Statement on Auditing Standards SSAE-16 examinations • Performance Audits • Federal Information Security Management Act services • Information system security reviews

4. About Summit Consulting LLC • Founded in 2003, specializes in cutting-edge quantitative techniques, including statistical sampling, financial modeling, and program evaluation • More than 60% of staff professionals have advanced degrees in economics, statistics, finance, and other quantitative disciplines • Summit has provided sophisticated consulting services for many federal agencies, including the Treasury, HUD, USDA, VA, DOT, DOE, and DOL • Affiliated experts and staff members testified at courts of law and other judicial settings for matters involving economics and statistics

5. Presentation Objectives • Define audit sampling and distinguish attribute vs. variables sampling, and statistical vs. non-statistical sampling. • Discuss relevant activities of a sampling project. • Define the project team members’ roles and responsibilities. • Provide real life case studies to highlight the use of sampling and sampling best practices. • Provide sampling best practices for performing audit, internal control, and program evaluation sampling projects.

6. Presentation Outline • Why sampling? • What comprises a sampling project? • Who does what during a sampling project? • Types of sampling designs • Real life examples of sampling design • What are some of sampling’s best practices? • How should sampling results be used?

7. Why Sampling? • Primary objective of sampling is to create a subset of elements from a population that is as similar to the population as possible to allow users to evaluate certain characteristics of the population • Benefits of sampling: • Reduces costs associated with evaluation • Increases efficiency of population evaluation • Provides the ability to quantify variability (i.e., range of values) in the population • Allows measurement of the sufficiency of the evidence obtained • Attribute sampling (test for deviations) vs. variables sampling (test for misstatements)

8. Why Sampling? (cont’d) • Statistical (projectable) sampling vs. non-statistical (judgmental) sampling • Based on a statistical sample, important projections can be made about the population: • Averages, totals, ratios • Data validation and quality • Trends identification • Projection of errors and identification of misstatements • Rate of errors/deviations • Sampling risk can be quantified

9. What Comprises a Sampling Project? Conclusion & Reporting Planning Implementation

10. What Are the Planning Activities of a Sampling Project? • Sampling Frame Formation and Population Analysis • Designed to ensure the dataset used for selecting the sample corresponds to the scope of the project and adequately addresses risk areas. • Clearly define the objective of the project and the population subject to testing. • Determine completeness of the dataset and validity of data. • Analyze the population characteristics and assign risks to the various characteristics/strata. • Deploy data checks to eliminate duplicates and reversals.

11. What Are the Planning Activities of a Sampling Project? (cont’d) • Sample Design • Dictated by the project’s objectives with consideration for the project’s constraints. • Two broad categories of statistical sampling: • Attribute sampling – used primarily in tests of controls • Variable sampling – used to estimate/project a quantity and often used to test the monetary value of account balances • Clearly define the sampling unit. • Create subgroups of observations with similar characteristics that correlate with audit risk.

12. What Are the Planning Activities of a Sampling Project? (cont’d) • Sample Size Determination • Predicated on the sample design, risk assessment, and precision requirements. • Estimate the expected misstatement/expected deviation rate to achieve a prescribed level of precision. • Tailor sample size to address emerging risks or issues. • Balance precision with team resources (i.e., time, people, and money).

13. What Are the Implementation Activities of a Sampling Project? • Sample Selection • Select the items from the sampling frame that will be tested. • Sample items should be selected in such a way that the sample is representative of the population. • Sample items should have a fixed, known and positive probability of selection.

14. What Are the Implementation Activities of a Sampling Project? (cont’d) • Sample Testing and Evaluation • Test the sampled items and evaluate sample results mathematically. • Determine whether unexpected test results could skew conclusions. • Determine whether initial assessments should be revised and the sample expanded.

15. What Are the Conclusion & Reporting Activities of a Sampling Project? • Extrapolation and Conclusion • Project the sample findings to the population. • Conclude on the rate of error of attribute sampling or dollar misstatement in variables sampling. • Reporting • Document the sampling plan, including evaluation considerations, calculation of the point estimates (best guess value) along with the confidence intervals, and conclusions reached regarding the population.

16. Who Does What During a Sampling Project? Conclusion & Reporting Planning Implementation

17. Who Wants to be a Millionaire? • Background: Agency uses statistical sampling to collect improper payments. By law, the agency can only collect the lower bound of the improper payment estimate. • Objective: To collect as close to the true improper payment amount as possible. • Challenge: Historically, using a simple random sample design, the agency under collects due to the lack of precision. • The point estimate (e.g., $15 million) is much greater than the lower bound (e.g., $300K).

18. Who Wants to be a Millionaire? (cont’d) • Solution: Stratify the population according to risk characteristics. Using the same sample size as the simple random sample design, precision improves and the lower bound gets closer to the point estimate. • Point estimate remains $15 million, but the lower bound increases (e.g., to $3 million).

19. Frequent Flyer Miles, Anyone? • Background: Agency uses statistical sampling to identify assets that are misstated on the general ledger. • Objective: To determine the size and extent of misstatement of the agency’s recorded asset balance. • Challenge: Assets are located in all fifty states and six territories. Using a simple random sample it is likely to select as many locations as the sample size (i.e., a great deal of traveling).

20. Frequent Flyer Miles, Anyone? (cont’d) • Solution: Implement a multistage sampling design in which locations are selected in the first stage and assets to be tested are selected in the second stage within the selected locations. The sample controls the number of locations visited and the number of assets evaluated. • Benefit: Substantially reduce travel costs and significantly enhance audit efficiency and effectiveness.

21. Should I Stop or Should I Go • Background: Agency uses statistical sampling to determine the extent of misstatement in the UDO balance at year end. • Objective: To achieve precision by evaluating as small a sample size as possible. • Challenge: No prior history to estimate the sample size for a given precision. It is 9/30 and only two weeks remain to finish the audit.

22. Should I Stop or Should I Go (cont’d) • Solution: Prepare a sample with a randomized sort-order and evaluate sample batches sequentially to determine if the prescribed precision requirements have been achieved. If yes, STOP. If no, evaluate the next batch according to the random order. • Benefit: Performing the least amount of work to achieve the prescribed precision requirements.

23. Sampling Best Practices - Audit • Sampling frame corresponds to the target population. • Every transaction in the frame is substantive (i.e., net out all the reversals and eliminate phantom transactions). • Use stratification to test all large transactions with a dollar amount above a certain threshold. • Define methodology for determining the point estimate: Mean, proportion, totals, ratios. • Create a unique sort, and set random seeds to ensure sample can be reproduced. • Careful documentation of sampling design, sample size, expected precisions, point estimates, and confidence intervals.

24. Sampling Best Practices – Program Performance Evaluation • Sampling frame corresponds to the target population • Every subject in the frame is substantive, i.e., eliminate phantom subjects • Select sample designs according to data structure and availability • Define performance metrics and all the components • Create a unique sort, set random seeds to ensure sample reproducibility • Careful documentation, sampling design, sample size, expected precisions, point estimates, and confident intervals

25. How Should Sampling Results be Used? • Point estimate from a statistical sample is the unbiased and best guess of a population’s true value • Precision and confidence interval quantify how point estimates are likely to change from one sample to the next • Lower (upper) bound of a confidence interval is a threshold value below (above) which a population value is unlikely • These statistics determine the likelihood of an estimated population value (e.g., population error rate) • Based on these likelihoods, certain population values could be rejected for being extremely unlikely

26. Contact Information Albert J. Lee, Ph.D., CEO, Summit Consulting LLC albert.lee@summitllc.us 202-407-8300 Denise Wu, CGFM, CPA, Partner, CliftonLarsonAllen LLP Denise.Wu@cliftonlarsonallen.com 301-931-2050

27. Questions?