Chapter 6. Making Success Measurable Through Quality Control

1. Chapter 6.Making Success Measurable Through Quality Control

2. Chapter 6. Section 1. Introduction to Quality Control

3. Introduction • How do you know the quality and accuracy of your rent calculations? • Wait for HUD to tell you? • Close your eyes and hope for the best? • Assume that good people lead to good outcomes? • Wait until you tally SEMAP scores? • Or …..

4. Introduction • Are you interested in a better way that could: • Enable you know when to take action on performance issues, • Help document the improvements in error reduction you are making, • Enhance the quality of the services you provide?

5. Introduction • A good quality control program can be that better way that helps you: • Prevent errors • Documents compliance with program requirements • Demonstrate performance improvement over time

6. Introduction • An effective quality control program: • Facilitates conversation about expectations and performance for individuals and the organization as a whole. • Enables program managers to be proactive, rather than reactive

7. Introduction In this session, we will talk about: • Three types of quality control • Illustrate the thought process needed to develop a QC program • Provide sample QC tools • Provide guidance on communicating with your staff about quality issues.

8. Introduction • We will focus on HUD’s expectations related to income and rent determination • BUT, the QC concepts we discuss apply equally well to other parts of a PHA operation.

9. Why Quality Control? • Quality control can help to confirm what you already know. It can: • Provide documentation that supports recommendations you want to make, • Generate statistics that document staff performance, • Provide valuable information on a systematic basis to share within and outside your organization. .

10. Why Quality Control? • QC can be even more helpful if the results are a surprise. Then the results may highlight: • Problems of which you were not aware • Good news about improved performance or excess capacity. • Were there aspects of the RIM review for your agency that were a surprise?

11. Chapter 6. Section 2.Types of Quality Control

12. Preventive Types of Quality Control Continuous Types of Quality Control Periodic

13. Preventive QC • Preventive QC is checking on the quality before the transaction is completed. Use Preventive QC when: • The consequences of errors are high, or • The error is not recoverable. • Examples: • Supervisors sign termination letters even though specialists sign other correspondence • Occupancy supervisor reviews 100% of rent calculations for new staff (for some period of time).

14. Preventive QC • More examples: • Intake specialist presents a “dry run” of the briefing to staff before doing the real thing in front of participants. • What examples of preventive QC occur at your PHA? • Try to identify at least one additional area (related to income and rent) where preventive QC might be helpful.

15. Limits of Preventive QC • Preventive QC can be an important tool but what are its limitations? • PHA’s don’t have time or staffing to double and triple check every transaction –you need to prioritize. • Use Common Sense: Not everything requires this level of effort.

16. Periodic Quality Control • In contrast, periodic quality control happens after the fact. Work has already been done and reported. • It’s purpose is to take a “picture-in-time” assessment of PHA operations that will help improve the program and prevent future errors. • Sound familiar? Yes – HUD’s RIM reviews and SEMAP audits are kinds of periodic QC.

17. Periodic Quality Control • During a periodic review HUD systematically looks at: • Policies, procedures, and practices and • Individual files. • But you don’t have to wait for HUD. You can also do your own. You could: • Look at the same things HUD will be checking, or • Focus your periodic review differently to learn even more.

18. Limits of Periodic QC • Periodic QC can provide an accurate point-in-time assessment, but: • It doesn’t help managers see trends • Results come infrequently so it doesn’t help managers take corrective action during the year. • The level of effort required may seem overwhelming (but HUD’s RIM Review guide provides a good model for the review).

19. Continuous Quality Control • Continuous QC is used to help with day-to-day management. • It involves analyzing a few (but very important) factors on a regular, continuing basis. For example: • Tracking on-time reexaminations each month • Looking at accuracy rates by individual or transaction

20. Continuous QC • Continuous QC helps managers: • Focus more quickly on vulnerable areas. • Look at trends, rather than point-in-time information. • HUD does some continuous QC of 50058 submissions and uses the data to: • Send error reports to PHAs, and • Assess in general terms how PHAs are doing. • But only the PHA can implement an effective continuous QC program.

21. Limits of Continuous QC • Continuous QC is a powerful tool. • But, remember, the scope of the Continuous QC is limited. • This means: • Continuous QC provides only a limited picture (hopefully of the key areas) • Continuous QC should be supplemented by Periodic Reviews that take a broader view.

22. Continuous QC Example Page 6-11 and 6-12 • Look at the “Error Analysis” chart • QC results are shown by quarter & individual • Caseload = annual caseload • Errors = things HUD would call findings • Comments = errors that would not be findings • Types of errors are defined on page 6-12

23. Continuous QC Example • What issues does this report raise for you? • What are your concerns about: • Productivity? • Types of errors? • Error Rates? • What are the implications of the information you see for this PHA?

24. Chapter 6. Section 3.Designing a QC Program

25. Designing a QC Program • Now that we have discussed each type, it’s time to think about how to design your program. • Before you begin it is very important to think through: • What you want your QC program to look at. • What you will do with the information.

26. Designing a QC ProgramWhere to Start • To focus your program ask three questions: • What are the outcomes you want to achieve? • What would success look like for each outcome? • What is the best way to confirm (track, measure) performance in each area? • Then decide: • Which measures are useful and feasible. • How you are going to record the information. • Who needs to know the results.

27. A Simple Example- Sleep • What is my desired outcome? • A good night’s sleep • What would success look like? • Eight hours of uninterrupted sleep • What is the best way to measure? • An independent 3rd party could observe my sleep. • I could track myself the time I go to bed, time I wake up, and number of interruptions.

28. Simple Example (Cont) • Decision: • Having someone watch me sleep could be expensive and disruptive • Instead, I will track sleep patterns myself • I’ll write this down in a notebook • No one needs to see it but me (unless I think maybe my doctor should know)

29. A More Serious Example – Utility Allowances • What is my desired outcome? • Current and correct utility allowances will always be used in rent calculations. • What would success look like? • Utility allowances would be used correctly in 100% of the files. • Are there other ways you would describe success re: utility allowances?

30. A More Serious Example – Utility Allowances • What are possible ways to measure the outcome? • 100% file review in advance (preventive) • 100% file review after the fact • Sample file reviews after the fact • What else could we do to see if the correct utility allowances are being used consistently?

31. A More Serious Example • Decision: • 100% review would be too time consuming (before or after) • Therefore, review a sample of files (after the fact) to see how often staff have used the correct utility allowance • Generate a report that summarizes errors by employee, by program type, by transaction type.

32. Thinking About Outcomes • Choosing the outcomes you want to use is an important part of the design process. • Outcomes can be expressed in very general or very specific terms. • Some “big picture” outcomes might include: • Consistency in data collection, documentation • Complete and accurate verifications • Accuracy in rent calculations • Timeliness • Complete, accurate reporting

33. Thinking About Outcomes • Even broader outcomes might include: • No findings on the next RIM review • Informed decision making by applicants, participants and HCV landlords • Fair and equitable treatment • Equitable workload • Being a SEMAP high performer • Positive perceptions – we’re a good agency.

34. Thinking About Outcomes • Caution: Generally more specific outcomes are better understood and easier to measure. For example: • Reduce departmental error rate to less than one percent • Complete 100% of annual reexaminations accurately and on-time • The more specific the outcome, the easier it is to track performance.

35. Designing a Program to Support Outcomes • Page 6-20 through 6-21 show examples of turning desired outcomes into a QC plan. • Follow along on the chart • We will walk through Row 1 and Row 2 together

36. The End Result • The end result of this thinking will be a Quality Control Plan. (see page 6-25). • The planning itself is more important that the format for the plan. • Think through the process before you begin data collection • Collect information about the items that have the most effect on error reduction • Be realistic about what data you can collect.

37. What’s Next? • The balance of this chapter discusses 5 important “how to” aspects of QC: • Conducting file reviews • Conducting a “general” review of operations • Using reports to track quality • Addressing staff skills and skill gaps • Communicating about error reduction and quality control

38. Chapter 6. Section 4.How to Aspects of Quality Control

39. Conducting File Reviews • Files reviews are used: • To determine accuracy and completeness • To identify potential policy and management issues • “Files” for this purpose include the 50058 record as well as paper files. • Caution: A file review can tell you what has happened, but not necessarily why.

40. Conducting Tenant File Reviews • Sample size and characteristics of files selected depend upon the purpose. • Generally PHAs use a “directed” sample. • Directed means selecting the files that contain the characteristics you want to monitor. • Page 6-28 shows characteristics that may be important to consider for the sample. • Not every characteristic must be represented each time you review files.

41. Conducting File Reviews • For Continuous QC • Purpose is timely feedback on few issues. • File Selection Priorities would be: • Each specialist • Basic transaction types • Files completed over time

42. Conducting File Reviews • For Periodic QC • Purpose is a more comprehensive review. • File Section Priorities: • Each specialist (but perhaps more files) • More transactions types • Larger cross section of income sources and household types

43. Conducting File Reviews • Tools for Selecting Files: • PIC reports can generate lists for some transactions. (See Appendix A, A-25) • PHA vendor software may be able to generate reports that list files in categories. • How many files do I need to review? • QC sample should be sufficient to meet SEMAP requirements

44. SEMAP File Sampling • Useful guideline for determining number of files to review

45. Conducting File Reviews • Logistics • Use a file checklist to document your review. (RIM checklists are a good start.) • Complete 50058 calculations without looking at the completed calculations in the files. • Look for consistency between the paper and electronic files.

46. What The Reviews May Show • Key RIM Review programmatic findings: • Earned income/pension/public assistance calculations • Incomplete/outdated PHA policies • Data not being transmitted to PIC • Lack of verification of income and deductions • File data doesn’t match data used to calculate rent • Improper calculation of utility allowances

47. What the Reviews May Show • RIM reviews also found errors related to automation. • In general, these errors include: • Programming problems(software vendor) • Errors related to the way PHAs use automated systems such as: • Not attending to error reports and • Inadequate training for staff on the system • See page 6-32 for details

48. Conducting the “General” Portion of a Periodic Review • Periodic Reviews enable PHAs to take the same broad look that HUD takes during a RIM Review. • The agenda: • Look at policy and procedural documents • Conduct an expanded file review • HUD’s RIM checklists are good models.

49. Conducting the “General” Portion of the Review • You may need to add to HUD’s checklists: • Questions covering PHA-specific policies or performance standards • Protocols for checking how PHA-specific software handles income, rent and subsidy determinations.

50. Using Reports to Track Quality • Perspective is important. • Identifying individual errors is important but it’s only one aspect of QC. • Looking at trends (performance over time) provides important insights that lead to error reduction. • Looking at trends can also help set performance standards.