Using Consumer Surveys for Quality Management: A Modified PES as One Option

1. Using Consumer Surveys for Quality Management: A Modified PES as One Option Presentation to Cash & Counseling States Sara Galantowicz July 11, 2006

2. Agenda • Consumer Surveys and Quality Management • Modified Participant Experience Survey as one Survey Option • PES Overview • ME CD-PAS version • WV hybrid • Additional Issues for Consideration Consumer Surveys for Self-Direction

3. Consumer Surveys • Consumer surveys are one of many sources of quality data • Consumer experience • Knowledge, attitudes, and practices • Satisfaction • Where you want to end up is where you want to start • Research questions • Data gaps • Suspected problem areas • Mandates • Changes/policies requiring evidentiary support • CMS and other reporting • Who needs to know what Consumer Surveys for Self-Direction

4. Consumer Surveys Options • Designing your own instrument • Flexible • Specific • Reflects stakeholder input • Using/modifying existing instruments • Stealing can be good! • Efficient • Testing • Comparability • Either approach requires involving the right players and technical expertise Consumer Surveys for Self-Direction

5. Looking at Existing Tools • Multiple tools available • Vary on several dimensions • Target population • Domains or areas of emphasis • Method of administration • Role of proxy respondents • Length of survey • Resulting measures • Evaluation Criteria • Scope • Testing • Reliability and validity • Cost • Availability of benchmark data and support Consumer Surveys for Self-Direction

6. Participant Experience Survey PES is a technical assistance tool for States • Solicits participant feedback on HCBS waiver services/supports • Generates indicators for QA/QI • Developed by The Medstat Group, Inc. for CMS/DHHS • Experience survey for participants only • Not a satisfaction survey • No proxy version • Designed to be administered in-person • Optional • Extensively tested before release • Cognitive testing of individual items • Field testing to evaluate administration and tool properties Consumer Surveys for Self-Direction

7. PES Development • Combination of quantitative and qualitative methods • Work Group provided technical assistance and oversight, and specified priority areas • Review of related instruments • Draft items to operationalize domains • Cognitive testing (Phase I) • Field testing (Phase II) Consumer Surveys for Self-Direction

8. PES: Current Versions PES E/D • Frail elderly and adults with physical disabilities • 33 indicators • Free automated survey software available • Responses entered directly into laptops • Pull-down menus for data analysis • PES MR/DD • Adults with mental retardation or developmental disabilities • 51 indicators • Includes 8 “core” questions for participants with severe cognitive impairments • Automated version • PES BI • Adults with acquired brain injuries • Up to 58 indicators (two optional modules) Consumer Surveys for Self-Direction

9. PES: Domains • Access to Care • Are participants’ needs for personal assistance, adaptive equipment, and case manager access being met? • Choice and Control • Do program participants have input into the types of services they receive and who provides them? • Respect and Dignity • Are program participants treated with respect by providers? • Community Integration • Do program participants engage in activities and events of their choice outside their homes, when they want to? Consumer Surveys for Self-Direction

10. Sample Items • Access to Care • Do you ever go without eating when you need to? • Is this because there is no one there to help you? • Have you ever talked with your case manager or support coordinator about any special equipment, or changes to your home, that might make your life easier? • Choice and Control • Do you help pick the people who are paid to help you? • Would you like to help pick the people who are paid to help you? • When you are at home, can you eat when you want to? (MR/DD) • Can you talk to your case manager or support coordinator when you need to? Consumer Surveys for Self-Direction

11. Sample Items • Respect and Dignity • Do the people paid to help you treat you respectfully in your home? (E/D) • Do support staff in other places, such as at work, or at a day program, say “please” and “thank you” when they ask for something? (MR/DD) • Does anyone ever do mean things to you, such as yell at you? (MR/DD) • Community Integration • Do you like to go out to eat? (MR/DD) • Do you go out to eat? (MR/DD) • Do you get to help pick where you eat out? (MR/DD) • Is there anything you want to do outside your home that you don’t do now? Consumer Surveys for Self-Direction

12. Maine’s CD-PAS Survey • State seeking feedback from CD-PAS participants • Non-elderly adults with physical disabilities • Reviewed PES E/D and identified gaps • Crafted new survey items • Stakeholder input • Operating agency • Medicaid agency • Provider (Alpha One) • Consumers • Testing • Cognitive • Field Consumer Surveys for Self-Direction

13. Maine’s CD-PAS Survey • Domains • Program Referral and Support • Access to Services • Respect and Dignity • Technology and Community Integration • Interviewer comments and observations • Second year of use • Interviews conducted by staff from the Muskie School’s survey unit • Early results reflected political uncertainty • Modifications made based on findings Consumer Surveys for Self-Direction

14. Modifying the PES in Maine • Added items specific to self-direction • How PAs located and hired • PAs’ relationship to waiver participant • PA retention and back-up • Support from Alpha One Independent Living Specialists • Experience with self-direction • Retained relevant items from PES • Respect and dignity • Unmet need for personal assistance • Adaptive equipment • Physical and verbal abuse, theft Consumer Surveys for Self-Direction

15. Crafting the new items in Maine • General design principles and traps to avoid • Consistent response categories • Appropriate response categories • Adequate response categories • “Problem” words • Sound-alike words • Compound or double-barreled questions • Jargon and unfamiliar words • Complicated concepts • Relevance to consumers (screening questions) • Sensitive, appropriate language • Hypothetical questions • Scales • Specificity of times, dates, frequency, and amounts Consumer Surveys for Self-Direction

16. Testing the new items in Maine • How people answer a survey question • Comprehension • Retrieval • Estimation • Response • Each area is a potential source of bias • Cognitive testing: determining if the question you think you are asking is the one waiver participants are hearing and answering • Iterative process of asking draft questions with probes • Small n, longer interviews • Discuss and act on results • Resulted in many changes Consumer Surveys for Self-Direction

17. Field Testing in Maine • Larger n, shorter interviews • Evaluate survey logistics • Reaching consumers • Scheduling interviews • Interview length • Interview acceptability • Obtain baseline data • Evaluate appropriateness of tool • Examine utility of findings Consumer Surveys for Self-Direction

18. West Virginia’s Combined PES Tool • State seeking feedback from both traditional E/D waiver participants and those enrolling in new Personal Options waiver • Developed hybrid tool with separate skip patterns for traditional and self-directing waiver participants • Multiple players • Medicaid agency • Operating agency • UCED staff • Stakeholder advisory group Consumer Surveys for Self-Direction

19. WV’s Combined PES tool (con.) • Many common items • Access to care/unmet need • Respect and Dignity • Abuse/theft • Choice and control items for traditional waiver • Primarily same questions from PES E/D • Some new items • New Program Referral and Support items for Personal Options participants • Modified the Maine questions • Additional items on resource consultants and financial management services Consumer Surveys for Self-Direction

20. Testing the WV Survey • Iterative development process • Reviewed PES E/D and Maine tool • Developed new questions • Lots of review and feedback • Consumer review • Expert review • Limited cognitive testing • Field testing to begin this summer Consumer Surveys for Self-Direction

21. Issues for Consideration • What are your goals for the survey effort? • Whom do you want to survey and why? • Where will these data fit into your QM strategy? • What additional information would you like to capture? • With whom do you plan to share the data? • One-time vs. longitudinal use • What is your target population? • Sampling frame • Criteria for inclusion or exclusion • Sampling plan • Sample size • Stratification • Proxies? Consumer Surveys for Self-Direction

22. Additional Issues for Consideration • Interviewer Selection and Training • Who will be conducting interviews? • Who will attend training? Who will conduct it? • Level of effort • Ongoing data collection vs. limited period • Contacting participants • How will sample respondents be contacted? • State staff, case managers, other • Scheduling and briefing respondents • Data collection on non-respondents Consumer Surveys for Self-Direction

23. Additional Issues for Consideration • Pre-survey information • What kind of information can you access about participants before the interview? • How can you use this information to tailor the interview and provide appropriate supports • Assess ability to complete survey • Special issue for consideration! • Timeframes • When will interviews start • How long will the process last • When would you like to be able to have and share results Consumer Surveys for Self-Direction

24. Other Issues for Consideration • Electronic data entry • Psychometrics • Inter-rater reliability studies • Validity concerns/cognitive screening • Data Analysis • Who will develop the analysis plan? • Who will conduct the analysis • Acting on and sharing your results • Who should see the findings and how best to share them? • Determining appropriate QI projects • Involving stakeholders • Need to consider this up front Consumer Surveys for Self-Direction