RRFSS Evaluation: Issues and Strategies

RRFSS Evaluation:Issues and Strategies RRFSS Workshop, June 19th 2002 Catherine Bingle Sarah Feltis

RRFSS Evaluation Project 2001-2 • Project background and development • Evaluation objectives and framework • Methodology, tools, respondents • Results • Key issues and strategies

Evaluation Project Contributors • RRFSS Evaluation Work Group • RRFSS Working Group 2001 • Research assistant and project consultant • PHRED Funding

Background • Evaluation Work Group was formed May 2001 • Evaluation framework developed • Identified need for support • PHRED funding supported hiring of Project Consultant and Research Assistant

Evaluation Objectives • Evaluate the current state of RRFSS • Evaluate the effectiveness of system processes • Evaluate the utility of system • Identify areas for improvements and successes of RRFSS

Expectations of the Evaluation • Inform changes to RRFSS procedures • Provide a baseline/template for future RRFSS evaluations • Contribute to continuous quality improvement

Evaluation Framework • Developed by Evaluation Work Group • Looked at CDC and WHO surveillance system frameworks • 4 KEY components • Process Issues • Collaboration Among Sites and Key Players • Utility or Usefulness • Cost Effectiveness

Data Collection Tools • 1st drafts completed in October • At least 4 rounds of comment and revision completed in November after review from the Working Group • In total 5 self- administered questionnaires and one semi-structured interview were developed and used to collect data

Data Collection Tools • RRFSS Representative Questionnaire • Focus • questionnaire development, project administration, awareness and use of results, data analysis and quality, costs, partnerships, benefits and areas for improvement • MOH Questionnaire • Focus • familiarity and satisfaction, perceptions of value and impact on health units, feedback on ways to sustain development

Data Collection Tools • Program Questionnaire • Focus • program staff awareness, satisfaction with content of the questionnaire, decision making process, access to results, use of results, impact on programs, and areas for improvements • Non-participating Health Units • Focus • familiarity, health unit interest in future participation, reasons for not participating, and perceived value of RRFSS

Data Collection Tools • ISR Interview • Focus • working relationship between ISR and health units, questionnaire development process, costs issues and expectations, quality of data and methods, future development of RRFSS • MOHLTC Questionnaire • Focus • familiarity, impact and value, interest in partnership with RRFSS, strategies for increasing visibility and participation

Data Collection and Response Rate • All 6 tools were disseminated December 2001 • All data collected by end of January 2002 • 109 of 151 (72%) of targeted respondents participated • RRFSS Representative - 11 of 12 HU’s • MOH - 10 of 12 HU’s • Non-participating HU’s - 17 of 24 HU’s • Program - 69 of 101 program managers/staff • MOHLTC - 1 of 1 • ISR interview

Data Analysis • Quantitative data • Entered and analyzed in SPSS • Frequencies • Qualitative Data • Entered into Word files for content analysis

Results • Process Issues • Collaboration • Use and Usefulness • Cost Issues • The Future

Process Issues • Questionnaire development processes • Satisfaction with RRFSS questionnaire • Quality of data / methods • Data analysis • RRFSS dissemination • RRFSS awareness

Process Issues Questionnaire Development Processes • RRFSS Reps (RR) - RR’s and Program Staff (PS), mostly managers, involved in processes for Q development and/or revision • PS - about 2/5 were involved • RR and PS - nearly all very or somewhat satisfied with processes • RRs more satisfied with new module development than with revision processes

Process Issues Questionnaire Development ProcessesAreas for Improvement • PS concerns: • Q dev. process complexity and length • Need to involve staff more • RR concerns and suggestions: • greater clarity and documentation of policies, responsibilities, procedures, timelines, decision rationale • greater commitment to process • ISR suggestions (in addition to some of above themes) : • procedures to assure more equality of HU access to opportunities for new questions / changes

Process Issues Satisfaction with Questionnaire • RR - generally meets HU needs • MOHs - split bet. very and somewhat satisfied • PS - more somewhat than very satisfied, esp. with number of questions relevant to their programs

Process Issues Satisfaction with QuestionnaireAreas for Improvement • PS and RR - opportunities for more questions - both new topics and expansion of topics • RR - explore rotating core • RR - explore adjusting core : optional balance

Process Issues Quality of Data / Methods • Most RRs see sample size as adequate • Those who do not indicated need for larger sample for subpopulation analyses • Most viewed response rate as good, some excellent. Likely based on partial info 67% completion rate, June ‘01. • Question quality viewed (ISR) as good overall

Process Issues Quality of Data / MethodsAreas for Improvement • RR and ISR - further strengthen Q quality procedures - Q review, editing, & testing. • ISR - longer Q testing - pilot study (n=100) where possible (responsiveness important too) • Assessment of existing modules - keeping flawed modules over time problematic • Need feedback mechanism for problems from data user back to Q design • Need yearly conference to discuss methods

Process Issues Data Analysis • RR - dissatisfaction with timeliness of data receipt at HU • RR - All HU's had begun to analyze RRFSS data • RR - HU's have not enough skilled staff to analyze data and make it usable

Process Issues Data Analysis • ISR, RR - analysis getting short shrift • MOH, RR - analytic capacity a key sustainability challenge • RR - major staff capacity differences among HU's • Key factor for RR's who were somewhat dissatisfied overall with RRFSS • Key concern of HU's not to participate

Process Issues RRFSS Awareness in HU’s • RR - most said most or all of HU staff aware of RRFSS • PS - 2/3 somewhat, 1/3 very familiar with Q content • MOH - most very familiar with current status of RRFSS, most somewhat familiar w/content • RR - ½ assess awareness raising processes in HU's as very effective, ½ somewhat eff.

Process Issues RRFSS Dissemination in HU’s • RR - most HU's had disseminated results in some manner; ½ to broader community • PS - 2/3 had accessed results. Of these: • Most had via their epi/analyst staff • Most very satisfied w/how results provided • 1/3 needed support to use results - all said it was available. 15% weren't sure if needed supports; ½ didn't need.

Process Issues RRFSS Awareness & Dissemination in HU’sAreas for Improvement • PS - dissatisfaction w/access to results most frequently due to results not being available • PS - need for repeated dissemination • PS - need better explanation of data methods

Process Issues Overall Satisfaction w/RRFSS Implementation • Most RR very satisfied. Those "somewhat" concerned about cost and staff resources needed for participation • Most PS were satisfied, 42% very satisfied w/how RRFSS is implemented in their health unit

Process Issues RRFSS Awareness Among Non-Participants • At non-participating HU's (epi / alt.), ½ very, ½ somewhat familiar w/RRFSS status • Most somewhat familiar w/RRFSS content, some very familiar • PHB rep - very familiar w/status, somewhat familiar w/content

Collaboration • Satisfaction with partnership • Contributions to decisions, activities • Perceived work group effectiveness • Ways to improve partnership

Collaboration Issues Satisfaction with Partnership • MOH - most very, some somewhat satisfied with how partnership works • RR - half very, half somewhat satisfied • RR - most say partnership is worth investment • ISR very satisfied with effectiveness of working relationship among partners • ISR - benefits of partnership: • provides ISR w/feedback on data use • local value of study • collegial, knowledgeable, committed working group

Collaboration Issues Contributions to Group Decisions and Activities • RR - most feel all HU's have equal voice in decisions, although those involved longer seem to influence decisions more • HU's do not make equal contributions, due to differences in HU staffing, ability to support involvement, choices to be involved • Expectation that these differences will recede as more HU's gain experience

Collaboration Issues Perceived Work Group Effectiveness • RR- Working G, Advisory G, and Ad Hoc WG: roughly split 1/2 very, 1/2 somewhat effective • Analysis G and Evaluation G - more somewhat than very effective • ISR perceived as very effective • ISR very satisfied with working relationship among partners

Collaboration Issues Perceived Work Group EffectivenessChallenges; Areas for Improvement • RR - some duplication between Adv. G and Work. G; difficulty of managing WG meetings • ISR - • Adv. G'd challenge representing larger group • Increasing # of HU's increases coord. difficulty • Range of expertise among HU's • Meeting structures and scheduling • Critical role of Adv. Group - stability must be ensured

Collaboration Issues Ways to Improve Partnership • RR - dedicated funding to Adv. G functions • Develop MOU outlining policies, procedures, expectations, roles, conflict resolution proc. • Funding for central coordination of RRFSS • Annual face to face meetings • Schedule chairs and recorders for WG meets • Better documentation of Adv. G and TOR

Use and Usefulness • Use of Results • Benefits to Health Units from RRFSS • Impact on Program Planning & Evaluation • Impact of Not Having RRFSS • Importance for Public Health Surveillance • Needs Not Met By RRFSS

Use Issues Use of Results - RR • Virtually all HU's had used RRFSS data • Use highest among epi's and program staff. Managers, media, and MOH also cited. • Use highest among CDP programs. • Use also frequently cited for Env. HP & VPD programs; research & communications staff

Use Issues Use of Results - PS • 38% used results in their programs • Of those who had not, ½ said this was due to results not yet being available • Remainder said they had not yet had time • Some indicated data not of sufficient importance / relevance to use

Use Issues Use of Results - RRBarriers; Areas for Improvement • Barriers to use: • time required for data analysis and administration • awareness among PS • Key needs: • funding for central project coordination • common syntax files • better dissemination efforts in HUs

Use Issues Key Benefits from RRFSS - RR • Current, timely, local data, filling data gaps • Data for emerging and locally relevant issues • Staff use for planning and evaluation • Better quality than typical HU surveys • Strengthened partnership among HUs and epidemiologists, and w/community partners • Increased understanding of surveillance

Use Issues Expected Impact on Program Planning and Evaluation • All MOH -- RRFSS results will have impact on HU ability to meet PP&E standards • 8 RR agreed, 3 "too early to say" • 44% of PS agreed; 26% did not; 30% not sure • Non-part. HU's - 14 agreed; 3 "too early" • PHB rep - agreed

Use Issues Impact of Not Having RRFSS • MOH, RR, & PS identified same range of negative impacts on PH effectiveness: • Reduced access to local, timely, adaptable, continuous data • Reduced ability to measure program objectives, reduced effectiveness in PP&E, and overall effectiveness / accountability • Need for additional investment in HU- and topic-specific surveys

Use Issues Importance for Public Health Surveillance • Majority (8/11) of RRFSS Reps were “very satisfied” w/RRFSS as surveillance method • Majority (8/10) of the MOH’s from participating health HU’s view RRFSS as a “very important” tool for public health surveillance • 11 of 17 non-participating HU’s also view as a “very important”

Use Issues Needs not Met by RRFSS • Virtually all HU’s indicated the need for other community health survey work beyond RRFSS • Child and youth health assessment • Surveying of sub-populations e.g. pregnant women • HU specific topics • Sensitive topics • Information not obtained through telephone survey

Costs and Cost-Effectiveness • Key Staff Involvement in RRFSS • Return on Investment • Cost Reduction Strategies

Cost and Cost Effectiveness Staff Involvement in RRFSS • Time • Annual estimated key staff = 0.68 FTE per HU (range 0.16 to 2.0 FTE per HU) • WHO • Epidemiologist • Other research staff • Activities • New module dev., data analysis, and RRFSS WG meetings • Data dissemination and presentation of data to HU staff

Cost and Cost Effectiveness Return on Investment • RRFSS Reps - (5/11) rated as “good” and 3 rated as “excellent” (time, money, and resources) • MOH - divided between “very satisfied” and “somewhat satisfied” with their HU’s investment in RRFSS • Program Savings - most RRFSS Reps indicated “to early to say” whether RRFSS had resulted in any program saving or would create revenues for HU in future

Cost and Cost Effectiveness Cost Reduction Strategies • Support for central coordination of RRFSS • Encourage partnerships with other organizations • Over time - improvements in organization and efficiency

The Future • Sustainability Challenges • Expanded HU and Provincial Participation • Strategies for Increasing Participation • Suggestions for Raising RRFSS Visibility

The Future Sustainability Challenges • MOH Perspective • Adequate staff/resources • Rapid analysis of data • Funding/Cost • Keeping program staff involved • RRFSS Rep’s Perspective • Cost/Funding • RRFSS administrative work • Questionnaire content

The Future Sustainability Challenges • ISR Perspective • Whether or not able to do all 38 HU • Need signed commitment from HU earlier, and for longer term • Working group needs support for methodology improvements and analysis

RRFSS Evaluation: Issues and Strategies