Evaluating Effectiveness of Participation projects Evaluation design; learning from EEP project EOI submissions

1. Evaluating Effectiveness of Participation projectsEvaluation design; learning from EEP project EOI submissions Cochrane Consumers and Communication Review Group (CC&CRG) Dr Sophie Hill and Helen Dilkes

2. A practical perspective on Cochrane support provided during Expression of Interest (EOI) submission development phase Who requested support 5 metropolitan hospitals 3 regional health services What support was sought by EEP projects How Cochrane responded Providing support: a practical perspective; up until now who requested assistance what they requested how Cochrane responded Cochrane response in more detail Providing support: a practical perspective; up until now who requested assistance what they requested how Cochrane responded Cochrane response in more detail

3. EEP projects address DHS� key policy levels of the health service system Summary of EEP project aims (8 projects): Improve the flow/quality of patient care; Increase family involvement; Give more appropriate care; Implement quality improvement plan; Improve whole community communication; Embed consumer participation in hospital process/practice; Improve patient safety. Who requested assistance 5 city hospitals 3 regional health services Who requested assistance 5 city hospitals 3 regional health services

4. Interventions: Patient information and staff education; Implementing new models of patient care; Comparing models of consumer participation in health service planning; Implementing consumer participation activities, or staff training around how to set up consumer committees; Consumer partnerships. Patient information; staff education around clinical practice guidelines; Implement new model of care including staff training; Staff training in how to set up consumer committees; New model of patient care; Models of consumer participation in health service planning; New model for health service planning; Implementing consumer participation activities; Consumer partnerships.Patient information; staff education around clinical practice guidelines; Implement new model of care including staff training; Staff training in how to set up consumer committees; New model of patient care; Models of consumer participation in health service planning; New model for health service planning; Implementing consumer participation activities; Consumer partnerships.

5. Outcomes: Patient/consumer/carer: Informed/actual involvement in decision making; feeling of input being valued; Anxiety, satisfaction, knowledge and awareness; Change in attitude and behaviour. Staff/professional: Doctor adherence to guidelines/standards; Knowledge/skills, change in attitude/practice; Admin time, adverse events, costs. Patient/consumer/carer: Informed decision making, uptake, anxiety, satisfaction, knowledge and awareness; Actual involvement in decision making, satisfaction; Actual consumer input, feeling of input being valued; Change in attitude and behaviour. Staff/professional: Doctor adherence to guidelines, behaviour change; Knowledge and skills, change in attitude and practice; Adherence to standards, admin time, number people seen, adverse events, costs; Uptake/actual implementation. Patient/consumer/carer: Informed decision making, uptake, anxiety, satisfaction, knowledge and awareness; Actual involvement in decision making, satisfaction; Actual consumer input, feeling of input being valued; Change in attitude and behaviour. Staff/professional: Doctor adherence to guidelines, behaviour change; Knowledge and skills, change in attitude and practice; Adherence to standards, admin time, number people seen, adverse events, costs; Uptake/actual implementation.

6. What support was sought Queries ranged from: One page documents with brief project outlines; Fully-fledged (EOI) submission. Queries consisted of: Assistance developing relevant outcomes; Advice on specific measurement tools; Study design/implementation issues. What assistance was sought Queries ranged from: One page documents with brief project outlines and a few questions; to, Fully-fledged Expression of interest (EOI) documents, put together by collaborating partners, requesting comment from Cochrane. Queries consisted of: Assistance developing relevant outcomes for consumer participation (this often meant that we needed more detail about the project background, aim/intervention in order to assist meaningfully) Advice on specific measurement tools (which we couldn�t answer at that time, without doing extensive searches); Study design issues (e.g. difficulty with control group/site and hence contemplating an alternative ITS study design); and, More in-depth questions about study design/implementation issues (e.g. number of participants, statistical and analysis issues) What assistance was sought Queries ranged from: One page documents with brief project outlines and a few questions; to, Fully-fledged Expression of interest (EOI) documents, put together by collaborating partners, requesting comment from Cochrane. Queries consisted of: Assistance developing relevant outcomes for consumer participation (this often meant that we needed more detail about the project background, aim/intervention in order to assist meaningfully) Advice on specific measurement tools (which we couldn�t answer at that time, without doing extensive searches); Study design issues (e.g. difficulty with control group/site and hence contemplating an alternative ITS study design); and, More in-depth questions about study design/implementation issues (e.g. number of participants, statistical and analysis issues)

7. How Cochrane responded Assistance - minimal to more extensive; Listening and conversations to clarify issues; Created a response document outlining the project - focus on aim, research question, intervention, outcomes, references. How Cochrane responded Assistance ranged from minimal to more extensive; Listened and had conversations with some applicants to clarify issues; We created a response document that outlined the project (PICOT), with a focus on aim, research question, intervention, outcomes, target participants, references; Assistance ranged from one contact via email to several conversations and to several drafts of the response document; there were some queries we could not address at that time (statistical and analytical) and felt it was not appropriate at the EOI stage.How Cochrane responded Assistance ranged from minimal to more extensive; Listened and had conversations with some applicants to clarify issues; We created a response document that outlined the project (PICOT), with a focus on aim, research question, intervention, outcomes, target participants, references; Assistance ranged from one contact via email to several conversations and to several drafts of the response document; there were some queries we could not address at that time (statistical and analytical) and felt it was not appropriate at the EOI stage.

8. We created a response document that aimed to assist EEP applicants to: Narrow the research focus/project scope - developing a research question; Craft/hone the intervention; Expand the range of outcomes; Conceive of EEP project within a specific research framework e.g. CBA. We created a response document that aimed to assist applicants to: Narrow the research focus and project scope; Craft/hone the intervention; Expand the range of outcomes; Conceive of the study within a specific research framework e.g. CBA. We created a response document that aimed to assist applicants to: Narrow the research focus and project scope; Craft/hone the intervention; Expand the range of outcomes; Conceive of the study within a specific research framework e.g. CBA.

9. Response document consisted of: A summary of the project; With a parallel summary - teasing out the issues, commenting/expanding on; Headings: research question, aim, interventions, comparison/control, outcomes, study design, references; Most applicants found the response document useful. Response document consisted of: A summary of the project as described by the applicant; With a parallel summary that teased out the issues and commented and expanded on the applicant�s summary; Document headings: research question, background, aims, interventions, comparison/control, outcomes, study design, references (around study design and range of outcomes) Most applicants stated that they found the response document very useful PICOT format: participants, intervention, comparisons, outcomes, target Response document consisted of: A summary of the project as described by the applicant; With a parallel summary that teased out the issues and commented and expanded on the applicant�s summary; Document headings: research question, background, aims, interventions, comparison/control, outcomes, study design, references (around study design and range of outcomes) Most applicants stated that they found the response document very useful PICOT format: participants, intervention, comparisons, outcomes, target

10. Framing a research question: Ambitious notions of what could be achieved; Framing a research question helped to focus/narrow the study; Research question based on PICOT format: participants, intervention, comparisons, outcomes, target; Manageability of study scope; Contained study - meaningful findings. Research question: focus the research and narrow the project scope In many cases people had ambitious notions of what they could achieve and we drew their attention to this and assisted to narrow the research focus Framing a research question (and a title) helped to focus the study Research question is assisted by the PICOT format: participants, intervention, comparisons, outcomes, target; A narrow focus for the study enables manageability of study scope Keeping it small leads to meaningful findings in relation to cause and effect (aims relate to outcomes) Research question: focus the research and narrow the project scope In many cases people had ambitious notions of what they could achieve and we drew their attention to this and assisted to narrow the research focus Framing a research question (and a title) helped to focus the study Research question is assisted by the PICOT format: participants, intervention, comparisons, outcomes, target; A narrow focus for the study enables manageability of study scope Keeping it small leads to meaningful findings in relation to cause and effect (aims relate to outcomes)

11. Developing the intervention: Craft/hone the intervention Some applicants had difficulty conceiving of their project in terms of intervention and we assisted in this Developing an intervention that reflects the intention of the study (based on the purpose of the study and other background information from applicant) Expand the range of outcomes We assisted with expanding the range of outcomes, focusing on the causal effects (positive and adverse) of the intervention Reference to Nilsen systematic review framework offered a broad range of outcomes Craft/hone the intervention Some applicants had difficulty conceiving of their project in terms of intervention and we assisted in this Developing an intervention that reflects the intention of the study (based on the purpose of the study and other background information from applicant) Expand the range of outcomes We assisted with expanding the range of outcomes, focusing on the causal effects (positive and adverse) of the intervention Reference to Nilsen systematic review framework offered a broad range of outcomes

12. Conceiving of the study within a research framework: Fitting within a research framework; Considering Controlled Before-and-After (CBA) study design issues (comparison/control etc); Alternative study designs e.g. Interrupted Time Series (ITS), or Preference trial; Highlighting relevant references (CBA studies, a framework of indicative interventions/outcomes). Conceive of the study within a research framework Fitting the study within a research framework; i.e. considering CBA study design issues (comparison/control etc) in relation to interventions and outcomes; and/or Suggesting alternative study designs e.g. ITS, preference trial highlighting relevant references (individual CBA studies, Nilsen systematic review) Conceive of the study within a research framework Fitting the study within a research framework; i.e. considering CBA study design issues (comparison/control etc) in relation to interventions and outcomes; and/or Suggesting alternative study designs e.g. ITS, preference trial highlighting relevant references (individual CBA studies, Nilsen systematic review)

13. EEP project applicants� comments/issues: DHS had big expectations; Difficulty finding control site, ethical issue; Qualitative methodology more appropriate; Some health services needed encouragement to apply; Requesting detailed information - measurement tools, analysis. Other issues: EEP project applicants� comments Applicants commented that DHS had big expectations in relation to study design and implementation Comments about difficulty finding control site, or the ethical issues about the notion of control; qualitative methodology considered more appropriate by applicant A qualitative methodology was considered more appropriate than the CBA study design; Some applicants requested detailed information about measurement tools and statistics during submission phase that Cochrane could not supply at that time; Some health services needed encouragement to apply citing the onerousness of the task and lack of support within their own organisations Other issues: EEP project applicants� comments Applicants commented that DHS had big expectations in relation to study design and implementation Comments about difficulty finding control site, or the ethical issues about the notion of control; qualitative methodology considered more appropriate by applicant A qualitative methodology was considered more appropriate than the CBA study design; Some applicants requested detailed information about measurement tools and statistics during submission phase that Cochrane could not supply at that time; Some health services needed encouragement to apply citing the onerousness of the task and lack of support within their own organisations

14. Concluding comments about Cochrane support: Role providing CBA resources, developing interventions and outcomes; Proactive and reactive - sending Resource and Evidence bulletins, waiting for requests for support; Not able to assist with detailed statistical, analytical advice; Assisted focusing project on what was manageable within EEP timeline. Concluding comments: primarily our role was to direct people to CBA resources and assist with developing interventions and outcomes in relation to consumer participation we have been proactive in sending bulletins, and reactive in waiting for requests of assistance we have been able to assist with some study design issues but not all we have been able to assist with interventions and outcomes, and with CBA study design resources, but not with specific statistics and other analytical advice � in the latter case we have referred projects on to experts in the field and other resources (e.g. EPOC and Shadish) Sorts of interventions considered for investigation in EEP projects e.g. staff training, patient information, new models of participation and patient care, Outcomes staff and consumer/carer e.g. actual consumer involvement, knowledge and skills Study designs being considered e.g. CBA, ITS, preference trial, qualitative elements (not within EEP component) Concluding comments: primarily our role was to direct people to CBA resources and assist with developing interventions and outcomes in relation to consumer participation we have been proactive in sending bulletins, and reactive in waiting for requests of assistance we have been able to assist with some study design issues but not all we have been able to assist with interventions and outcomes, and with CBA study design resources, but not with specific statistics and other analytical advice � in the latter case we have referred projects on to experts in the field and other resources (e.g. EPOC and Shadish) Sorts of interventions considered for investigation in EEP projects e.g. staff training, patient information, new models of participation and patient care, Outcomes staff and consumer/carer e.g. actual consumer involvement, knowledge and skills Study designs being considered e.g. CBA, ITS, preference trial, qualitative elements (not within EEP component)

15. Providing support from July 1 2007: Information searches tailored to EEP sites; Two visits to each site; Tailored advice on measurement tools; Ongoing monthly bulletins on evidence and resources; Seminar in February 2008 to promote evaluation of participation within quality and safety setting. Providing support from now on: Information searches tailored to EEP sites e.g. �how to write search strategies� Two site visits to each site to provide feedback Tailored advice on measurement tools for assessing key outcomes Ongoing monthly bulletins on evidence and resources relevant to consumer, carer and community participation and good communication A full day seminar in February 2008 to promote evaluation of participation within a quality and safety settingProviding support from now on: Information searches tailored to EEP sites e.g. �how to write search strategies� Two site visits to each site to provide feedback Tailored advice on measurement tools for assessing key outcomes Ongoing monthly bulletins on evidence and resources relevant to consumer, carer and community participation and good communication A full day seminar in February 2008 to promote evaluation of participation within a quality and safety setting

16. Reflection on Cochrane involvement: Challenges around providing timely support; Could provide some information searches, measurement tools advice to all EOI applicants during submission phase; Projects require measurement tools for ethics submission; Period between notification of successful and project implementation needs to be longer. Timeliness of support (issues arising from support): There were difficulties with the sort of support provided during submission, some applicants requested more detailed support: It may have been worthwhile to provide �information searches tailored to EEP sites (e.g. how to write search strategies)� to all EOI applicants during submission phase not just for successful EEP projects after 1 July For successful EEP projects there were difficulties with the timeline, specifically the short period of time between notification of successful and implementation: It may be more timely to provide �tailored advice on measurement tools for assessing key outcomes� earlier; or To have a longer gap between notification and implementation Projects require measurement tools for ethics submission; Period between notification of successful and project implementation too short. preliminaries of projects needed to be sorted out with our assistance (response document) before tools could be found People wanted highly specific and specialised statistics advice Also wanted specific advice about measurement tools projects requested assistance with specific measurement tools in May prior to when we could give that sort of assistance; the timeline for projects was too short between notification of successful and beginnings of project implementation submission to ethics committee (with validated measurement tools) needs to happen early in implementation phaseTimeliness of support (issues arising from support): There were difficulties with the sort of support provided during submission, some applicants requested more detailed support: It may have been worthwhile to provide �information searches tailored to EEP sites (e.g. how to write search strategies)� to all EOI applicants during submission phase not just for successful EEP projects after 1 July For successful EEP projects there were difficulties with the timeline, specifically the short period of time between notification of successful and implementation: It may be more timely to provide �tailored advice on measurement tools for assessing key outcomes� earlier; or To have a longer gap between notification and implementation Projects require measurement tools for ethics submission; Period between notification of successful and project implementation too short. preliminaries of projects needed to be sorted out with our assistance (response document) before tools could be found People wanted highly specific and specialised statistics advice Also wanted specific advice about measurement tools projects requested assistance with specific measurement tools in May prior to when we could give that sort of assistance; the timeline for projects was too short between notification of successful and beginnings of project implementation submission to ethics committee (with validated measurement tools) needs to happen early in implementation phase

17. Future directions: Intelligence and infrastructure for building the evidence base; Complex interventions; Learning through examining the parallels with research into improving provider practice. SHSH