1 / 46

Piloting Your Research to Success: The Why, What and How of Pilot Study

Piloting Your Research to Success: The Why, What and How of Pilot Study. Nancy A. Baker, ScD, OTR/L University of Pittsburgh, Pittsburgh PA Susan L. Murphy, ScD, OTR University of Michigan and VA Ann Arbor Health Care System, MI. Conflict of Interest / Disclosure Statements.

lars
Download Presentation

Piloting Your Research to Success: The Why, What and How of Pilot Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Piloting Your Research to Success: The Why, What and How of Pilot Study Nancy A. Baker, ScD, OTR/L University of Pittsburgh, Pittsburgh PA Susan L. Murphy, ScD, OTR University of Michigan and VA Ann Arbor Health Care System, MI

  2. Conflict of Interest / Disclosure Statements Nancy A. Baker has nothing to disclose. Susan L. Murphyhas nothing to disclose.

  3. Getting Funded: Why Pilot Data are Important for Grant Applications Nancy A. Baker ScD, MPH, OTR/L

  4. Objectives: • Define pilot studies • Examine the relationship between pilot studies and the main research project

  5. Pilot Studies • Preparatory studies designed to testthe science, process, management, and resources of subsequent larger studies. • Pilot studies tend to test piecesof a project that will inform the conduct of the larger study • This is in contrast to ‘pilot trials’ • Pilot studies are vital for developing larger studies Charlesworthet al., BMC Medical Research Methodology 2013, 13: 78; Moore et al., CTS. 2011;4:332-337; Thabane et al. BMC Medical Research Methodology. 2010;10:1

  6. Phases of Intervention Trials Campbell, M. et al. BMJ 2000;321:694-696

  7. Preclinical - Theory • Systematic Reviews (All relevant disciplines) • Literature • Qualitative studies Campbell, M. et al. BMJ 2000;321:694-696

  8. Phase I - Modeling • Mechanistic studies • Descriptive studies • Qualitative studies • Identify outcome measures Campbell, M. et al. BMJ 2000;321:694-696

  9. Phase II - Exploratory • Refine intervention/ control • Feasibility of protocols • Train personnel • Test outcome measures • Preliminary effect • Qualitative studies Campbell, M. et al. BMJ 2000;321:694-696

  10. Phase III – Definitive Trial • Main trial • Multi-site • Sample size Campbell, M. et al. BMJ 2000;321:694-696

  11. Phases of Intervention Trials Pilot Studies Campbell, M. et al. BMJ 2000;321:694-696

  12. Pilot Studies Aims, research questions and design should address issues that affect future trials Moore et al., CTS. 2011;4:332-337; Thabaneet al. BMC Medical Research Methodology. 2010;10:1

  13. Process Data • Data on the feasibility of processes Tickle-Degnen. AJOT. 2013;67:171-176

  14. Resources Data • Data on the availability and sufficiency of resources Tickle-Degnen. AJOT. 2013;67:171-176

  15. Management Data • Data on Human and Data optimization Tickle-Degnen. AJOT. 2013;67:171-176

  16. Scientific Data • Data on intervention/control safety, dose, effect, and variance of effect Tickle-Degnen. AJOT. 2013;67:171-176

  17. Pilot Studies • Provide specific information for designing future studies • Technical aspects • Develop preliminary data • Foster investigator development • Ensures that main trial appropriate and feasible

  18. Getting Started with Pilot Studies: Appropriate Designs, Interpretation and Reporting Strategies Susan L. Murphy ScD, OTR/L

  19. Objectives • To discuss research design considerations for pilot studies • To examine interpretation of pilot studies • To discuss reporting of results of pilot studies • To discuss funding opportunities

  20. Pilot Studies are Important • Research studies that are adequately powered to test hypotheses occur because all or some elements of them were tested previously • Pilot studies allow researchers to know facts that will be important in conduct of larger trials before research stakeholders commit resources (Tickle-Degnen, 2013)

  21. Design Considerations • Scope needs to be carefully considered • The optimal pieces to test should be selected based on an understanding of the theory, research evidence, and other sources of pilot data that support the larger study • In other words, ‘where is the science at?’

  22. Art of Designing Pilot Studies • Rooted in understanding of where science is at for your research question • Scope suited to available resources • Prioritization of the pilot study aims • Set criteria for success when assessing feasibility

  23. Examples: Phase I -- Modeling • Define intervention components • Which are most relevant to the outcome?

  24. Example of Phase I Pilot Studies • LTG: To design an knowledge translation (KT) intervention for rehabilitation therapists in acute care to use evidence in practice • STGs: Determine what to include in intervention, optimal intervention delivery • Possible research methods? • Focus groups • Semi-structured interviews • surveys with health professionals/supervisors

  25. Focus Groups from Health Care Professionals

  26. Interpretation of Findings • Researcher designed an intervention including: • Tools for electronic access of research • Assessing quality of research • Identification of potentially applicable assessments for use in acute care • Researcher worked with clinical setting • to support initiative for EBP training • to allow time and space for intervention

  27. Phase II: Exploratory studies/trials • Define trial and intervention design • Prioritize elements to test

  28. Some pieces that can be tested in pilot studies: • Is your project feasible/acceptable to participants? • subject recruitment, procedures/intervention not overly burdensome, are people compliant)? • Are the risks understood and planned for (adverse events)? • Are you (and team) able to carry out such a project? • ability to coordinate, collect and analyze the data • Is there support for your intervention or its components? • Do you have procedures to ensure treatment fidelity? • Are the study instruments you have chosen to use appropriate?

  29. Establishing criteria for success

  30. Prioritizing pieces affects design • 1 group versus 2 group design • Example • Goal: test out intervention protocol and fidelity of therapists -- 1 group design • Goal: examine burden and drop-out rates with and without tx -- 2 group design

  31. Example: KT Intervention Feasibility Study • KT intervention components are established (Phase I study) • Goal of this study to examine feasibility of delivering the intervention and treatment fidelity • Research design? • One group pretest/posttest

  32. KT Intervention Feasibility Study • Researcher plans to enroll 20 participants from OT acute care unit • Group intervention that will occur biweekly for three months at lunch hour • The clinical setting has agreed to support the intervention including extended time for lunch

  33. Measuring Chosen Feasibility Aims Treatment Fidelity Article--Bellg et al. Health Psychology, 2004, 23: 5: 443-451.

  34. Power and Pilot Studies • Power analysis for a pilot study? • Important that study has enough sample to allow test of feasibility • Some review groups look for this power analysis, disagreement in literature about appropriateness of powering • Should estimating power be a study aim? • Caution is recommended • pilot study effect size ≠ population effect size Kraemer HC, et al. Arch Gen Psychiatry 2006; 63: 484-489.

  35. Interpreting and Reporting results • Important to establish a-priori criteria for success • Process Examples • Recruitment--At least 70% of all eligible patients can be recruited (POET pilot study Chou et al., 2009) • Follow-up—Complete follow-up in at least 95% of all recruited subjects • Feasibility of Intervention Example • Low perceived levels of fatigue, anxiety, and perceived difficulty during task training of people with PD (Canning et al., 2008)

  36. Pilot Study Characteristics • Exploratory • Test of feasibility of processes • Test of issues with resources or management • Safety/dose considerations • Data generating • Not for hypothesis-testing • May or may not include an attention or control group

  37. From a Talk by Kelli Allen PhD “How to get funding and resources to conduct pilot studies” Funding opportunities

  38. Funding Opportunities: Grants for “Stand-Alone” Pilot Studies Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • NIH & AHRQ • R03s (some just for new investigators, some do not allow pilot trials) • R21s (require higher level of innovation) • Minority and other Supplements • Department of Veterans Affairs • Foundations • Rheumatology Research Foundation (targeted disease areas) • American Occupational Therapy Foundation • National Psoriasis Foundation • Foundation for Physical Therapy • American Federation on Aging & Arthritis National Research Foundation • Scleroderma Foundation • Internal Institutional Awards

  39. Grants for “Stand-Alone” Pilot Studies: Pros and Cons Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Pros: • Resources for novel data collection • Experience with grant writing • Establishes funding track record • Cons • Time delay with review process (tends to be shorter with internal awards) • Funding rates • Limited budget • Some provide minimal support for investigator • Some funding periods brief

  40. Funding Opportunities: Career Development Awards Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • NIH & AHRQ – Various K-Awards • Institutional Career Development Awards (CTSA, T32) • Department of Veterans Affairs • Rheumatology Research Foundation • Scientist Development Award • Investigator Award • Robert Wood Johnson Foundation • Other Foundations and Professional Societies

  41. Career Development Awards: Pros and Cons Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Pros: • Protected research time for several years • Allows for well developed pilot data • Resources for novel data collection • Experience with grant writing • Establishes funding track record • Formal mentorship and training • Cons • Time involved in submission and review process (shorter for internal) • Funding rates

  42. Other Resources for Pilot Studies: Publicly Available Data Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Are there publicly available data sets that support aspects of your proposed research question(s)? • Examples: • Osteoarthritis Initiative • National Health and Nutrition Examination Surveys • National Health Interview Survey • Health and Retirement Study • CDC Behavioral Risk Factor Surveillance System • Mining Electronic Health Records • Many others!

  43. Publicly Available Data: Pros and Cons Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Pros: • Free (usually) & often immediately available • Often large samples with strong generalizability • Can help establish publication track record • Cons • May not exactly answer your question • No resources for analysis or protected time • Does not provide grant writing practice or establish funding track record

  44. Other Resources for Pilot Studies: Data from Mentors / Collaborators Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Secondary analyses of existing data • Adding variables to ongoing projects • Collecting qualitative data related to new project • Networking: • Reach beyond your immediate study team and institution • Who is doing work in your area(s) of interest, and how might you collaborate?

  45. Summary & Considerations:Which Road to Take? Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Primary Consideration: What kind of pilot data will result in a competitive grant application? • Consult with mentors, collaborators, program officer • What resources do you have? • Internal pilot / career awards? • Data from ongoing studies? • Statistical support? • What resources do you need? • Salary support? • Mentorship? • Training?

  46. References Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ 2000, 321:694-696. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1118564/pdf/694.pdf Charlesworth G, Burnell K, Hoe J, Orrell M, Russell I. Acceptance checklist for clinical effectiveness pilot trials: a systematic approach. BMC Medical Research Methodology 2013, 13: 78. http://biomedcentral.com/1471-2288/13/78 KianifardF, Islam MZ. A guide to the design and analysis of small clinical studies. Pharmaceutical Statistics 2011, 10:363-368. doi:10.1002/pst.477. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. Journal of Evaluation in Clinical Practice, 10, 2, 307-312. Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res, 2011, 45: 626-629. Moore C, Carter RE, Nietert PJ, Stewert PW. Recommendations for planning pilot studies in clinical and translational research. Clinical and Translational Science 2011; 4:332-337. doi: 10.1111/j.1752-8062.2011.00347.x Shanyinde M, Pickering RM, Weatherall M. Questions asked and answered in pilot and feasibility randomized controlled trials. BMC Medical Research Methodology 2011, 11: 117. http://biomedicalcentral.com/1471-2288/11/117 ThabaneL, Ma J, Chu R, et al. A tutorial on pilot studies: the what, why, and how. BMC Medical Research Methodology 2010, 10:1 http://biomedicalcentral.com/1471-2288/10/1 Tickle-Degnen L. Nuts and bolts of conducting feasibility studies. AJOT 2013, 67; 171-176. doi 10.5014/ajot.2013.006270.

More Related