A Survey of REB Systems in Canada (CAREB AGM Montreal 2014)

1. A Survey of REB Systems in Canada(CAREB AGM Montreal 2014) Michael Wilson (Research Ethics Officer) & Dr. Brian Detlor (Chair) McMaster Research Ethics Board (MREB)

2. The Access Database Year 2000 Year 2004

3. MREB Active – Master Paper Record

4. Born to Scan • Sharp 2700N copier / scanner • Close to my office • Archiving and emailing PDFs

5. Program or Be Programmed!

6. Coldfusion – Life is but a Dreamweaver

7. On your mark…choose a vendor…

8. UK – National Research Ethics Service

9. So Many REB Systems? • One software system for UK research ethics review – developed by Infonetica • I developed, designed & programmed my own REB system • RFPs for larger systems i.e, > $100,000 • IT projects have a high 40% - 70% failure rate • In-house system – or what happens if your developer leaves? • Our REB systems survey – what didn’t we know?

10. REB Systems in Canada Survey

11. Purpose of the Research Project • To obtain a descriptive snapshot of the human-participant REB systems landscape in Canada. • REB systems in Canada implemented? • Key characteristics of successful REB systems? • Satisfaction of stakeholders with REB systems? • What works well & what could be improved? • Context of systems & influences how fared? • Recommendations on implementing REB systems

12. Motivation • Our own “home-grown” REB system is at its maximum capacity • We need a more robust system • Handle increases number of protocols • Automate many tedious, monotonous tasks • Better reporting capabilities • Expedite the review of applications • Eliminate the “master” paper record • Better security / more reliable servers/ backups & redundancy • Better track revisions • Provide researchers with a better experience • Creating protocols; obtaining status updates on the review process; submitting “change requests”

13. Motivation • For a new and improved system we want to keep • Design our forms and workflow in a way that works for our Board • Control changes system and make them quickly • Low cost • DeLone & McLean Model of Information System Success

14. Methods • Web survey sent CAREB members via CAREB listserv before & after Calgary AGM May 2013 • Recruitment in person in Calgary • Survey instrument (Limesurvey on secure McMaster server) • Quantitative and qualitative questions on REB background, characteristics of REB system, user satisfaction, important REB system features • Probing questions about the respondent’s REB system • A very rich data set was collected – 64 complete responses • Many detailed qualitative comments

15. Respondent Profile

16. Respondent Profile

17. Current REB System

18. Current REB System

19. What are the key characteristics of a successful REB system? • Answers in rank order by frequency • Useful functionality (45 comments) • E.g., generates reports; generates reminders; generates statistics; creates meetings and agendas; facilitates robust and easy searching; tracks versions of documents; check status of a protocol • Efficient and meaningful processes and procedures (25 comments) • E.g., workflow meets the needs of all users; processes are streamlined; facilitates quick processing of protocols • Easy to use / User-friendly (24 comments) • Intuitive forms; smart forms; minimal form navigation

20. What are the key characteristics of a successful REB system? • Answers in rank order by frequency (cont’d) • Sufficient support (18 comments) • E.g., adequate number of admin staff; competent admin staff; knowledgeable REB members; adequate IT support in terms of staff and infrastructure • Captures all information needed (9 comments) • Automates tasks wherever possible (9 comments) • Flexible(8 comments) • Integrates with other institutional systems (7 comments) • Provides online access and electronic submissions (4 comments) • Follows good project management practice (2 comments)

21. How well do these characteristics compare with what’s in place? • Not as well as one would like... • Functionality offered by the current REB system • Permits researchers to submit an electronic application (protocol) 44% • Handles annual renewals and amendments electronically 36% • Handles serious adverse events reporting 25% • Has an online reviewer system 30% • Tracks applications through the review process 30% • Serves as the master record 45% • Permits data extraction for internal report purposes 41% • Creates online statistics 20% • Creates meetings, agendas, reports, reminders 28% • Permits electronic signatures or variations of such 17%

22. How satisfied are respondents with their current REB systems? • Most are satisfied

23. How satisfied are respondents with their current REB systems? • Twice as many negative comments given than positive ones • Negative comments (in rank order) concerned • Usefulness (23 comments) • Flexibility (10 comments) • Support / Infrastructure (8 comments) • Ease of Use (5 comments) • Positive comments (in rank order) concerned • Usefulness (9 comments) • Support / Infrastructure (9 comments) • Flexibility (1 comment)

24. Observations • “Satisfied” or “Very Satisfied” respondents utilized a wide variety of REB systems • No single type of REB system determined satisfaction • Satisfied respondents utilized the full gambit of REB systems • expensive solutions & in-expensive solutions, • vendor-based solutions & custom in-house solutions, • paper-based solutions, desktop database solutions, & full electronic solutions • Why? • What does this imply for practice?

25. Observations • Why does no one type of REB system determine satisfaction? • Satisfaction with an REB system seems to depend on the context in which they are situated • Critical factors include: • The number of protocols processed by the REB • The greater the # of protocols, the more likely a full robust REB system was needed • The expectations of researchers and staff (e.g., lower expectations lead to greater satisfaction) • The amount of support available (both REB staff and IT support) • The more support available to handle administrative work and IT concerns, the greater the satisfaction • The willingness of an institution to supply this support is critical

26. Observations • Certain factors can undermine satisfaction with implementing an expensive vendor-based REB solution • Factors that inhibit satisfaction include: • high cost • loss of control making timely changes to the system • loss of prior functionality, absence of needed functionality, inclusion of unwanted functionality • using processes and workflows that do not match the way the institution works • REBs should ensure that these factors are addressed prior to implementation • However, this is not always possible to do

27. Other thoughts • Though electronic REB systems offer many benefits, they are, at the end of day, still just administrative systems • They tend to offer more efficiency benefits (e.g., automated processes; online submissions; quicker tracking; easier reporting) than effectiveness benefits (e.g., better quality reviews) “Researchers, reviewers and ethics staff are spared clerical, paper-heavy administriva in favour of more time and energy to focus on content. However, online systems will not compensate for poorly advised researchers, reviewer shortages or poorly informed ethics staff.” “REBs and administrators focus a lot on improving efficiency, but I’d like to explore a system that improves effectiveness of REB processes.”

28. Other thoughts • There is a desire for standardized software and harmonization of software “There should be a coordinated effort to have all REBs using the same software.” “Canada needs to develop one system only for multi-university research projects so that researchers are not being asked to fill in one for each university they have as participants.” • Change management is critical when implementing a new REB system “There is an incredible amount of time involved from REB administrators to set up an efficient system (consulting, adapting system and policies and procedures, training etc.) that I think is often underestimated by senior administration. Managing change alone can be an enormous task and plans have to be in place to support implementation.”

29. Recommendations? • Get your REB systems / IT project in writing – have a Project Charter signed by key stakeholders at the highest level • Look in-house first – your organization may have the talent and resources • Canadian REB system vendors and developers know the Canadian landscape • A web-based REB system solution reduces data entry, paper usage, improves efficiency & maximizes effectiveness • Integration with other internal systems is nice – but first have a system that first works for your REB, not some other office

30. Questions? • Detailed Survey results online and in Handout here https://reo.mcmaster.ca/REBsurvey • Later: E-mail us • Journal publication forthcoming ...