Games to Improve Clinical Practice and Healthcare Administration

1. Gaming  for  Organizational,  Clinical   and  Patient  Risk  Management Comprehensive  Unit-­based  Safety  Program  (CUSP)   for  Enhanced  Recovery  Protocol  for  Surgery   “Simu-­Leader” (a  gaming  application)   YouTube  link

2. What  were  we  trying  to  do? Johns  Hopkins  Medicine  implemented  the  Comprehensive  Unit-­based  Safety  Program   (CUSP)  and  has  demonstrated  significant  outcomes.  Armstrong  Institute  conducted  a   brief  table  top  game  in  2012  to  facilitate  learning  and  reinforce  best  practices  amongst   hospital  leadership.  Based  on  those  results  a  grant  was  requested  to  build  an   automated  application  to  achieve  the  same  results  on  a  larger  scale. Two  developed  are: 1. Patient  Risk:  Application  geared  to  improve  the  safety  and  quality  of acute   patient  care ,  strongly  impacted  by  all  levels  of  organizational  leadership.  Game   provides  evidence-­based  methods  for  developing  the  knowledge  and  skills  necessary   for effective  safety  leadership 2. ERAS/CUSP:  Gamification  and  gaming  interfaces  measurably  improve  patient   outcomes  by  implementing  enhanced  recovery  after  surgery  (ERAS)  practices  in   hospitals  through  applying  the  Comprehensive  Unit-­based  Safety  Program  (CUSP).

3. Overall  Study  Aims § Specific  Aim  #1:  Evaluate  the  comparative  effectiveness  of  simulation  and  traditional  methods  of   management  instruction  on  multi-­level  learning  outcomes.   Ø H1a-­c:  Compared  to  traditional  instruction,  simulation  will  produce  significantly  higher  individual  level  a)   learner  reactions,  b)  learning,  and  c)  behavior  change.   § Specific  Aim  #2:  Compare  the  impact  of  ad  hoc  and  intact  management  team  configurations  on  multi-­ level  learning  outcomes  and  the  quality  of  strategy  generation  in  organizational  leadership  simulations.   Ø H2a-­c:  There  will  be  a  significant  difference  due  to  the  team  configuration  manipulation  for  individual  level   a)  learner  reactions,  b)  learning,  and  c)  behavior  change.   Ø H3a-­c:  There  will  be  a  significant  difference  due  to  the  team  configuration  manipulation  for  team  level   creativity  of  ideas  generated  in  terms  of  a)  novelty,  b)  usefulness,  and  c)  feasibility  of  ideas.   § Specific  Aim  #3:  Evaluate  the  impact  of  a  mindfulness  pre-­training  intervention  on  multi-­level  learning   outcomes  and  management  team  performance.  Pre-­training  and  pre-­performance  interventions  are   effective,  efficient  ways  to  enhance  learning  and  performance  outcomes.   Ø H4a-­c:  Compared  to  no  mindfulness  training,  mindfulness  training  will  increase  the  efficacy  of  the   organizational  leadership  simulation  by  producing  significantly  higher  individual  level  a)  learner  reactions,   b)  learning,  and  c)  behavior  change. Ø H5a-­c:  Compared  to  no  mindfulness  training,  mindfulness  training  will  produce  significantly  higher  team   level  creativity  of  ideas  generated  in  terms  of  a)  novelty,  b)  usefulness,  and  c)  feasibility  of  ideas.  

4. ERAS/CUSP  Overall  Aims Task  1: Develop  a  public  domain  version  of  a  CUSP  for  ERAS  educational  kit  for  national   distribution. Ø Johns  Hopkins  APL  to  leverage  software  engineering  expertise  as  well  as  existing   applications  to  address  the  requirements. APL  will  design,  develop,  test  and  deliver  the  software   application Task  5: Assess  the  Adoption  of  CUSP  for  ERAS  and  Evaluate  the  Effectiveness  of  the   Intervention  in  the  Participating  Hospitals. Ø Johns  Hopkins  APL  will  develop  software  and  provide  a  credible  and  systematic  data   collection  schema  allowing  for  data  collection.  The  software  application  will  provide  a  robust  back-­ end  database  structure  for  in-­stride  assessments,  metadata  visualization  and  distributed  access. Ø Subtask  5.1:  Design  and  develop  data  collection  protocols  and  analysis  plan.  Support  the   data  collection  protocol  provided  by  AI  collaborators.  Make  software  application  available  to   participating  hospitals  to  voluntarily  input  and  share  data.

5. Key  Metrics,  Measures  &  Performance Performance  in  the  application  was  measured  in  a  variety  of  ways: • Individual  progress  +  translation  to  practice • Team  effectiveness  within  scenario(s) • Between  team  comparisons  conditioned  on  mindfulness  training Armstrong  Institute  will  define  each  metric  for  JHU/APL  to  include Detailed  design  connecting  data  needs  to  GUI Team  v  Team Measures How$ How$ Expected$ Format$ Weigh:ng$ Teams Represented?$ Collected?$ Transparency* Reliability* Leadership* Strategic*Plans* Impact* 8*8*8*other*metrics* Cond  v  Cond Measures Candidate  metrics  for  response/resolution: • Rate  of  re-­admission • Patient  falls • Acquired  infections • Preventable  VTE

6. Software  Design  &  Computer  Science

7. Technical  Collaboration As  elements  of  the  application  are  developed  in  parallel  by  Johns   Hopkins  Medicine,  Armstrong  Institute  and  JHU/APL,  we  anticipated   evolving  requirements.  Therefore,  the  Agile  software  development   methodology  best  suited  the  development  team. “Agile  software  development is  a  group  of software  development  methods in   which  requirements  and  solutions  evolve  through  collaboration  between  self-­ organizing, cross-­functional  teams.  It  promotes  adaptive  planning,  evolutionary   development,  early  delivery,  continuous  improvement,  and  encourages  rapid  and   flexible  response  to  change.” The  web  application  is  updated  frequently   during  the  development  process  in  order   to  facilitate  frequent  feedback  from   collaboration  discussions.

8. Participant  Rhythm1 Player  Feedback

9. Software  Architecture Web  App Amazon  Cloud  Server Game  Manager Game  Session Windows  Server  2012 MS  SQL  Server   Database Client  Layer Game  Session … MS  IIS  Webserver Game  Session Production  Web  App Security  Services Game  Engine Text  Chat Performance  Measurement  Reporting Player  Interface Web  Page  Access PDFs Database Principal  Inv.  I/F Designed  Reports Spreadsheets

10. Game  Navigation  Concept Concept  discussions  with  Armstrong  Institute   produced  a  hub-­spoke  structure  which  migrated  to   a  more  functional  physical  representation  of  users’   expected  normal  work  environment. Shown  here  is  early  concept  art  used  to  scope  the   application  and  conduct  early  usability  tests.  

11. User  Interface  via  Web  Application  (prototype)

12. Web  Portal  on  Amazon  Web  Services

13. Software  Demonstration

14. Human  Factors  &  Usability

15. Human  Factors  Tasking ✔ ✔ In  Progress:   Completed Round  1,  N=9  ✔ Round  2,  N=9  ✔ Completed         N=4

16. Human  Factors  Tasking Concept  Vetting  -­ Preparation

17. Human  Factors  Tasking Alpha  Testing  &  Usability  Testing Proctored   Activity   Scenarios Background   Questionnaire Usability   Questionnaire Retrospective   Interview

18. Human  Factors  Tasking Alpha  Testing  &  Usability  Testing Proctored   Activity   Scenarios Background   Questionnaire Usability   Questionnaire Retrospective   Interview

19. Human  Factors  Tasking Alpha  Testing  &  Usability  Testing Proctored   Activity   Scenarios Background   Questionnaire Usability   Questionnaire Retrospective   Interview

20. Human  Factors  Tasking Alpha  Testing  &  Usability  Testing Proctored   Activity   Scenarios Background   Questionnaire Usability   Questionnaire Retrospective   Interview What  did  the  participants  find  important  during  the  session? What  problems  did  they  encounter?  (Flanagan,  1954).   Their  perception  of  the  session  and  their  self-­reflection  may  offer  insight  into   how  the  simulation  was  used  and  how  it  could  be  improved  (Rosson &   Carroll,  2002). Questions  included: • Are  the  instructions  clear? • Do  you  feel  you  know  what  to  do/where  to  navigate? • What  would  make  this  easier  for  you? • Can  you  recall  any  activities  being  particularly  successful?  Unsuccessful? • What  did  you  like  the  most?  Dislike  the  most?

21. Human  Factors  Tasking Alpha  Testing  &  Usability  Testing  – Results  (3/3) Post-­Session  Usability  Questionnaire • Round  1  – July  (purple),  n=9.   Round  2  – November  (blue),  n=9.   • Four  participants  completed  both  rounds.

22. Metrics  and  Verification

23. Scoring  Algorithm § Outcome  =  Pos.  tactics  +  Neg.  tactics Ø Pos.  Behaviors  =  (6RCpos  +  3TRpos+  BCpos)  *  ((TL  +1)  +  BS) Ø Neg.  Behaviors  =  (-­2.2RCneg  +  -­3TRneg+  -­1.6BCneg)  *  (TL  +1) § Term  definitions/MOPS/Tactic  Domains Ø RC  =  Responsibility,  role  clarity,  and  feedback  tactics Ø TR  =  Time  and  resources  tactics Ø BC  =  Building  capacity  tactics Ø TL  =  Transformational  leadership  tactics Ø BS  =  Boundary  Spanning  tactics § Term  values  will  be  based  on  proportion  of  (+)  and  (-­)  actions  selected   during  game  (TL  &  BS  will  be  a  count  of  actions) § Initial  weights  determined  through  adapted  ProMES process

24. How  much  does  each  indicator  contribute  to  effective   organizational  performance? Three  primary  types  of  contingency  shapes: 100 100 80 80 60 60 40 EFFECTIVENESS EFFECTIVENESS 40 20 20 0 0 -­20 -­20 -­40 -­40 -­60 -­60 -­80 -­80 -­100 -­100 80 90 100 110 120 130 80 85 90 95 100 TRAINING  MET PERCENT  PASSING  INSP. Critical  Mass Linear Diminishing  Returns Images:  (Pritchard,  2009)

25. Selecting  Different  Tactic  Sets Surgical  Site  Infections Patient  Readmissions 250 300 200 150 SSI RA 100 50 0 0 5 7 9 11 13 15 17 19 21 23 25 27 5 7 9 11 13 15 17 19 21 23 25 27 # of Tactics Selected # of Tactics Selected Length  of  Stay Patient  Satisfaction 350 300 250 200 LOS Sat 150 100 50 0 0 5 7 9 11 13 15 17 19 21 23 25 27 5 7 9 11 13 15 17 19 21 23 25 27 # of Tactics Selected # of Tactics Selected

26. Impact  of  “positive”  and  “negative”  tactics 300 300 250 200 200 150 100 100 50 0 0 -50 positive negative overall positive negative overall Surgical Infections Readmission 350 300 300 250 200 200 150 100 100 50 0 0 positive negative overall positive negative overall Length of Stay Patient Satisfaction

27. Risks  for  our  small  budget  game § On-­demand  help  is  not  available  during  game  play  increasing  the  level  of   software  reliability  and  testing § Technology  barriers  that  exist  at  the  player  location,  equipment  or  access § Depth  and/or  breadth  of  scenario  data  required  to  represent  a  realistic   decision  environment § Voice  communication  implementation  for  system  users § Adjudication  formulation  represents  ‘real’  hospital  outcomes § No  timing  mechanism  included  to  compel  player  action

28. Summary Major  Take-­Aways Ø Everyone  wants  gaming  and  it  is  ‘accepted’ Ø Professional  game  success  is  measured  differently Ø Represent  “real  life”  with  plausible  actions  and  effects Ø Technical  skills  and  experimentation  over  graphics Ø Experimental  testing  focuses  on  impact  over  entertainment The  game  and  initial  results  perform  well  against  the  study   hypotheses.  Participants  from  around  the  world  have   registered  and  data  collection  is  on-­going.  We  anticipate   significant  engagement  and  persistent  effects  for  individuals   and  organizations  participating.  

29. Scott  Simpkins;;  scott.simpkins@jhuapl.edu;;  240-­228-­3718 Principal  Staff,  Principal  Investigator Supervisor,  Health  Concepts  &  Systems  Analysis Health  Systems  Engineering  Group

30. Patient  Risk  Management  Summary The  safety  and  quality  of  acute  patient  care  is  strongly  impacted  by  all  levels  of   organizational  leadership,  yet  few  evidence-­based  methods  for  developing  the   knowledge  and  skills  necessary  for  effective  safety  leadership  currently  exist.   Simulation  clearly  improves  technical  and  non-­technical  competencies  of  healthcare   workers,  as  well  as  organizational  learning  and  continuous  improvement.  However,   simulation  has  not  yet  been  broadly  applied  to  the  development  of  leadership  for   patient  safety.   This  work  assembled  a  multi-­disciplinary  team  with  unique  and  diverse  expertise   (simulation,  training,  gaming,  engineering,  patient  safety  leadership,  business  and   management,  social  science  of  creativity,  human  factors  and  organizational   psychology),  and  capitalizes  on  existing  projects  and  strong  relationships  with  state,   national  and  international  healthcare  organizations  to  evaluate  a  game  for  patient   safety  leader  development.  

31. ERAS/CUSP  Summary To  measurably  improve  patient  outcomes  by  increasing  the  implementation  of   enhanced  recovery  after  surgery  (ERAS)  practices  in  hospitals,  through  the   use  of  an  adaptation  of  the  Comprehensive  Unit-­based  Safety  Program   (CUSP).  CUSP  for  ERAS  shall  be  developed  as  an  integrated  combination  of   clinical  and  cultural  interventions.   § Develop  a  consensus  description  of  the  essentials  of  ERAS,  develop   consensus  recommendations  for  implementation  of  ERAS  bundles  for   various  surgeries  and  determine  accurate  measures  of  effectiveness  of   CUSP  for  ERAS.   § Develop  an  adaptation  of  the  CUSP  protocol  and  materials  to  be  applied  to   ERAS.   § Recruit  hospitals  to  implement  ERAS  in  a  phased  approach,  beginning   with  100  hospitals  in  the  base  period,  and  expanding  to  include   approximately  200-­250  hospitals  during  each  option  period,  in  order  to   cumulatively  include  750  or  more  hospitals.  Participating  hospitals  should   be  from  all  States,  Puerto  Rico  and  the  District  of  Columbia.