Atlanta, GA August 2009

1. DISCUSSION DOCUMENT AIM: VPOP Evaluation update Vaccine Management Business Improvement Project Atlanta, GA August 2009 This document is confidential and is intended solely for the use and information of the client to whom it is addressed.

2. VPOP Evaluation Results Grantee responses Overall perception of the system Communications and grantee calls Provider Advisory Committee Provider training Identity proofing and testing Provider responses Overall perception of the system Identity proofing and system access VTrckS vs. fax orders and training Accountability data and vaccine list Shopping cart and search functionality

3. Overall perception of the system Grantee Response Breakdown (9) Overall System Performance 70% agree with the statement 14% agree with the statement n=7 n=7

4. Communications and grantee call questions 100% agree with the statement 100% agree with the statement 100% agree with the statement 100% agree with the statement

5. Provider Advisory Committee 43% agree with the statement 71% agree with the statement 58% agree with the statement

6. Provider training 71% agree with the statement

7. Identity proofing and testing 72% agree with the statement 87% agree with the statement

8. VPOP Evaluation Results Grantee responses Overall perception of the system Communications and grantee calls Provider Advisory Committee Provider training Identity proofing and testing Provider responses Overall perception of the system Identity proofing and system access VTrckS vs. fax orders and training Accountability data and vaccine list Shopping cart and search functionality

9. Overall perception of the system Provider Response Breakdown Overall System Performance 57% agree with the statement 45% agree with the statement n=105 n=105

10. User experiences with VPOPIdentity proofing and system access “How many business days did it take to have your picture identification reviewed by an authorized agent (Notary Public or Immunization Program representative) for the Vaccine Tracking System (VTrckS)?” (1) “The process to obtain access to the Vaccine Tracking System (VTrckS) was easy to complete.” 65 respondents (60%) used a notary while 43 used an immunization program representative n=101 87% System Access Questions

11. User experiences with VPOP VTrckS vs. fax orders and training 0 % 1 % 0 % 55% agree with the statement Which parts of the Vaccine Tracking System (VTrckS) training were helpful?

12. User satisfaction with VPOP functionalityAccountability data and vaccine list 43% agree with the statement 44% agree with the statement 57% agree with the statement 54% agree with the statement

13. User satisfaction with VPOP functionalityShopping cart and search capability

14. Go-live and Identity Proofing System Changes Communication and Training Help Desk System Performance and User Satisfaction Transition Planning to VTrckS The questions asked of providers will change as we progress through the evaluation period Provider Evaluation Themes July August September October November June Complete Complete Complete • Experiences with id proofing • Have you placed an order? • Any login issues? • Was the system intuitive – first impressions? • What would you change about the system? • Change request process • Was feedback incorporated into the system? • Were communications received? • Were communications effective? • Was provider training completed? • Was training beneficial? • Did you need to call the help desk? • Was the help desk helpful? • What else should they do? • Do you find the system easy to use? • Is the system a satisfactory replacement? • Pending review 10-15 questions 5-7 recurring questions will be asked with each survey that will focus on system usability and performance

15. This Document • This document can be found on the CDC website at: • http://www.cdc.gov/vaccines/programs/vmbip/downloads/newsletters/2009/vpop-eval-acv3.ppt