Why Everyone Should Assess Program Implementation

1. Why Everyone Should Assess Program Implementation Matthew Swain Sarah Sunde Tisha McCoy-Ntiamoah Sara Finney Jerusha Gerstner Assessment Graduate Assistant Associate Director of Orientation Assessment Graduate Assistant Director of Orientation Associate Assessment Specialist

2. Goals • Define implementation fidelity assessment • Explain the importance of implementation fidelity assessment • Describe how to practically collect implementation fidelity data • Create an implementation fidelity assessment checklist • Integrate results from both outcomes and fidelity assessment

3. Transfer Orientation at JMU • History of Transfer Summer Springboard • Who are JMU Transfer students? • Phases of Orientation • Many programming aspects are implemented by staff outside of Orientation, so out of direct control

4. Transfer Summer Springboard • Academic Requirements Knowledge: • Upon completion of Transfer Summer Springboard, students will be able to correctly identify the academic requirements for major, degree, and graduation completion at JMU. • Resource Knowledge: • As a result of attending Transfer Summer Springboard, students will demonstrate an increase in knowledge of JMU resources by correctly matching university resources to student needs. • Social Acclimation and Community Building: • As a result of attending Transfer Summer Springboard, students will indicate a significant increase in their cohesion to the JMU community.

5. Transfer Summer Springboard • Programming involves: • Instruction in academic policies • Instruction about resources available to them on campus • Welcoming students to the JMU community • Assess outcomes using a pretest/posttest design, no control group

6. Transfer Summer Springboard 7:45 – 8:15 a.m. Check-in Warren Hall 2nd & 3rdfloor Confirm your e-ID username and have your JAC (JMU Access Card) photo taken. Refreshments are available on the Warren Hall patio. 8:25 – 9 a.m. University Welcome Grafton-Stovall Theatre Dr. Linwood H. Rose, President of James Madison University, will welcome you to the community. Dr. Roger Soenksen, a JMU faculty member, will offer a special message to prepare you for the JMU learning experience. 9:10 – 11 a.m. Peer discussion with your ANT & TOPA Taylor Hall Here’s your chance to get your questions answered, meet other transfer students, get the scoop from your ANT (Assisting New Transfers) and TOPA (Transfer Orientation Peer Adviser). You will also learn more about GenEd, course registration and transfer credits. Check out the green insert in your folder for your meeting location. 11:10 – 11:35 a.m. JMU Student Services Grafton-Stovall Theatre This informative video presentation will provide essential information about dining, health services, information technology, financial aid, on- and off-campus housing, banking on campus and the University Business Office. Follow along using the handout in your folder. 11:40 a.m. – 12:25 p.m. Student Resource and Housing Fair Warren Hall, 2nd Floor You won’t want to miss this! Full of information and resources, the fair will provide you with the details you need to navigate your way at JMU. Still need to find housing for the fall? Come talk with local property managers and find a place just right for you! 12:25 – 1:15 p.m. LunchGibbons Hall Entrance 4 & 5 1:30 – 5 p.m. Advising and Registration Check your folder label Students will meet with an advisor in departmental offices and register for courses online in one of the following computer labs: Ashby Hall (ground floor entrance), ISAT 336 and Showker Hall 208. Don’t leave for the day without stopping by Taylor Down Under! Come on down to TDU for: Complete your program evaluation ◊ Your JAC! ◊ Refreshments ◊ Chatting with ANTs & TOPAs ◊ Campus tours

7. Assessment Cycle 1 2 6 5 3 4

8. Implementation Fidelity Program Components A B C D

9. Implementation Fidelity Program Components “Black Box” A B C D

10. Assessment Cycle 1 2 6 • Implementation Fidelity Check 5 3 4

11. Fidelity Components • Program Differentiation • A determination of the specific features of a program. • Exposure • Measures the duration of each program component in actuality and how many students actually attend the various components. • Adherence • Assesses whether all features of the program were administered. • Quality • Assesses how well the program features were delivered. • Responsiveness • The extent to which the participants were engaged in the treatment.

12. Program Differentiation • What is covered or presented in Transfer Summer Springboard? What are the specific features associated with each program component? What do implementers do to enable students to ultimately meet the outcome? • Important phase, essential for other phases • Involves collaboration with stakeholders

13. Exposure • Exposure measures (a) the duration of each program component in actuality and (b) how many students actually attend the various components • Record actual duration of program component • Record attendance at events (if applicable)

14. Adherence • Was the full, prescribed program dose administered? • Were all features covered?

15. Quality • How well were the specific features presented? • May include presenter enthusiasm and method of delivery • Rating of the quality of presentation • We used a 3-point scale, but qualitative comments are appropriate • Only applicable if the feature was presented at all (adherence)

16. Responsiveness • Engagement of participants during the program • Example Measure: • Observe and rate in audit the responsiveness of students • Survey administered to students regarding responsiveness to material

17. Audit Checklist Program Differentiation Exposure Adherence Quality

18. Fidelity Audit • Day of auditing • Pretended to be students • Attended all programming • Evaluate if aspects of programming were completed to match the outcomes • Utilized audit checklist when observing the program • Requested that implementers evaluate their program • Incorporated into results • Provide information beyond the scores on the outcome measure • Give insight into reasons for the outcome scores

19. Using Fidelity Results • Objective-level results from 2010 and 2011

20. Closing the Assessment Loop • In 2010, our outcomes and fidelity results directed our attention to programming mapped to the Resource Knowledge objective • Specifically, the Student Services video’s script was updated • Where do students go to purchase a meal plan: 2010 Changed programming to increase clarity 2011

21. Closing the Assessment Loop • Additionally, we wanted to increase retention of important information presented about Academic Requirements • Using the Outcomes and Fidelity information, we informed our stakeholders and program implementers of areas of weakness • Minimum number of credits granted by any 4-year institution: 2010 Changed programming to increase clarity 2011

22. Benefits • Implementation fidelity assessment has a myriad of benefits: • Before the program, the program differentiation stage assists in structuring the program • A good way to begin the process • Helps the Student Affairs practitioner and stakeholder • A tool to establish a curriculum with program implementers • Focuses our efforts as a staff

23. Benefits • Implementation fidelity assessment has a myriad of benefits: • Before the program, the program differentiation stage assists in structuring the program • During the program, the audit informs practitioners of program drift and the student’s perspective

24. Benefits • Implementation fidelity assessment has a myriad of benefits: • Before the program, the program differentiation stage assists in structuring the program • During the program, the audit informs practitioners of program drift and the student’s perspective • After the program, all data is considered with the fidelity data:

25. Limitations • Fidelity is resource intensive • Preparation of checklist and structuring program • Human resources • Different implementers of same program • Inability to audit all programming

26. Discussion • Everyone should utilize implementation fidelity assessment • Using what you have learned today, what is one program that you are involved in that could use implementation fidelity?

27. Thank you! Questions?