Dr. Debbie Hahs-Vaughn Dr. Mary E. Little

1. Dr. Debbie Hahs-Vaughn Dr. Mary E. Little Welcome to week 7:eex 7320 Program planning and evaluation in special education-Program Evaluation of 325T federal grant project

3. Advance Organizer • Changes in Teacher Education-325T OSEP • Overview of 325 T and Evaluation teams • UCF project and procedures • Evaluation Questions, procedures, and discussions • BREAK • Research Designs and Methods-Slavin • Quantitative Designs-Chapters 2-6 • Qualitative and Action Research-Chapters 7-9 • Planning and Implementing Research-Chapters 10-12)

4. Think-Pair-Share • Consider the issues raised by Brownell et al and Ravitch (among others) re: teacher education and the need for change within teacher education and special education. • Name and defend several major goals for change. • Consider specific priorities for innovative projects in teacher preparation in special education within institutions of higher education. What priorities should be established by OSEP?

5. 325T RFP Initiatives-OSEP • Conclusion of 325 competitions focused on personnel preparation in high incidence disabilities. • New competition, 325T, began in 2006. • Focus of the priority: enhance and revise current teacher preparation programs to address current needs in teacher preparation through more collaborative programming in the following domains: • A-Licensure and State Certification (Credentials) • B-Organizational Structure • C-Curricular and Course Content • D-Student Support (Induction and Mentoring) • E-Program Evaluation

6. 325 T Program enhancement grants Annual competition of federal awards Clearly articulated components to address Monthly updates and collaborative discussions across sites and among grant recipients to share research, procedures, and activities. Collaboration across stakeholders (colleges, departments, school district and state personnel) Program revision and evaluation; continued funding based upon approval of program enhancements for duration of five years, if continued. the Office of Special Education Programs (OSEP)

7. Co-Principal Investigators: Mary Little, Ph.D., Suzanne Martin, Ph.D., Martha Stewart, Ph. D. Program Evaluator: Debbie Hahs-Vaughn Bridging Schools & Universities to Improve Math and Science Recruitment - Mentoring - Technology BRIDGE Preservice * Initial Teacher * Professional Development

8. Purpose of Project To enhance existing, approved initial certification programs to prepare highly-qualified teachers of students with high incidence disabilities (HID) within the Masters program. The major content focus will be math and science instruction for students with HID, including CLD students, in co-taught and resource settings within urban middle school that have not met AYP targets.

9. Improve recruitment to increase the number of highly qualified teachers in ESE, especially from diverse backgrounds; Enhance and implement an effective preparation program focusing on math and science content instruction at the middle school level within the ESE Masters and certification to assure HQT status and skills to successfully teach in urban settings; Improve mentoring and induction; Implement technology to support overall program effectively. 325T Project Goals

10. Our Task: • With our colleagues at UCF and using the resources provided in readings, websites, and in class, develop a program evaluation plan, instrumentation, and methodology to address one of the four goals form the Program Evaluation Plan of this OSEP 325-T project. • Specific resources: • Assignment overview and rubric; • Program evaluation overview; • Summary for UCF 325T project; • Team of doctoral students and professors.

11. Evaluation Team: Goal 1 • Improve recruitment to increase the number of highly qualified teachers in ESE, especially from diverse backgrounds; • Julie Joseph • Jonathon Rasmussen • Eric Wells • Ian Gesundheit

12. Evaluation Team: Goal 2 Enhance and implement an effective preparation program focusing on math and science content instruction at the middle school level within the ESE Masters and certification to assure HQT status and skills to successfully teach in urban settings; • Kathy Becht • Betsy Theis • Jenn Gallup • Erika Niecsinger

13. Evaluation Team: Goal 3 Improve mentoring and induction • Stacy Hardin • Jillian Gourwitz • Hannah Ehrli • Kelly Schaeffer

14. Evaluation Team: Goal 4 • Improve and enhance technology to support overall program effectively. • Barb Serianni • Aaron Koch • Onur Kocaoz • Lauren Delisio

15. Establish Bridge Program Team UCF ESE Students prepared to use technology in classroom Increase in teachers’ understanding of math/science concepts Recruit ESE Graduate Students & Gather Baseline Data UCF ESE Students prepared to teach ESE students in Math/Science Increased knowledge of math/science content Assemble Stakeholders & Solicit Input Dissemination& Presentations Model Exceptional Education Graduate Program (multiple pathways-content and delivery) Increase in the number of teachers from CLD backgrounds UCF Faculty, Bridges Program Staff & UCF Students Increase in the number of teachers from CLD backgrounds Middle/High schools-Districts HQ/NBCT Teachers, Students, Counselors Parents Curriculum modules aligned with state/federal standards for HQT Increase the knowledge and skills in math and science of teachers-HQT Review and Revise ESE Graduate Curriculum – Integrate New Technologies State and Community Partners – FLDOE, CEC, NMTE, NSTE, Industry, Volunteers Increase the retention of highly qualified teachers into schools Bridges Logic ModelResources Activities Outputs Outcomes Impacts USDOE Cooperative Funding & IRIS, FLDOE and UCF Resources Improved Mentoring and Induction Activities Establish data collection system to analyze/revise program Technology: Recruitment, Instruction, Coaching/ Mentoring (especially CLD) Increase in the technology for program communication and improvement Technology -Lab facilities & equipment FLDOE-ATEN Develop Virtual Activities – Mentoring, Instruction, Collaboration Effective use of technology to support overall program effectively

16. Improve recruitment to increase the number of highly qualified teachers in ESE, especially from diverse backgrounds; Enhance and implement an effective preparation program focusing on math and science content instruction at the middle school level within the ESE Masters and certification to assure HQT status and skills to successfully teach in urban settings; Improve mentoring and induction; Implement technology to support overall program effectively. 325T Project Goals

17. EEX 7320 Presented by: Dr. Debbie L. Hahs-Vaughn Procedures

18. Sections 1 & 2 • Select ‘evaluation goal’ and ‘evaluation subgoal’

19. Section 3: Research Question • There are multiple types of research questions and resulting research designs that can be used to answer the subgoals

20. Section 4:Instrumentation • Most applicable to: • Survey research • Other research designs that employ a test or assessments • Elements listed in the procedures template may not be applicable to and/or will need to be adapted for other types of instrumentation, such as: • Content analysis (instrumentation should describe the rubric/coding form used to gather data) • Interviews (instrumentation should describe interview protocol) • Factual data collection (may have a “data source” section other than instrumentation section)

21. Section 5:Description of Sample/Population • ‘Who’ will participate (i.e., unit of analysis, could be people, such as faculty or students, or documents, such as syllabi) • How they will be selected (census if population or sampling method if sample) • How many anticipated (i.e., anticipated sample size)

22. Section 6:Data Collection Procedures • Who (any process for selecting the sample that was not transparently and comprehensively detailed in section 5 should be explained here. In addition to other “who’s” applicable to the data collection procedures) • What (what will be done to them or gathered to get the measured data) • Where (where will data be gathered e.g., online survey, in-class, interviews conducted at…and other where’s that are applicable to the data collection process) • When (anticipated timeline) • How (how will activities be performed) • KEY IS TRANSPARENCY AND COMPREHENSIVENESS

23. Section 7:Data Analysis • Data analysis should correspond to the research question • Quantitative analysis • Descriptive • Inferential • Qualitative analysis (see next slides)

24. Leech, N. L. & Onwuegbuzie, A. J.  (2008).  Qualita-tive data analysis: A compendium of techniques and a framework for selection for school psychology research and beyond.  School Psychology Quarterly, 23, (4), 587-604.

26. Reliability and Validity in Qualitative Data Anfara, V. A., Brown, K. M, & Mangione, T. L. (2002). Qualitative analysis on stage: Making the research process more public. Educational Researcher, 28-38.

27. Kathy Becht, Lauren Delisio, Aaron Koch, Onur Kocaoz, and Erika Nicsinger Slavin Chapters 10, 11, 12

28. Chapter 10: Measurement • Reliability • Reliability is the degree to which a measure is consistent in producing the same readings when measuring the same things. • Ex.: a ruler • Low Measures of Reliability • Importance of Reliability • Validity • Validity refers to the degree to which a measure actually measures the concept it is supposed to measure. • A measure can be reliable, but not valid. • Types of Validity

29. Chapter 10: Measurement • Types of measures • Achievement and Aptitude Tests • Descriptions • Ex.: IQ Tests, IOWA Tests, Stanford Achievement Test • Standardized Tests • Description • Ex.: FCAT • Criterion-Referenced Tests • Description • Ex.: driving test, classroom assessment • Questionnaire Scales • Description • Ex.: CBAM, Likert-scaled questions • Advantages and Disadvantages

30. Chapter 10: Measurement • Behavioral Observation • Use • Construction • Caution against bias • Determining Adequate Sample Size

31. Chapter 10: Measurement • Connections to 325T Grant • Valid and reliable measures are needed to evaluate the efficacy of outcomes. • How will you know if the program has met the project goals? • Additional Resources • Web Resources: • Web Center for Social Research Methods, Knowledge Methods Resource Base: http://www.socialresearchmethods.net • The Mental Measurements Yearbook: http://www.ebscohost.com/academic/mental-measurements-yearbook • Educational Testing Service: http://www.ets.org/tests_products • Print Resources: • Drost, E.A. (2011). Validity and reliability in social science research. Education Research and Perspectives, 38(1), 105-123. • Gall, M. D., Gall., J. P., & Borg, W. R. (2006). Educational research: An introduction (8th ed.). Boston: MA: Allyn & Bacon.

32. Chapter 11: Threats to Internal/External Validity • Internal Validity • Internal validity is the degree to which the result of a study can be attributed to the treatments or other independent variables identified in the hypothesis. • External Validity • External validity is the degree to which the findings of a particular study have meaning for other settings and samples.

33. Chapter 11: Threats to Internal/External Validity • Threats to Internal Validity • Threats: history, maturation, testing effects, instrumentation effects, ceiling effect, floor effect, selection bias, statistical regression, attrition, confounding variables. • Examples of Internal Threats • Threats to External Validity • Threats: lack of internal validity, nonrepresentativeness, artificiality, reactivity. • Examples of External Threats

34. Chapter 11: Threats to Internal/External Validity • Connections to 325T Grant • Internal validity is the degree to which we eliminate confounding variables inside the study which can cloud the results of our research and minimize the efficacy of our grant. • External validity is the generalizability to larger audiences or settings which is needed to justify our grant outcomes. • Additional Resources • Web Resources: • Meissner, H. & Vinson, C. (2011). External Validity: Why It Matters [PowerPoint slides]. Retrieved from http://conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011/presentations /Aug5/Meissner&Vinson_TIDIRH%20External%20Validity%20Print%20Version.pdf. • Print Resources: • Gall, M. D., Gall, J. P., & Borg, W. R. (2006). Educational Research: An Introduction (8th ed.). Pearson. • Steckler, A., & McLeroy, K. R. (2008). The Importance of External Validity. American Journal of Public Health, 98(1), 9–10. doi:10.2105/AJPH.2007.126847

35. Chapter 12: Planning and Implementing the Study • Planning the Study • Selecting the Problem • Literature Review Components Coverage Synthesis Methodology Significance of Research Problem Writing Style

36. Chapter 12: Planning and Implementing the Study • Writing a Proposal • statement of the problem • hypothesis • brief literature review • detailed procedures • schedule • roughing out and discussing proposal • evaluation of the proposal • IRB

37. Chapter 12: Planning and Implementing the Study • Planning Study Procedures • Manuals • Testing instructions • Ethical research

38. Chapter 12: Planning and Implementing the Study • Gaining Access to Schools and Implementing the Study • FERPA • Integrity • Maintain group size & quality • Monitor • Report to participants

39. Chapter 12: Planning and Implementing the Study • Connections to 325T Grant: • Literature Review Format http://ncipp.education.ufl.edu/files_31/LlitReviewCourseMatching.pdf • Additional Resources: • Web Resources: • What Works Reporting Guide for Study Authors. http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_gsa_v1.pdf • Print Resources: • Slavin, R. E. (2007). Educational research in the age of accountability.(p. 225). Boston, MA: Pearson. • Boote, D., & Beille, P. (2005). Scholars before Researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher: (34) 6 pp. 3-15.

40. Slavin Chapter 10-12 • What questions do you have regarding: • Measurement • Internal Validity • External Validity • Planning the Study • Implementing the Study • Connections to 325T Grant • Additional Resources

41. Where are we with Teacher Quality? http://education.ufl.edu/grants/ncipp

42. Our Course Questions: • What questions can be addressed by qualitative methodologies? (Slavin) • How do we use data to implement change? (ALL) • What types of data should be collected and used for different purposes? (Slavin…then, used and discussed) • What is my role as the researcher? (Personal reflection) • How do we collect and use data to make change? (Collins; Guskey; some course readings/reflections) • What different models of program evaluation are there to use? (Research completed…now used/discussed) • How do we begin this process? (325T and Ind. Projects) • What data sources should be used? (Slavin; Guskey-applied)

43. Next Steps: • Review all materials, websites provided • Review chapters/resources-Slavin • Complete ONE response-Wiki-Elmore • Begin Program Evaluation of your section of 325T authentic learning assignment according to directions and rubric • Prepare to meet with your team next week during class time after 1 hour presentation