1 / 35

Pilot STEM in OST Evaluation Preliminary Report

Pilot STEM in OST Evaluation Preliminary Report. March 2012 University of California , Irvine. Study Sample Selected. 17 programs in 9 Regions participating Pilot Evaluation Study Regions 1, 2, 3 4, 7, 8, 9, 10, 11 Total of 65 sites recruited 52 Elementary 9 Middle School 4 K-8

zola
Download Presentation

Pilot STEM in OST Evaluation Preliminary Report

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pilot STEM in OST EvaluationPreliminary Report March 2012 University of California , Irvine

  2. Study Sample Selected • 17 programs in 9 Regions participating Pilot Evaluation Study • Regions 1, 2, 3 4, 7, 8, 9, 10, 11 • Total of 65 sites recruited • 52 Elementary • 9 Middle School • 4 K-8 • Criteria for Selecting Study Sample: • Range of STEM curriculum approaches • Range of student age groups (grades 3-12) • Diverse ethnic and socio-economic backgrounds, representative of students in the State of California • Internet access

  3. Overview of Evaluation Activities: Selected Study Sites engage in the following activities: • Administer pre (fall 2011) and post (spring 2012) online student surveys • Administer online staff surveys at two time points: pre (fall 2011), and end-of-year (spring 2012) • Document Weekly Stem Activities—to be recorded daily by site implementers on STEM Activity Documentation Forms • Provide UC Irvine with copies of program schedule and lesson plans

  4. Pilot Study Launch Fall 2011 • Activities to date: • Selection of pilot study sample sites • Set up of staff pre-survey and student pre-survey online links • Communication with pilot study sites: • Email sent informing Program Liaison of selected sites and overview of evaluation activities to be carried out • Individual emails sent to STEM Implementers at each study site including: • Individual staff IDs assigned & instructions distributed for staff surveys • Site IDs assigned & instructions distributed for student surveys • STEM Activity Documentation Form (electronic and hard copies) distributed with instructions & envelopes for returning forms to UC Irvine • Packages sent to each study site or program liaison with: • Hard copies of STEM Activity Documentation Forms and instructions • Prepaid and addressed envelopes completed forms to UC Irvine

  5. Winter 2012 Evaluation Activities • Follow-up Communications & Reminders: • Program liaisons sent a list of study sites and corresponding staff IDs and site IDs Nov. 30/Dec. 1, 2011 • Reminders sent to Program Liaisons & STEM Implementers • Reminders to complete surveys and begin sending STEM Activity Documentation Forms • Accounting of data collected from their program sites • All site and staff codes and survey instructions resent with email • Ongoing contact with program Program Liaisons & STEM Implementers by phone and email

  6. Data Collected to Date—March 15 2012 • 90 staff pre-surveys have been completed • 35 sites have completed 1,277 student pre-surveys • Preliminary results of Staff and Student Pre-Surveys summarized in slides that follow • 103 STEM Activity Documentation forms received reporting on 310 Individual Stem Activities • Items reported by STEM implementers about each activity include: • Date and Duration of Activity • Name of activity • STEM content area addressed • Number of students and grade level • 4 point ratings of • Level of student Engagement • Level of challenge • Overall assessment of success of activity • Data are being entered in the data base for analysis and will be correlated with staff survey data and student survey site level data.

  7. Summary STAFF Pre-Survey Data Data collected December 2011–March 2012

  8. Summary of Staff Pre-Survey Results 90 staff surveys collected between November 2011 and March 2012 73% respondents female. 1/3 are between 18 and 25 years old. 72% are 35 or under.

  9. Staff Survey—Level of Education Achieved • Nearly half (48%) have attended college or have an AA degree • 42% have B.A. Degrees or higher

  10. Staff Reflect the Diversity of Students and Communities Served

  11. Staff Position in Afterschool Program • 40% staff have less than one year in current position • 33% have more than 5 years in position • 27% have 1-3 years in position

  12. 62% have some experience as a classroom aide or TA 23% 1-5 years 14% more than 5 years 37% reported having some classroom teacher experience 15% 1-5 years 10.5% more than 5 years 15% have some school administrative staff experience 8% student support staff experience Few staff report any administrative experience (6%) Staff Experience in other School Settings

  13. Staff Self-Identified Instructional Roles

  14. Variety of STEM Activities Implemented Prompt: In your current position, do you implement any STEM activities with students? If so, please specify which types of activities (select all that apply):

  15. Prior Experience & Time per Week Implementing STEM • 30 %have no prior experience implementing STEM at another program • Prompt: In your current position, how much time per week do you spend implementing STEM activities with students? • 22% spend NO time implementing STEM • Nearly half (49%) do at least 30 minutes to 2 hours of STEM per week

  16. Staff Meetings Around STEM • 21 % report never discussing STEM at staff meetings • 55% discuss STEM in staff meetings at least once a month to once a week Frequency of Staff Discussion of Program and STEM Issues n =90

  17. Compensation for Staff Meetings ( n = 89) • 66% staff report being compensated for all meetings • 10% for most meetings • 7% for some meetings • 15% receive no compensation for meetings • 2% staff report having no meetings

  18. Training and Support • A little over half (52%) of staff surveyed have had at least 1-4 sessions of Stem Related Training in the past academic year • 42 % have had NO Stem Training n = 90

  19. Staff Relations with Teachers • 32% never speak with classroom teachers about STEM concepts taught in classroom During the past academic year, how often have you discussed…

  20. Staff Relations with Parents • 76% of staff report never holding STEM-Related events for parents • 36% say they never speak with parents about STEM activities n = 89

  21. Staff Outcome Measures Beliefs & Sense of Competency • Two staff outcome measures: • 5 point rating scale • strongly disagree, disagree, neither agree or disagree, agree, strongly agree [some reversed coded] • Staff beliefs about STEM in the afterschool program. • Example Items • I think the students enjoy doing STEM Activities • In general, I think these students [in the afterschool program] are very capable of doing hands-on science activities • I don’t think there is enough time at the program for students to learn much about STEM • Staff sense of competency implementing STEM in the afterschool program • Example Items • I have a strong background in at least one area of STEM • I do not know enough about Science, Technology, Engineering and/or Mathematics to teach any of them well • I feel confident about teaching Science, Technology, Engineering and/or Mathematics in the afterschool program

  22. Summary Student Pre-Survey Data Data collected December 2011–March 2012

  23. Student Survey Sample • 1,277 in Full Sample • 51% Boys and 49% Girls Grade and Gender of Student Respondents

  24. Time to Complete Survey • 51%students overall report taking 15 minutes or less to complete survey • 55% Middle School & 28% Elementary students report taking 9 minutes or less

  25. Survey taking Experience • The majority of students find the surveys Easy to Read, Understand • and Answer • 83-85% state “mostly true” or “really true” Full sample: n = 1200

  26. Student OutcomesPre-Survey Results • 1,277 surveys completed by students grades 3-8 • Eight scales: • Work Habits • Misconduct • Social Competencies • Math Efficacy • Science Efficacy • Interest and Engagement in STEM • Future Outlook • Science Career Aspirations

  27. Work Habits • Work Habits—mean of 6 items • Assessed on a 4-point scale (1 = not at all true, 4 = really true) • Sample items include: • “I work well by myself” • “I finish my work on time.” Full Sample n = 1,277 SD = .67 Alpha = .81

  28. Misconduct • Misconduct—mean of 9 items • Assessed on a 4-point scale: (0 = never, 3 = more than once a week) • Lower score means LESS misconduct • Sample items include: • “I have gotten into a fight at school” • “I have taken something that belongs to someone else.”

  29. Social Competencies • Social Competencies—mean of 7 items • Assessed on a 4-point scale: (1 = not at all true, 4 = really true) • Sample items include: • “I work well with other kids” • “I can tell other kids what I think, even if they disagree with me.”

  30. Math Efficacy • Math Efficacy—mean of 4 items • Assessed on a 4-point scale: (1 = not at all true, 4 = really true) • Sample items include: • “I expect to do well in math” • “I am interested in math.”

  31. Science Efficacy • Science Efficacy—mean of 4 items • Assessed on a 4-point scale: (1 = not at all true, 4 = really true) • Sample items include: • “I expect to do well in science” • “I am interested in science.”

  32. Excited, Engaged and Interested Science Learner [PEAR-Harvard] • Excited, Engaged & Interested Science Learner—mean of 24 items • Assessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree) • Some items reverse coded • Sample items include: • “Science is something I get excited about.” • “I am curious to learn more about science, computers or technology.” • “Science is boring” • “I pay attention when people talk about recycling to protect our environment.” • “I enjoy visiting science museums or zoos.”

  33. Future Outlook • Future Outlook—mean of 7 items • Assessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree) • Sample items include: • “I will go to college.” • “I will have a job that I enjoy doing.”

  34. Science Career Aspirations • Science Career Aspirations—mean of 4 items • Assessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree) • Sample items include: • “I will have a career in Science, Technology, Engineering, or Mathematics.” • “I will graduate with a college degree in a major area needed for a career in science.”

  35. Next Steps • Initiate Post-Survey administration: April 15, 2012-June 15, 2012 • Ensure staff have received information, instructions and forms for completing Activity Documentation Forms • UC Irvine to Request sample copies of site schedule and weekly lesson plan • Final analysis of pre/post student and staff outcome measures and program implementation data • Refinement of measures for 2012-13 STEM in OST evaluation study

More Related