NOVA: Program Evaluation

1. NOVA: Program Evaluation Creating change in higher education to enhance science, mathematics and technology literacy of pre-service teachers. NASA Headquarters, Washington D.C. December 6, 2001

2. NOVA • Overview: NOVA Program Evaluation Goal: Stimulate and conduct research and evaluation on the effectiveness of NOVA's pre-service education model.

3. NOVA/NASA Engineering Education Science Mathematics

4. The NOVAModel

5. Phase I: Workshops Phase I: Faculty and course development awards Phase II: Research and Dissemination Awards Phase III: On-line technology enhancement development awards NOVA/NASA network site visitation National NOVA/NASA Leadership Development Conference NOVA Network collaborations NOVA Evaluation, research, and dissemination NOVA Local reform effects and product development dissemination Implementation of the Model

6. NASA Strategic Enterprises • Interdisciplinary Approaches • Inquiry-based Learning • Using Technology to Facilitate • Learning • Innovative Instructional • Strategies • Overcoming Barriers • Assessment Strategies that • Facilitate Learning • Action Research for College • Faculty • Teaching Science for All • Americans • Writing Proposals for NOVA • New Curriculum Goals • Connections to NSES, NCTM, • ISTE Standards NOVA Phase I: Workshops Over 200 institutions have attended 16 NOVA workshops for Phase I. They received professional development in the following areas:

7. NOVA Phase I: Faculty and Course Development Awards NOVA has funded 59% of the Proposals submitted from 1996 to 2001. The NOVA Program has provided over 1.6 million dollars to institutions to create change in higher education.

8. Phase I: Institution Awards 79 NOVA Institutions found in every NASA Research Center Region

9. Institution Course Titles Brief Description Austin Peay State University, TN Introduction to Chemistry and Physics Thematic integration of chemistry and physics using energy - www.apsu.edu/~nova California Polytechnic University at Pamona, CA The Chemical Sciences Chemistry through exploration of the earth resources and systems – www.ceemast.csupomona.edu/nova/overview.html Michigan State University, MI Applications of Biomedical Science Integrated course on the functional biology of animals and the history of the inquiry http://www.eng.ua.edu/~nova University of Alabama, AL Introduction to Artificial Intelligence of Systems Systems in science explored through processes used in artificial intelligence http://galabl.mh.ua.edu/classes/esm130.shtml University of Dayton, OH The Physical Universe: The Dynamic Earth; Organisms, Evolution & Environment 3 courses thematically integrated using the environment – universe.uiwtx.edu/~mccormic/nova/sylabus.html University of the Incarnate Word, TX Diversity of Life Evolution explored through the study of biodiversity www.udayton.edu/~physics/nova/xl.html

10. Dissemination: California State Polytechnic University, 1999 California State University, Northridge, 2001 Columbus State University, 2000 George Fox University, 2001 Louisiana Technical University, 2001 Susquehanna University, 1999 Western Kentucky State University, 2000 Research and Development Fort Hays State University, 1999 Michigan State University, 1999 Indiana University, 2001 University of the Virgin Islands, 2001 Phase II: Research and Dissemination Awards

11. Earth Science- NASA Goddard Research Center, Md., 1999 -10 institutions Mission Research – NASA Marshall Space Flight Center, Al., 2000 - 5 institutions Space Science – NASA Jet Propulsion Laboratory, Ca., 2001 - 5 institutions Biological Science – NASA Space Center, Tx., 2001 - 5 institutions Phase III: On-line Technology Enhancement Awards

12. NOVA Network Site Visitations

13. November 1997, Washington, DC November 1998, Washington, DC November 1999, NASA Kennedy Space Center January, 2001, Arlington, Va March, 2002 – NASA Goddard Research Center February, 2003 – NASA Johnson Space Center NOVA Leadership Development Conference

14. NOVA Evaluation Goal: Stimulate and conduct research and evaluation on the effectiveness of NOVA pre-service education model. Leads to the Question: Does the NOVA professional development model create change in higher education to enhance science, mathematics, and technology literacy of pre-service teachers?

15. Contains Sub-questions • What are the characteristics of NOVA participants, their institutions, courses and students? • What are the participant reactions to the NOVA professional development model? • Did the participants acquire and apply the intended knowledge for reform action to take place?

16. Sub-questions Continued • Did the organizational climate change as a result of the professional development model facilitating reform action? • What aspects of professional development affect student learning outcomes?

17. Research data collected: Demographics Status surveys Efficacy Site visit suite Barriers EDCATS *Final Reports *Proposals *Interviews *Student Surveys Dissemination Activities: NOVA Conferences Reports Journal Articles *Products and Materials (Phase I & III) Phase III Online Curricula NOVA Network Conference and Journal Articles NOVA Network Dissemination (Phase II) NOVA Evaluation, Research, and Dissemination

19. 2002 Report Prospectus for Evaluation of the NOVA Preservice Education Program – developed 12/2002 1.Identify: Characteristics of NOVA institutions, courses and students matriculate in NOVA courses. • Evaluation Questions 2.Describe: The extent the NOVA Model is being disseminated and implemented by other faculty at NOVA institutions or at other institutions. 3.Congruency: How congruent is the NOVA program with recommendations found in major reports on the preparation of teachers?

20. 2002 Report Prospectus • Evaluation Questions 4 Evaluation: Is the NOVA Program effective at accomplishing its stated goals and objectives? • Assess: How has the implementation of the NOVA Model impacted the collaborative work of science and education faculties? • b. How has the Model impacted the content and pedagogical preparation of prospective teachers?

21. 2002 Report Prospectus • Evaluation Questions 5 Evaluation: Is the NOVA Program effective at accomplishing its stated goals and objectives? a. What are the courses being offered? b. To what extent does the majority of course content integrate data and information that are unique to NASA, its fundamental questions, and its missions? c. To what extent do the courses developed through NOVA support implement the NOVA Model?

22. Data Collection Five Critical Levels of Evaluation Background Characteristics Reactions Acquire and Apply Intended Knowledge Organizational Climate Student Learning Outcomes Demographics  Status Surveys     Efficacy    Site Visit:      Barriers     EDCATS   Final Report    Proposals  Interviews (Telephone)          TEACHER FOLLOWUP Preservice Inservice K-12 Students    STUDENT SURVEYS Nature of Science Nature of Mathematics Science attitude Other knowledge & skills

23. Data Collection Purpose Change Process Climate Professional Development Standards Driven Reform Student Outcomes Faculty Demographics    Status Surveys      Efficacy     Site Visit:       Barriers     EDCATS    Final Report      Proposals      Interviews (Telephone)        TEACHER FOLLOWUP Preservice Inservice, K-12 Students     STUDENT SURVEYS Nature of Science Nature of Mathematics Science attitude Other knowledge & skills Beliefs

24. NOVA Faculty Fellow 2001 – 2003 Schedule • National orientation session for NOVA Faculty Fellows • Site Visit protocol and training session – KSU • Site Visits by NOVA Faculty Fellows • Analysis and summary report of NOVA data • Conduct a national research project on a specific aspect of the impact of NOVA • Summer research synthesis at the University of Alabama. • Online web pages illustrating national impact of NOVA • Dissemination of NOVA research and evaluation results

25. NOVA Faculty Fellow Schedule 2. Site Visit Evaluation • Protocol • Site visit activities • Individual summary reports • Inter-rater reliability determination • Final summary report

26. 3a. Complete NOVA Network site evaluation visits for NOVA Phase I institutions during fall 2001 through spring 2003. 3b. Analyze and write summary reports using protocol format for site visits made. Institutions have been selected for each Faculty Fellow to reduce the time and resources needed to complete the site evaluation schedule. NOVA Faculty Fellow Schedule

27. 4. Analysis and summary report of NOVA data set collected on institutions, faculty, materials, local programs, and/or students. Research data collected: *Final Reports *Proposals *Interviews *Products and Materials *Student Surveys Analysis of relationships between variables in NOVA data collection matrix NOVA Faculty Fellow Schedule

28. 4. Continued. NOVA Data Collection Variables: • ID • University • State • NOVA Workshop Attended/Year • Proposals Submitted • Proposals Funded, Nonfunded • Carnegie Classification, Designation • Phase I Amount, Matching • Phase I Course Content • NASA Strategic Enterprise Connections • NASA Field Centers Service Regions • Number of Faculty and Students • Connections to Standards • Number of Courses and Course Level • Phase II Proposals Submitted • Phase II Funded, Nonfunded • Phase III Funded, Nonfunded • Phase III Funding Year, Site • Faculty and Graduate Fellows • GLOBE • Spacegrant • Minority Student Representation • Summer Faculty • Presentations • Journals/Books • Spinoffs, Grants • Proposed Articles • Dissertations

29. 5. Conduct a national research project on a specific aspect of the impact of NOVA in the national setting. Possible Research Themes: Enhancement of NASA mission and data as carried out by NOVA Student Understanding of the Nature of Mathematics Impact on graduates of NOVA courses – preservice or inservice Telephone and email interviews of NOVA faculty. Impact on climate and organization in departments and colleges at NOVA institutions. Analysis of NOVA Network web sites Analysis of NOVA Phase III online instructional modules NOVA Faculty Fellow Schedule

30. 6. Summer research synthesis at the University of Alabama. Week research analysis and writing session to develop an overall evaluation report for NOVA June 3-7, 2002, at the University of Alabama. Disseminate results from fall and spring. NOVA Faculty Fellow Schedule

31. 7. Online web pages illustrating national impact of NOVA Communicate results from Fall, 2001 and Spring, 2002 using NOVA Internet web site. Develop web page plan in spring. Design web page concept in summer Final web page construction completed by consultants. NOVA Faculty Fellow Schedule

32. 8. Dissemination of NOVA research and evaluation results Disseminate results of the impact of NOVA in an article submitted for publication, be a lead author with others from the NOVA Faculty Fellow team Presentation at a national conference(s) - AAAS, NARST, AERA NOVA Faculty Fellow Schedule

33. 9. NOVA evaluation report summarizing work of 2001-2003 NOVA Faculty Fellow Schedule

34. NOVA: Program Evaluation Creating change in higher education to enhance science, mathematics and technology literacy of pre-service teachers. NASA Headquarters, Washington D.C. December 6, 2001