Modeling Assessments of Innovative Physics Courses

1. Modeling Assessments of Innovative Physics Courses A Symposium

2. Institution’s Educational Outcomes To prepare students who… • possess a breadth of integrated, fundamental knowledge in basic physics • build a depth of knowledge in exploring the contradictions between personal assumptions and evidence from the physical world.

3. can frame and resolve ill-defined problems • can communicate effectively • are independent learners • can work effectively with others • are intellectually curious • can apply knowledge and skills to the unique tasks of a profession

4. Physics Department’s Curriculum Goal • to develop introductory curriculum which exploits the use of new technology in order to promote the development of critical thinking and problem solving skills, thus enhancing career preparation.

5. ZX Approach to Teaching Physics • required inventive problem solving that • addressed complex situations which had ill-defined approaches and unclear solutions • deluged students with relevant and irrelevant information and • required analysis, criticism, and synthesis

6. ZX Approach to Teaching Physics • modeled conditions met by human beings in real world vs. working within an abstracted or generalized framework • encouraged scholarly research using the CD-ROM resources • planned for individualized problem-solving • required learner creation of problems to be solved by peers.

7. Policy Support for ZX Approach Part of the education students receive is responding to stress and allocating precious time to a variety of demands. Responding to the demands of coursework is one of the areas in which [students] must often make critical decisions. While the faculty should understand that [students] have many things to do in a given day, this level of activity no way justifies diminishing the rigor of our courses. [Students] must also learn responsibility, and we must allow them the chance to fail as well as succeed. Faculty Handbook, p. 30.

8. Action Research OverviewPurpose • To “track” the impact of innovation in the approach to teaching physics for a semester: • Implementation of CD-ROM resource • Use of e-technology as major instructional tool • Attitudes of instructors and students • Use of ill-defined problems as central instructional approach

9. Multiple Methods ElementsQuantitative Data CollectedPre- and Post-data • Multiple “aptitude” entry measures (Pre-) • Nature of Science Profile (Pre- Post-) • Strategies in Teaching Physics (Pre- Post-) • Attitude to Science (Physics) (Pre- Post-) • Force Concept Inventory (Pre- Post) • Common summative examination (Post-)

10. Multiple Methods ElementsQualitative Data Collected • 3 student interview sessions • Weekly email student diaries • Instructional staff interviews and notes on change process

11. Theoretical FrameworksAdultIntellectual Development • William Perry (1970, 1981) • Dualism -- instructor and textbook is authority • Multiplicity – everyone has right to an opinion • Relativism – analytical thinking skills emerge; ability to critique own ideas and those of others • Validity – not all ideas/positions are equally valid

12. Theoretical FrameworksExpert vs. Novice Experts have…. • Ability to chunk info into familiar patterns • Knowledge organized around core concepts or big ideas • Knowledge reflects contexts of applicability s • Automaticity in retrieving relevant characteristics for problem solving • A need to possess content pedagogical knowledge • Mental flexibility, a positive view of one’s own accomplishments and a belief that there is much more to learn.

13. Theoretical FrameworksRIGOR DEFINED • Rigor is the goal of helping students develop the capacity to understand content that is complex, ambiguous, provocative, and personally or emotionally challenging. Strong. R., H. Silver and M. Perini. (2001) Teaching What Matters Most: Standards and Strategies for Raising Student Achievement. Alexandria, VA: Association of Supervision and Curriculum Development. P. 7.

14. Interview Protocol • 3 instructors on experimental team • 12 students, 6 from each experimental course • Selected randomly, ensuring gender balance • 3 interviews • Entry level – first week of semester • Mid-point • Exit level – 2 weeks before final exam Feedback from interviews given to instructors

15. Interview Cohort12 students from experimental group Reasons for volunteering…. • 9 use of computers to learn • 7 CD-ROM instead of textbook • 3 no books to carry • 1 many new resources available • 1 a new opportunity • 1 can explore Physics deeper

16. Interview CohortPre-study interviews Previous Physics experience… • 4 one year High School • 3 two years High School • 2 one year advanced level physics • 1 one year College plus 1 year High School • 2 none

17. Interview CohortPre-study interviews Subjects described learning style… • 5 as visual • 3 as hands-on • 2 as problem-solvers • 2 need an interesting instructor

18. Primary subject interest… 5 Math and Science 1 Science 2 Math 1 Computer Science 1 History 1 Political Science 1 Spanish Least favorite subject… 6 English 3 Writing 2 History 1 Math LESLIE Interview CohortPre-study interviews

19. Interview CohortPost-study interviewsReported experiences and attitudes… Appreciated… • working with others toward a common goal • using technology as a tool for studying Physics

20. Interview CohortPost-study interviewsReported experiences and attitudes… Concerns… • negative impact of time constraints • confusion in choice-making activities • concern about adequate preparation for final departmental assessment

21. Quantitative Research Review:Instruments Used • Strategies in Teaching Physics (SITP) Pre-Post • Nature of Science Profile (NOSP) Pre-Post • Attitude to Science (Physics) (ATP) Pre-Post • Force Concept Inventory (FCI) Pre-Post • Multiple “aptitude” entry measures: Pre- • Common summative final exam: Post-

22. Why Use Surveys?Some of the Research Questions • What are the students’ and instructors’ approaches to the teaching and learning (curriculum) context of physics? (SITP Questionnaire) • What are the students’ and instructors’ approaches to the philosophical context of science (physics)? (NOSP Questionnaire) • What are the students’ and instructors’ attitudes to social and educational contexts of physics - including the use of electronic technology? (ATP Questionnaire)

23. Strategies in Teaching Physics(SITP) Pre- Post-Administered to Instructors and Students Pre-post80 items - 3 subtests - 6 point scale SITP: Examines Three Curriculum Models • Sub-test 1: Neo-classicism(reproduction) (memorize and recall physics facts, concepts and processes) • Sub-test 2: Liberal-progressivism (reconstruction) (active personal constructionof physics knowledge) • Sub-test 3: Social-criticism (interactive reconstruction) (students participate as co-learners of physics)

24. ONE TEACHING-LEARNING CONTEXT INSTRUCTORS’ RESULT (SITP) Four instructors exhibited noticeable differences in preferred teaching strategies.

25. Nature of Science Profile (NOSP)Instructors & Students: Pre- Post-24-item, 11 point Likert scale, 5 subtests Examines the philosophical context of science (physics) along five axes: • Relativism < -- > Positivism (RPP axis) • Inductivism < -- > Deductivism (IDP axis) • Contextualism < -- > Decontextualism (CDP axis) • Process < -- > Content (PCP axis) • Instrumentalism < -- > Realism (IRP axis)

26. ONE PHILOSOPHICAL CONTEXT INSTRUCTORS’ RESULTS (NOSP) For the Relativist-Positivist subtest (RPP), the more negative the score, the stronger the Relativist position. The higher the positive score, the stronger the Positivist position – similarly for IDP, CDP, PCP & IRP.

27. Conflicts of Instructors’ Ideas about the Nature of Science (Physics) Context • The NOSP graph shows strongly differing ideas amongst the instructors about the philosophical context of science (physics). • Such philosophical differences about the nature of science (physics) are NOT INCONSEQUENTIAL. • Such fundamental differences of philosophy undercut attempts at curriculum reform.

28. Comparing SITPCurriculum Contexts with the NOSPPhilosophy of Science Contexts Neo-classicism (NC) vsLiberal-progressivism (LP) (knowledge reproduction) (knowledge construction) Realism <-> Instrumentalism (IRP) Inductivism <-> Deductivism (IDP) Positivism <-> Relativism (RPP) Decontextualism <-> Contextualism (CDP) Content <-> Process (PCP) Social Criticism is not represented in the table above because the philosophical differences between LP & SC are far more subtle than between NC & LP (or NC & SC).

29. Attitude to Physics and E-Technology(ATP) – 1 87 questions - 7 Subtests - 7 point scale Subtests: • Social implications of physics: SP • Attitude to scientific inquiry in physics: AP • Attitude to teaching/learning physics: TP • Physics as a personal threat: PTP • Physics as a practical hands-on activity: HP • Socio-cultural value of physics: VP • E-technology in learning/teaching physics: ET

30. The bar graph shows instructors’ variability. Note SP; AP; HP & VP.

31. Attitude to E-Technology(ATP) – 2; 27 Extra Items; 4 sub-tests Created specifically for the research study at the request of the focus group: • Interest in E-technology (7 ZIP items) • Useful aid to study (7 ZUP items) • Aid to Multi-tasking (6 ZMP items) • Personal Control (7 ZCP items)

32. Instructors do not exhibit a uniformly high response. Some head in opposite directions.

33. Action Research Summary: 1 Instructors’ Quantitative Result • If a Physics Department is genuinely interested in developing a coherent and effective approach to collaborative teaching and shared instructional goals, then the skills, attitudes, and dispositions of the instructors must be examined and taken into account in the process of curriculum reform.

34. Students’ Quantitative Result 1Aptitude Pre-tests c.f. Final Exam • 7Aptitude Predictors Employed: • 2 Standardised Achievement Tests • Math Aptitude; Algebra Aptitude; Calculus Aptitude • Chemistry Aptitude; Physics Aptitude • ACCOM Score (Academic Composite of Aptitudes) – used to allocate students to Honours Classes. • THE ONLYPREDICTOR TO PRODUCE A SIGNIFICANT ASSOCIATION WITH FINAL PHYSICS EXAMINATION RESULTS WAS THE CHEMISTRY APTITUDE TEST. WHY?

35. Some Little-known Facts About:Neo-classical Instruction and Examinations. The “Neo-classical Instruction Effect”: • Teaching procedure: “Equal exposure”/”Equal rate” • Student progress: By differential entry aptitude/ability rate – “Slow” students progress slower (differentially) by aptitude… “Fast” students progress faster (differentially) by aptitude. • Student “performance” order: Entry “aptitude” order should be THE SAMEASexit summative exam order (except for minor “accidental” influences – poor teaching, illness, lack of motivation, etc) • Student performance distribution: Remains “normative” but with a differentially greater “spread”/variance (caused by differential “spreading” of rates of progress by aptitude). • Grading:Could be done “fairly” and “equally” at the beginning of semester using valid aptitude/ability test order (theoretically). • If you are a neo-classical teacher, just for “fun”, compare your pre-course aptitude order with your post-course final exam order using Spearman rho!! • You have a problem somewhere if there is a significant difference.

36. Students’ Pre-Post Quantitative Result 2Force Concept Inventory • Hº: “No difference amongst average scores by treatment groups”: RETAINED (Kruskal-Wallis One-way Anova test) If the FCI is a valid and reliable independent test of understanding of Newtonian mechanics, then: • What did the physics course instruction teach the students about Newtonian Mechanics during the semester?

37. Students’ Pre-Post Quantitative Result 3Strategies in Teaching Physics (SITP) • Neo-classical Sub-test (NCP): Pre-Post Preference Scores: DECLINED • Liberal-progressive Sub-test (LPP) Pre-Post Preference Scores: NO DIFFERENCE • Socially-critical Sub-test (SCP) Pre-Post Preference Scores: NO DIFFERENCE Further: No difference by gender, class grouping, or instructor.

38. Students’ Quantitative Result 4 (NOSP) Favoured Ideas Profile • Relativism<->Positivism (RPP): Undiscriminated • Inductivism<->Deductivism (IDP): Undiscriminated • Contextualism<->Decontextualism (CDP): Decontextualism • Process<->Content (PCP): Process • Instrumentalism<-> Realism: (IRP): Realism Further: No difference by gender, class grouping, or instructor.

39. Students’ Pre-Post Quantitative Result 5Nature of Science (Physics) (NOSP) • Relativism<->Positivism (RPP): No difference • Inductivism<->Deductivism (IDP): No difference • Contextualism<->Decontextualism (CDP): Nodifference • Process<->Content (PCP): No difference • Instrumentalism<-> Realism: (IRP): No difference CONCLUSION: Students retained the ideas formed in High School. Physics 110 had no effect on students’ views about the nature of science (physics).

40. Students’ Pre-Post Quantitative Result 6Attitude to Physics (ATP) Pre-Post Sub-test Results: • Social implications of physics (SP): Significantly Lower • Attitude to scientific inquiry in physics (AP): Significantly Lower • Attitude to teaching/learning physics (TP): No difference • Physics as a personal threat (PTP): Significantly Lower • Physics as a practical hands-on activity (HP): No difference • Socio-cultural value of physics (VP): Significantly Lower Further: All pre-post attitudes had positive support. No pre-post difference by gender. There were significant differences by class section and instructor.

41. Students’ Pre-Post Quantitative Result 7 (ATP) Attitude to E-Technology Pre-PostSub-test Results: • Interest in E-technology (ZIP): Significantly lower • Useful aid to study (ZUP): Significantly lower • Aid to Multi-tasking (ZMP): Significantly lower • Personal Control/Vulnerability (ZCP): Significantly lower Further: FemaleZIP, ZMP & ZCP post-tests were significantly lower. Section grouping post-tests were allsignificantly lower. Instructor grouping post-tests were significantly lower for ZIP only.

42. Students’ Final Summative Written Exam Results - 8 Overview of Final Summative Exam Results: • There were no significant differences amongst Final Exam scores by experimental section, by instructor, or by treatment group. • In general, students were not disadvantaged through participation in the CD-ROM project. • It is not possible to make any quantitative statement about academic gains by the CD-ROM project groups through the use of the CD-ROM. • In considering the total student population, Final Exam results for females were significantly lower than for males.

43. Some of the Conclusions and Recommendations 1 • No pre-course aptitude measures were valid outcome predictors – except the chem aptitude test. • There is no value in using the FCI with the current physics curriculum and final examination. • The current teaching and final examination system discriminates against females (shown by pre-post FCI results and course entry aptitude measures). • There were no differences in final performance across sections, instructors, or treatment group.

44. Some of the Conclusions and Recommendations 2 • There was a significant drop in students’pre-post dispositions to the neo-classical teaching model. • Dispositions to liberal-progressive and socially-critical models of curriculum and instruction remained stable. • There were no pre-post changes to sub-test dispositions by gender, section grouping or instructor. • Both instructors and students may be ill-prepared for the teaching of physics using inquiry procedures to support critical thinking and problem-solving approaches directly related to the institutional objectives. • Instructors could benefit from workshops involving history and philosophy of science, curriculum theory, educational psychology, and educational measurement.

45. Some of the Conclusions and Recommendations 3 • Four of six ATP attitudes declined significantly across the semester. Section attitudes converged so that all sections became more alike. • Students’ E-technology attitudes declined across the semester. Some attitudes remained more positive than others. Sections by instructor attitudes converged to become more alike. Reasonable support remained for E-technology. • The Physics 110 programme made no impact on the sophistication of students’ metaphysical viewpoints about the nature of science (physics). • Physics 110 students’ attitudes tend NOT to support inquiry, critical thinking and complex problem-solving teaching and learning procedures.

46. Some of the Conclusions and Recommendations 4 • “Immersion in”, “exposure to”, and “coverage of” will NOT be sufficient to attain the departmental and institutional goals. • Instructors most aligned with departmental and institutional goals are those who receive the least favourable end-of-semester evaluation of teaching . • Students need to be re-educated to think differently about what it means to learn, understand, do, and be competent in physics. • Physics 110 students misunderstand, disagree, oppose, and show discontent with teaching that is not seen to focus directly upon final examination specifications.

47. Some of the Conclusions and Recommendations 5 • The Physics 110 curriculum and instruction model must be reviewed in the light of the last 30 years of research on the intellectual development of young adults. • The Physics 110 curriculum and instruction model must respond to the cognitive agility, flexibility, and innovation required by the ambiguity, complexity and change features of the 21st century world. • Students need to be re-educated to think differently about what it means to learn, understand, do, and be competent in physics and be moved from a passive to an active responsibility for their learning and understanding. • The Physics Department must re-orient its approach to academic “rigour” so that the term no longer implies the brain is a muscle that must be subjected repeatedly to severity, hardship, adversity and stress to improve its functioning and performance.

48. Features of Action Research • Aim: • Create and study change. • Minimal requirements: • Social practice (not content) is the research subject. • Proceed systematically in repeated cycles of observing, planning, doing, and reflecting. • Involve those responsible for current practice and future change. • Collaborate with all stakeholders through appropriate representation. • Useful words: • Re-examine; reinvent; reconceptualize; re-imagine; refine; refresh; reform; rebuild; reconstruct; revise; remodel; regenerate; revisit.

49. Action Research Procedures Procedure: ( = What we did.) • Form a reference group. • Identify or define the problem(s) or issue(s). • Identify clearly what happens now. • Reflect and refocus on how and what might, should, and can be done differently. • Seek disagreement. • Redefine the problem as necessary. • Settle on the research question(s) • Develop a strategy for collecting data, solving the problem or implementing an idea. • Use multiple flexible methods. • Collect data, implement action, problem-solve, test ideas. • Observe, evaluate, conclude – study the consequences of actions, specify learnings, make sense of experience, describe, explain. Ask, “So what?” and “What next?” • Continue with ongoing action research cycles to make further improvements. • Share your learning with others.

50. What We Learned….Still requires work.. What were strengths of study? Institutional, departmental, staff, and student involvement. Cooperative extensive participation of staff and students Extensive data gathering was possible. Non-threatening Non-invasive Non-personal Thorough Unpleasant or unexpected findings were fairly reported Recommendations were made without fear or favour What was missing? Impact of time and budget? Influence of institutional culture? Influence on instructors’ risks?