Dealing with Project Evaluation and Broader Impacts (An Interactive, Web-Based Workshop)

1. Dealing with Project Evaluation and Broader Impacts(An Interactive, Web-Based Workshop)

2. Important Notes • Most of the information presented in this workshop represents the presenters’ opinions and not an official NSF position • Local facilitators will provide the link to the workshop slides at the completion of the webinar. • Participants may ask questions by “raising their virtual hand” during a question session. We will call on selected sites and enable their microphone so that the question can be asked. • Responses will be collected from a few sites at the end of each Exercise. At the start of the Exercise, we will identify these sites in the Chat Box and then call on them one at a time to provide their responses.

3. Framework for the Session • Learning must build on prior knowledge • Some knowledge correct • Some knowledge incorrect – Misconceptions • Learning is • Connecting new knowledge to prior knowledge • Correcting misconceptions • Learning requires engagement • Actively recalling prior knowledge • Sharing new knowledge • Forming a new understanding

4. Preliminary CommentsActive & Collaborative Learning • Effective learning activities • Recall prior knowledge -- actively, explicitly • Connect new concepts to existing ones • Challenge and alter misconceptions • Active & collaborative processes • Think individually • Share with partner • Reportto local and virtual groups • Learn from program directors’ responses

5. Facilitator’s Duties • Coordinate the local activities • Watch the time • Allow for think, share, and report phases • Reconvene on time -- 1 min warning slide • Ensure the individual think phase is devoted to thinking and not talking • Coordinate the asking of questions by local participants and reporting local responses to exercises

6. Participant Activities • Long Exercise ---- 6 min • Think individually -------- ~2 min • Share with a partner ----- ~2 min • Report in local group ---- ~2 min • Short Exercise ------ 4 min • Think individually --------- ~2 min • Report in local group ---- ~2 min • Individual Exercise ----------- 2 min • Questions ----------- 5 min • Reports to Virtual Group----- 5 min

7. Goal for Project Evaluation Session The session will enable you to collaborate more effectively with evaluation experts in preparing credible and comprehensive project evaluation plans…. it will not make you an evaluation expert.

8. Session Outcomes After the session, participants should be able to: • Discuss the importance of goals, outcomes, and questions in the evaluation process • Cognitive and affective outcomes • Describe several types of evaluation tools • Advantages, limitations, and appropriateness • Discuss data interpretation issues • Variability, alternative explanations • Develop an evaluation plan in collaboration with an evaluator • Outline a first draft of an evaluation plan

9. Evaluation and Assessment • The terms evaluation and assessment have many meanings • One definition • Assessment is gathering evidence • Evaluation is interpreting data and making value judgments • Examples of evaluation and assessment • Individual’s performance (grading) • Program’s effectiveness (ABET and regional accreditation) • Project’s progress and success (monitoring and validating) • Session addresses project evaluation • May involve evaluating individual and group performance – but in the context of the project • Project evaluation • Formative – monitoring progress to improve approach • Summative – characterizing and documenting final accomplishments

10. Project Goals, Expected Outcomes, and Evaluation Questions

11. Evaluation and Project Goals/Outcomes • Effective evaluation starts with carefully defined project goals and expected outcomes • Goals and expected outcomes related to: • Project management • Initiating or completing an activity • Finishing a “product” • Student behavior • Modifying a learning outcome • Modifying an attitude or a perception

12. Learning Goals and Outcomes • Goals provide overarching statements of project intention What is your overall ambition? What do you hope to achieve? • Expected outcomes identify specific observable or measureable results for each goal How will achieving your “intention” be reflected by changes in student behavior? How will it change their learning and their attitudes?

13. Goals, Expected Outcomes, and Evaluation Questions • Goals → Expected outcomes • Expected outcomes → Evaluation questions • Questions form the basis of the evaluation process • The evaluation process consists of the collection and interpretation of data to answer evaluation questions

14. Exercise I: Identification of Goals/Outcomes • Read the abstract -- Goal statement removed • Suggest two plausible goals • One on student learning • Cognitive behavior • One on some other aspect of student behavior • Affective behavior • Focus on what will happen to the students • Do not focus on what the instructor will do • Long Exercise ---- 6 min • Think individually -------- ~2 min • Share with a partner ----- ~2 min • Report in local group ---- ~2 min • Watch time and reconvene after 6 min • Use THINK time to think – no discussion, Selected local facilitators report to virtual group

15. Abstract The goal of the project is …… The project is developing computer-based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of the project are … Non engineers should substitute: “Organic chemistry” for “statics and mechanics of materials” “Interactions” for “external forces and internal reactions”

16. One Minute

17. PDs’ Response: Goals on Cognitive Behavior GOAL: To improve conceptual understanding and processing skills • In the context of course • Draw free-body diagrams for textbook problems • Solve 3-D textbook problems • Describe the effect(s) of external forces on a solid object orally • In a broader context • Solve out-of-context problems • Visualize 3-D problems • Communicate technical problems orally • Improve critical thinking skills • Enhance intellectual development

18. PDs’ Response: Goals on Affective Behavior GOAL: To improve • Self- confidence • Attitude about engineering as a career

19. Exercise II: Transforming Goals into Outcomes • Write one expected measurable outcome for each of the following goals: • Improve the students’ understanding of the fundamental concepts in statics (cognitive) • Improve the students’ self confidence (affective) • Individual exercise ~ 2 minutes • Individually write a response

20. One Minute

21. PDs’ Response: Expected Outcomes Understanding of the fundamentals • Students will be better able to: • Describe all parameters, variable, and elemental relationships • Describe the governing laws • Describe the effects of changing some variable in a simple problem • Changes in the frictional force on a block when the angle of an inclined plane changes • Changes in the forces in the members of a simple three element truss when the connecting angles change Self-Confidence • Students will: • Do more of the homework • Have less test anxiety • Express more confidence in their solutions • Be more willing to discuss their solutions

22. Exercise III: Transforming Outcomes into Questions • Write an evaluation question for these expected measurable outcomes: Understanding of the fundamentals • Students will be better able to describe the effects of changing some variable in a simple problem Self-Confidence • Students will express more confidence in their solutions • Individually identify a question for each • Report to the group

23. One Minute

24. PDs’ ResponseEvaluation Questions Understanding of the fundamentals • Are the students better able to describe the effects of changing some variable in a simple problem • Are the students better able to describe the effects of changing some variable in a simple problem as a result of the intervention Self-Confidence • Do the students express more confidence in their solutions • Do the students express more confidence in their solutions as a result of the intervention

25. Tools for Evaluating Learning Outcomes

26. Examples of Tools for Evaluating Learning Outcomes  • Surveys • Forced choice or open-ended responses • Concept Inventories • Multiple-choice questions to measure conceptual understanding • Rubrics for analyzing student products • Guides for scoring student reports, tests, etc. • Interviews • Structured (fixed questions) or in-depth (free flowing) • Focus groups • Like interviews but with group interaction • Observations • Actually monitor and evaluate behavior Olds et al, JEE 94:13, 2005 NSF’s Evaluation Handbook

27. Surveys Efficient Accuracy depends on subject’s honesty Difficult to develop reliable and valid survey Low response rate threatens reliability, validity & interpretation Observations Time & labor intensive Inter-rater reliability must be established Captures behavior that subjects are unlikely to report Useful for observable behavior Olds et al, JEE 94:13, 2005 Comparing Surveys and Observations

28. Example – Appropriateness of Interviews • Use interviews to answer these questions: • What does program look and feel like? • What do stakeholders know about the project? • What are stakeholders’ and participants’ expectations? • What features are most salient? • What changes do participants perceive in themselves? The 2002 User Friendly Handbook for Project Evaluation, NSF publication REC 99-12175

29. Tool for Measuring Conceptual Understanding – Concept Inventory • Originated in physics -- Force Concept Inventory (FCI) • Several are being developed in engineering fields • Series of multiple choice questions • Questions involve single concept • Formulas, calculations or problem solving skills not required • Possible answers include detractors • Common errors -- misconceptions • Developing CI is involved • Identify misconceptions and detractors • Develop, test, and refine questions • Establish validity and reliability of tool • Language is a major issue

30. Tool for Assessing Attitude • Pittsburgh Freshman Engineering Survey • Questions about perception • Confidence in their skills in chemistry, communications, engineering, etc. • Impressions about engineering as a precise science, as a lucrative profession, etc. • Validated using alternate approaches: • Item analysis • Verbal protocol elicitation • Factor analysis • Compared results for students who stayed in engineering to those who left Besterfield-Sacre et al , JEE 86:37, 1997

31. Tools for Characterizing Intellectual Development • Levels of Intellectual Development • Students see knowledge, beliefs, and authority in different ways • “ Knowledge is absolute” versus “Knowledge is contextual” • Tools • Measure of Intellectual Development (MID) • Measure of Epistemological Reflection (MER) • Learning Environment Preferences (LEP) Felder et al, JEE 94:57, 2005

32. Exercise IV:Considering an Existing Tool • Suppose you where considering an existing tool (e. g., a concept inventory) for use in your project’s evaluation of learning outcomes • What questions would you consider in deciding if the tool is appropriate? • Long Exercise ---- 6 min • Think individually -------- ~2 min • Share with a partner ----- ~2 min • Report in local group ---- ~2 min • Watch time and reconvene after 6 min • Use THINK time to think – no discussion • Selected local facilitators report to virtual group

33. One Minute

34. PDs’ Response: Evaluating a Existing Tool • Nature of the tool • Is the tool relevant to what was taught? • Is the tool competency based? • Is the tool conceptual or procedural? • Prior validation of the tool • Has the tool been tested? • Is there information concerning its reliability and validity? • Has it been compared to other tools? • Is it sensitive? Does it discriminate between a novice and an expert? • Experience of others with the tool • Has the tool been used by others besides the developer? At other sites? With other populations? • Is there normative data?

35. Questions Hold up your “virtual hand” to ask a question.

36. Interpreting Evaluation Data

37. Interpreting Evaluation Data

38. Exercise V: Alternative Explanations For Change • Data suggest that the understanding of Concept #2 increased • One interpretation is that the intervention caused the change • List some alternative explanations • Confounding factors • Other factors that could explain the change • Individual Exercise ---- 2 min • Individually write a response

39. One Minute

40. PDs’ Response: Alternative Explanations For Change • Students learned the concept out of class (e. g., in another course or in study groups with students not in the course) • Students answered with what they thought the instructor wanted rather than what they believed or “knew” • An external event distorted the pretest data • The instrument was unreliable • Other changes in the course and not the intervention was responsible for the improvement • The characteristics of groups were not similar

41. Exercise VI:Alternative Explanations for Lack of Change • Data suggest that the understanding of the concept tested by Q1 did not improve • One interpretation is that the intervention did cause a change that was masked by other factors • Think about alternative explanations • How would these alternative explanations (confounding factors) differ from the previous list?

42. Evaluation Plan

43. Exercise VII: Evaluation Plan • List the topics that need to be addressed in the evaluation plan • Long Exercise ---- 6 min • Think individually -------- ~2 min • Share with a partner ----- ~2 min • Report in local group ---- ~2 min • Watch time and reconvene after 6 min • Use THINK time to think – no discussion • Selected local facilitators report to virtual group

44. One Minute

45. PDs’ Response: Evaluation Plan • Name & qualifications of the evaluation expert • Get the evaluator involved early in the proposal development phase • Goals, outcomes, and evaluation questions • Instruments for evaluating each outcome • Protocols defining when and how data will be collected • Analysis & interpretation procedures • Confounding factors & approaches for minimizing their impact • Formative evaluation techniques for monitoring and improving the project as it evolves • Summative evaluation techniques for characterizing the accomplishments of the completed project.

46. Other Sources • Workshop on Evaluation of Educational Development Projects • http://www.nsf.gov/events/event_summ.jsp?cntn_id=108142&org=NSF • NSF’s User Friendly Handbook for Project Evaluation • http://www.nsf.gov/pubs/2002/nsf02057/start.htm • Online Evaluation Resource Library (OERL) • http://oerl.sri.com/ • Field-Tested Learning Assessment Guide (FLAG) • http://www.wcer.wisc.edu/archive/cl1/flag/default.asp • Student Assessment of Their Learning Gains (SALG) • http://www.salgsite.org/ • Science education literature

47. Reflection on Project Evaluation Identify the most interesting, important, or surprising ideas you encountered in the workshop on dealing with project evaluation

48. Questions Hold up your “virtual hand” to ask a question.

49. BREAK 15 min

50. BREAK 1 min