Writing More Effective Proposals IV

1. Writing More Effective Proposals IV Project Evaluation Dissemination 1

2. Evaluation

3. Evaluation and Assessment • Project Evaluation • May involve evaluating individual and group performance within the context of the project • Project evaluation • Formative – monitoring progress to improve approach • Summative – characterizing final accomplishments

4. Evaluation and Project Goals/Objectives/Outcomes • Effective evaluation starts with carefully defined project goals, objectives, and expected outcomes • Related to: • Project management • Initiating or completing an activity • Finishing a “product” • Student behavior • Modifying a learning outcome • Modifying an attitude or a perception

5. Approaches to Data Collection • Quantitative methods • What happened? • Qualitative methods • Why? • Mixed methods

6. Examples of Tools for Evaluating and Assessing Learning Outcomes • Surveys • Concept Inventories • Standardized tests • Measuring critical thinking, cognitive ability, attitudes • Rubrics for analyzing student products • Observations • Interviews • Focus groups

7. PD’s ResponseEvaluating a Existing Tool • Nature of the tool • Is the tool relevant to what was taught? • Is the tool competency based? • Is the tool conceptual or procedural? • Prior validation of the tool • Has the tool been tested? Is it sensitive? • Is there information on reliability and validity? • Has it been compared to other tools? • Does it discriminate between a novice and an expert? • Experience of others with the tool • Has the tool been used by others besides the developer? At other sites? With other populations? • Is there normative data?

8. Topics to be Addressed in an Evaluation Plan • Name & qualifications of the evaluation expert • Get the evaluator involved early in the proposal development phase • Explicit relationship between intervention and outcomes • Goals, objectives, and outcomes evaluation questions • Instruments for evaluating each outcome • Protocols defining when and how data will be collected • Analysis & interpretation procedures • Confounding factors & approaches to minimize impact

9. Formative evaluation techniques for monitoring and improving the project as it evolves • Summative evaluation techniques for characterizing the accomplishments of the completed project

10. Evaluation Resources • NSF’s User Friendly Handbook for Project Evaluation • http://www.westat.com/pdf/projects/2010ufhb.pdf • ATE Evaluation Resource Center • http://evalu-ate.org/ • Online Evaluation Resource Library (OERL) • http://oerl.sri.com/ • Field-Tested Learning Assessment Guide (FLAG) • http://www.wcer.wisc.edu/archive/cl1/flag/default.asp • Student Assessment of Their Learning Gains (SALG) • http://www.salgsite.org/ • Evaluating & Improving Undergraduate Teaching in STEM • National Research Council (2003)

11. How to Find an Evaluator Institution’s Center for Teaching and Learning Institution’s Center for Evaluation (in STEM) Colleges and Departments (Education, Ed Psych, Engineering, Science, Public Admin) Colleagues Professional Societies American Evaluation Association Website

12. What to Expect from Your Evaluator • Knowledge about evaluation design and methodology(evaluation competencies) • Experience in evaluating your type of project • A quality evaluation plan • A contract or an agreement with deliverables and timeline • Knowledge of evaluation principles, standards, ethics • [Increasing] knowledge about your project • Frequent, respectful communication

13. What Your Evaluator Expects from You • A clear definition of your project goals, objectives, and expected outcomes • All available information about your project • Frequent communication • Questions • Access to data/participants • Help with problem-solving • Resources and payment

14. Helpful Hints • Identify an evaluator in advance. • Match evaluation with goals, objectives, and outcomes • Ensure design provides evidence about what is and isn’t working for improvement • Remember to include not only accountability information, but also impact and effectiveness • Develop indicators to measure progress and include time lines

15. 10 Helpful Hints (con’t) • Develop the plan jointly with an evaluator • Assign responsibilities for various components • Use the evaluation literature. • Keep in mind your evaluation stakeholders • Use descriptions, tables, visuals

16. Fatal Flaws • The evaluation section • Is missing • States that after receiving funding, the evaluation will be developed • Only evaluates easy things • Has unreasonable or unrealistic budget • Does not align with priorities of funding program

17. Fatal Flaws (con’t) The evaluation section • States PIs will do all the evaluation • Is lacking in details • Was cut and pasted from another proposal • Uses too much jargon • Does not explain methods and instruments to be used

18. Questions and Answers

19. Dissemination

20. Dissemination The Standard Three • Conference presentation(s) • 2. Web site • Publication(s) • These are necessary but not sufficient, i.e., everyone lists these • Question: how many from “target” audience sees/hears this?

21. Program Officers’ Suggestions:Dissemination • Be proactivein promoting website & materials “If you build it they will NOT come” • Target and involve a specific sub-population • Those who teach similar course(s) at other locations • Ask them to reviewvarious products, data, and approaches • Make use of local/regional resources

22. Program Officers’ Suggestions:Dissemination • Work with others to organize • Email exchanges and listserves • Informal meetings at a conference or on-campus • Faculty development workshops (on-campus and at conferences) • Explore beta test sites

23. Program Officers’ Suggestions • Be specific in publication efforts • Indicate the specific conferences and journals • Include conference travel and journal page charges in budget • Explore other venues • Digital libraries, PKAL (http://www.pkal.org), State Academy of Science meetings, local regional associations • Science news publication and lay press • Professional society and specialty listserves

24. Program Officers’ Suggestions • Integrate dissemination with community building and evaluation • Be specific about how the project will serve as a “pilot” for subsequent expansion to other institutions. • How does the dissemination plan, and its evaluation, contribute to getting “buy-in” from others for future adoption? • Explore commercialization • Discuss contacts with software and textbook publishers

25. Questions and Answers

26. Sustainability

27. From Solicitation Sustainability: The purpose of NSF educational grant programs is to bring about lasting improvement in undergraduate STEM education. Proposals should address sustainability and should demonstrate that there is a reasonable expectation of persistent effects (i.e., continuation) of the grant funded work consistent with the aims of the project.

28. Internal Support To sustain/grow project activities after the grant period always requires ongoing institutional support This support typically requires financial resources – supplies, repairs, stipends, space, equipment, release time, etc. A demonstrably successful project and a good relationship with administration are keys for obtaining this support

29. External Support A successful project can be basis of A follow up NSF grant Grants from local businesses, industries, or foundations Fund raising from alums/others

30. Questions and Answers