Proposal Review Process Mock Review Webinar handout version

1. Proposal Review Process Mock Review Webinar(handout version) Louis Everett, Scott Grissom & Don Millard DonDon

2. Most of the information presented in this session represents the presenter�s opinion and is not an official NSF position Caution 2

3. Introduction (5min) Overview of the review process (5min) Instructions on producing a panel review (5min) Organization of participants into teams and designation of panel chairs (5min) Local teams discuss strengths and weaknesses (20min) Teams locally report/discuss results (10min) Reporting to virtual group with ratings � note: facilitators will be asked to select an individual to report (10min) BREAK (15min) Individuals consider ways to improve the proposal (5min) Local teams discuss suggestions for improvement (10min) Teams locally report/discuss results (5min) Report back to virtual group - note: facilitator picks an individual to report (5min) PD commentary on responses (10min) � Think �� What have I learned today that I will use in preparing my next proposal? (5min) Share your thoughts with local participants (5min) Facilitator reports results back to virtual group (5min) � Wrap-up Q&A (10min) Webinar Agenda

4. Help participants to: Become more familiar with the proposal review process Better understand the TUES-specific criteria Better understand the use of intellectual merit/broader impact criteria in reviewing proposals Develop more competitive proposals that effectively meet the expectations of the TUES program Please note: A number of the following slides are provided for informational purposes � we will not be going through all of them Webinar Goals - Expected Outcomes

5. TUES Program

6. Title changed to emphasize the�special interest in projects that have the potential to transform undergraduate STEM education� Review criteria was modified to emphasize the desire for projects that: Propose materials, processes, or models that have the potential to Enhance student learning Be adapted easily by other sites Involve a significant effort to facilitate adaptation at other sites Institutionalize the approach at the investigator's college or university as appropriate (e.g., for the Type) Have the potential to contribute to a significant advancement and cultural shift in undergraduate STEM education TUES vs. CCLI

7. TUES Program Vision: Excellent STEM education for all undergraduate students Reflects national concerns about producing: Skilled STEM professionals (including K-12 teachers) Citizens knowledgeable about STEM and how it relates to their lives Seeks to build a community of faculty committed to improving undergraduate STEM education Encourages projects with potential to transform undergraduate STEM education

8. Creating Learning Materials and Strategies Guided by research on teaching and learning Incorporate and be inspired by advances within the discipline Implementing New Instructional Strategies Contribute to understanding on how existing strategies: Can be widely adopted Are transferred to diverse settings Impact student learning in diverse settings Developing Faculty Expertise Enable faculty to acquire new knowledge and skills in order to revise their curricula and teaching practices Involve a diverse group of faculty TUES Project Components

9. Assessing and Evaluating Student Achievement: Develop and disseminate valid and reliable tests of STEM knowledge Collect, synthesize, and interpret information about student understanding, reasoning, practical skills, interests, attitudes or other valued outcomes Conducting Research on Undergraduate STEM Education: Explore how: Effective teaching strategies and curricula enhance learning and attitudes Widespread practices have diffused through the community Faculty and programs implement changes in their curriculum TUES Project Components (cont)

10. Projects developing instructional materials and methods should: Be based on how students learn Consider transferability and dissemination throughout the project's lifetime Involve efforts to facilitate adaptation at other sites in more advanced projects Instructional Materials and Methods Projects

11. Expect to award approximately 10% Total budget: up to $200,000 for 2 to 3 years $250,000 when 4-year and 2-year schools collaborate Typically involve a single institution & one program component � but there are exceptions Contribute to the understanding of undergraduate STEM education Informative evaluation effort based on the project's specific expected outcomes Institutionalized at the participating colleges and universities Deadlines: May 26, 2011 (A-M) May 27, 2011 (N-W) Type 1 Projects

12. Type 2 Projects 20 to 25 awards expected Total budget: up to $600K for 2 to 4 years Type 3 Projects 3 to 5 awards expected Budget negotiable, but not to exceed $5M over 5 years Tues Central Resource Projects 1 to 3 awards expected Budget negotiable, depending on the scope and scale of the activity Small focused workshop projects -- 1 to 2 years & up to $100K Large scale projects -- 3 to 5 years & $300K to $3M Deadline: January 13, 2012 Type 2, 3, and CRP Projects

13. Review Process NingNing

14. NSF Peer Review Process Reviewers are solicited by program directors For example - A Typical TUES Type 1 Panel Review: 4 panels/program director (60-70 proposals/PD) 6-8 reviewers/panel 17 proposals/panel � not all read by every panelist Approximately 130 engineering reviewers Reviewers assign individual ratings and prepare written comments on Intellectual Merit & Broader Impacts prior to coming to the panel meeting Panel meeting is held in DC area � over a 1.5 day period

15. Panel Review Meeting Panel Chair establishes order of proposal review process Proposals are discussed individually A �scribe� is designated to capture all of the points brought up in discussion and produce a summary review � called the �Panel Summary� All reviewers return on day 2 to approve all the Panel Summaries

16. NSF program directors Informs recommendations relative to funding Guides pre-award negotiations Applicants If proposal is funded: Provides suggestions for improving project If proposal is not funded: Provides information to guide a revision of the proposal Audience for Reviews

17. NSF Review Criteria All proposals are evaluated using the NSB-approved review criterion Intellectual merit Broader impacts The TUES Solicitation provides two sets of suggested questions to help define these criteria Standard NSF set TUES specific set

18. Suggested questions are only a guide for considering intellectual merit and broader impacts Suggested questions are NOT: A complete list of �requirements� Applicable to every proposal An official checklist Caution Regarding Suggested Questions

19. Will the project: Include activities important in advancing knowledge? Involve qualified proposer(s)? Contain creative and original concepts? Have a well conceived and organized plan? Include sufficient access to resources? NSF Suggested Questions for Intellectual Merit

20. Will the project: Advance discovery - promote teaching & learning? Broaden participation of underrepresented groups? Enhance the infrastructure? Include broad dissemination? Benefit society? NSF Suggested Questions for Broader Impacts

21. Will the project: Produce one or more of the following: Exemplary materials, processes, or models that enhance student learning and can be adopted by other sites Important findings related to student learning? Build on existing knowledge about STEM education? Have explicit and appropriate expected measurable outcomes integrated into an evaluation plan? Include an evaluation effort that is likely to produce useful information? Institutionalize the approach at the investigator's college or university (as appropriate for the Type) TUES Suggested Questions for Intellectual Merit

22. Will the project: Involve a significant effort to facilitate adaptation at other sites? Contribute to the understanding of STEM education? Help build and diversify the STEM education community? Have a broad impact on STEM education in an area of recognized need or opportunity? Have the potential to contribute to a significant advancement and cultural shift in undergraduate STEM education? TUES Suggested Questions for Broader Impacts

23. Writing the Review

24. The Entire Proposal is Used to Inform Reviewers Project Summary Project Description Biographical Sketches Budget Supplementary Documentation Review Material 24 During talk need to give the appropriate directive relative to the need to read or use Supplementary DocumentationDuring talk need to give the appropriate directive relative to the need to read or use Supplementary Documentation

25. Overview A review should indicate an opinion on the merit of the project The rating should indicate an overall evaluation of the proposal�s merit Excellent, Very Good, Good, Fair, and Poor Describe positive aspects Not just list them -- Provide details Identify concerns (or weaknesses) Not just list them -- Provide details Offer suggestions for improvement Rating and text should be consistent

26. Uses appropriate style Contains adequate details Contains understandable, specific, and complete statements Relates strengths and weakness to review criteria Indicates why an item is a strength or weakness Justifies the proposal rating in the written critique A reader should be able to guess the rating from the written text Characteristics of Informative Reviews and Panel Summaries

27. Specific and Complete Comments Identify a strength or weaknesses �The evaluation plan is a strength.� Identify a strength or weaknesses and indicate why it is one �The evaluation plan includes a competent, independent evaluator...�  �The background discussion is well referenced, shows a good understanding of the prior work, supports the proposed work...�

28. Specific and Complete Comments (cont.) Identify a strength or weaknesses and indicate why it is one and why it is important �The evaluation plan includes a competent, independent evaluator, � and it will guide the investigators as the project evolves and provide a measure of its effectiveness at the end.� Identify a strength, indicate why it is one, why it is important, and how it could be improved �The evaluation plan is a strength because it includes a competent, independent evaluator, � and this will guide the investigators as the project evolves and provide a measure of its effectiveness at the end. It could be improved by adding ��

29. Try to be constructive in your written comments Provide suggestions to help applicants improve their proposals Do not be overly critical in your ratings Most fundable proposals have some weaknesses Some are correctable through negotiations Other Important Ideas

30. Strengths & Weakness DonDon

31. Local teams discuss the strengths and weaknesses identified in the individual reviews Team �scribe� takes notes during discussion Teams locally report/discuss results with facilitator�s guidance Facilitators guide discussion and selection of an individual to report to the full virtual group Participating organizations will be called upon by webinar moderator � PLEASE BE READY TO REPORT BACK NSF Program Directors will offer comments on reports Team Activity Process

32. Redundancy is OK � indicative of common perception Responses will be solicited as follows: Intellectual Merit Strengths Weaknesses Broader Impacts Strengths Weaknesses Overall Perception Is this a proposal worthy of funding? Reporting

33. ONE MINUTEUntil Reporting Begins

34. Redundancy is OK � indicative of common perception Responses will be solicited as follows: Intellectual Merit Strengths Weaknesses Broader Impacts Strengths Weaknesses Overall Perception Is this a proposal worthy of funding? Reporting

35. Questions?

36. BREAK(15 min)

37. ONE MINUTEUntil Next Session

38. Suggestions for Improvement NingNing

39. Individuals consider ways that the proposal could be improved � create a list Local teams discuss the suggestions for improvement Team �scribe� takes notes during discussion Teams locally report/discuss results with facilitator�s guidance Facilitators guide discussion and selection of an individual to report to the full virtual group Participating organizations will be called upon by webinar moderator � PLEASE BE READY TO REPORT BACK NSF Program Directors will offer comments on reports Improvements Activity Process

40. Try to minimize redundant responses Identify approaches for: Building on strengths Overcoming weaknesses Responses should include potential improvements to: Idea Project infrastructure Project implementation plan Evaluation plan Dissemination plan Proposal participation/involvement Reporting

41. ONE MINUTEUntil Reporting Begins

42. Try to minimize redundant responses Identify approaches for: Building on strengths Overcoming weaknesses Responses should include potential improvements to: Idea Project infrastructure Project implementation plan Evaluation plan Dissemination plan Proposal participation/involvement Reporting - Reminder

43. Sample Proposal Review Comments Regarding the Proposal�s Intellectual Merit, Broader Impacts and Weaknesses DonDon

44. The following comments reflect a combination of the proposal�s panel review, individual reviewers and inputs provided by NSF program officers Sample Proposal Review Comments

45. Practical Aspects of Review Process NingNing

46. Reviewers have: Many proposals Ten or more from several areas Limited time for your proposal 20 minutes for first read Different experiences in review process Veterans to novices Different levels of knowledge in proposal area Experts to outsiders Discussions of proposals� merits at panel meeting Share expertise and experience Practical Aspects of Review Process 46

47. Write down a list of suggestions (guidelines) that you would suggest that a colleague should follow - to deal with practical aspects of the Review Process 2 minutes Practical Aspects of Review Process 47

48. ONE MINUTE

49. The following comments reflect perspectives provided by NSF program officers Considering the Practical Aspects of the Review Process

50. Reflection Don Don

51. Individuals consider: What have I learned today that I will be able to used in preparing my next proposal?  Create a list (5min) Share your list with local participants (5min) Facilitators report back to virtual group (5min) Participating organizations will be called upon by webinar moderators � PLEASE BE READY TO REPORT BACK Reflection Activity

52. Closing Comments

53. Model good practices that increase/improve learning Facilitate direct experience with the methods and processes of inquiry/experimentation Empower the student Enhance ability to produce and evaluate innovative results Products Solutions to problems Metrics on outcomes Establish a community that will help inform, disseminate and sustain engineering education efforts Toward Transformation�

54. Good idea + need Right people + infrastructure Can be readily adopted at other sites Assessment of outcomes that measure effect on student learning (with goals/outcomes linked to evaluation) Active dissemination plan Shows promise for institutionalization Efforts to broaden participation of underrepresented groups A Successful TUES Proposal�

55. Questions?

56. Thanks for participating!Please respond to the survey located at:http://www.step.eng.lsu.edu/nsf/participants Louis Everett � leverett@nsf.gov Scott Grissom � sgrissom@nsf.gov Don Millard � dmillard@nsf.gov