CASL Portfolio: Advancing Cognitive Science in Education

1. Grantee MeetingCognition and Student Learning 2010 IES Research Conference Carol O’DonnellProgram Officer

2. Agenda for Grantee Meeting (1:30 – 1:35) • 1:35 – 1:45 Introductions (and important note re: 2011 IES conference) • 1:45 – 2:15 State of the CASL Portfolio • Statistics • Highlights • Trends and Gaps • 2:15 – 2:45 Discussion Groups: Making Findings Relevant • 2:45 – 3:15 Report Out: Making Research Relevant • 3:15 – 4:00 Research Issues & Solutions • 4:00 – 4:30 Grant Monitoring Tips (Refresher & for New Grantees) • Grants Administration • Website • New Funding Opportunities

3. Introductions (1:35 – 1:45) • Note about 2011 IES Conference – size is increasing, need program committee to conduct peer review of papers, panels, poster; grantee meetings will piggyback off of SREE conference. • See list of grantees (handout) and posters (pp 105 – 115) • Rapid Introductions—put a face to a name • Name • University • Goal (Exploration, Develop., Efficacy, Measurement) • Area of study

4. State of the CASL Portfolio (1:45 – 2:15) • Purpose of CASL Program • Portfolio Statistics • CASL Highlights • Trends and Gaps in the CASL Portfolio

5. Purpose of CASL Program • To establish a scientific foundation for education by building on the theoretical and empirical advances of cognitive science and applying them to education practice with the goal of improving student learning and academic achievement.

6. Portfolio Statistics

7. Number of CASL Awards (2002-2010) From FY02-FY10 IES funded 78 CASL research projects, averaging ~ 9 grant awards per year. Funding rate (# awards / # responsive apps) average around 10-12% but will drop as the number of applications increases.

8. Content CASL Studies (2002-2010) Majority of CASL projects focus on math or science, the primary areas of need identified by Bob Bjork in his 2002 address to the IES staff. Fewer focus on reading or writing. Study skills are often contextualized within a domain. Some projects overlap domains.

9. Age of Participants in CASL (2002-2010) # of apps for college students alone dropped to 0% in FY09; many pre-K apply to Early Childhood Program.

10. Number of Funded CASL Projects Per Goal (2002-2010)

11. CASL Highlights(see highlights handout)

12. CASL Highlights • More than 20 CASL projects were highlighted in the IES Internal Weekly Report and IES Online Newsletter this past year. • More than 18 articles with positive findings were submitted to the Commissioner for our IES management goals. • See the attached sheet for some examples. • Thank you to those who send their publications to me at the time of their publication (not in press).

13. Findings from 38 separate CASL projects were highlighted in ten accepted symposia presented at the Association for Psychological Science 2009 and 2010 annual conventions. Symposia topics included: • Developing students’ mental models using animated pedagogical agents. • Learning from concrete and abstract representations. • Test-enhanced learning, spacing, and retrieval-feedback-monitoring. • Self-regulated learning. • Fundamental understanding of mathematics. • Conducting RCT’s of attention interventions in school settings. • Argumentation for critical thinking. • Problem solving in schools. • Neurocognitive functions underlying academic achievement. • Perceptual characteristics and concept mastery.

14. 12 CASL grantees contributed to a compendium, Handbook of Metacognition in Education(2009), which provides comprehensive coverage of the theoretical basis of metacognition and its applications to educational practice. Contributors include CASL grantees: • John Dunlosky (editor) • Arthur C. Graesser (editor) • Margaret McKeowen • Danielle McNamara • Joe Magliano • Keith Thiede • Thomas Griffin • Jenny Wiley • Joshua Redford • Barry Zimmerman • Adam Moylan • Janet Metcalfe

15. National honors bestowed upon two CASL grantees: • David Klahr honored by the 37th Carnegie Symposium on Cognition for his lifetime of scientific and education contributions to cognitive development, scientific discovery, and reasoning. • Katherine Rawson received the 2009 Presidential Early Career Award for Scientists and Engineers for her research on improving the comprehension of text and on helping students self-regulate their learning. Nicole McNeil won this prestigious honor last year. • CASL findings highlighted in national education news outlets and publications: • Education Week reported on three CASL studies focused on argumentation (Britt, Kuhn, and Anderson) and one focused on test-enhance learning (Roediger). • The Chronicle of Higher Education reported on the findings of CASL PI Henry Roediger’s work, which confirmed that frequently quizzing students on reading material helps them retain what they have learned. • Rohrer and Pashler’s review of experimental studies on optimal instructional strategies published in Educational Researcher.

16. Trends and Gaps in the CASL Portfolio(see chart handout)

18. Trends and GAPS in CASL Portfolio • Attention addressed in earlier grades (Neville, Steiner, Hooper, Rabiner), but not in middle or high school. • Perception studied in all grades (Cottrell, Mix, Siegler, Sloutsky, Uttal, Goldstone, Kellman, Alibali, Whitten, Novick, Puntambekar, Pavlick), but limited to Goal 2 with one exception (Pasnak—efficacy of patterning). • Memory—including retrieval, learning, and general conception and misconception—and optimal conditions of learning are the most comprehensively addressed in the portfolio; all grades and all Goals (Russell, Ward, Pashler, Rawson, Heckler, Metcalfe, Bjork, Roediger, Delaney-Black, Millis, Anderson, Blair, Aronson, Biswas, Thiede, Wiley, Zimmerman, Dunlosky).

19. Trends and GAPS in CASL Portfolio • Languagestudied predominantly in the early grades (Anthony, Mostow, Connor, Hooper, and Glenberg), with one in middle grades (McCutchen). None in high school. • Problem solvingcovers all grades in Math and Science but mostly Goal 2 (Whitten, Swanson, Star, Sternberg, Beal, Bottge, Booth, McNeil, Ward, Ross, Rabinowitz). • Argumentation and reasoning(including scientific and mathematical) are well-represented (Katz, Klahr, Lorch, Anderson, Kuhn, Gholson, Holyoak, Rips, Britt), but there are no CASL studies of reasoning in the earlier grades. • Bigger gaps exist within teacher cognition (Alibali) and social cognition / goal orientation (Dweck, Aronson, Mangels, Beilock).

20. 2:15 – 2:45 Discussion Groups: Making Findings Relevant(suggested division into Groups 1-7)

22. Classroom Scenario – Group Discussion • The previous list of Trends and Gaps in the CASL portfolio represents a series of interdependent cognitive processes involved in making sense of new information within the education setting—attention, perception, memory and learning, language, problem solving, argumentation, and reasoning. • At a practical level, all of the processes and structures of the cognitive system are interdependent and influenced by both the environment and by student characteristics. • Review the list of select CASL findings highlighted on handout. • Read the scenario, which attempts to demonstrate the complexity of what a teacher and student might consider if the findings from CASL were implemented into one school day.

23. Classroom Scenario – Group Discussion • Knowing your own findings and those highlighted, discuss the following: • What trends exist in the CASL portfolio that can cohesively inform a teacher’s instructional practice, development of instructional materials, and students’ study skills? • What gaps are missing? For example, while the CASL portfolio has several studies using artificial intelligence (Millis, Ward & Cole, etc) which involve constructing computer systems that produce intelligent outcomes, there are fewer CASL studies of computational modeling, which involves programming computers to model or mimic some aspect of human cognitive functioning (Anderson, Katz). Other gaps include teacher cognition & social cognition. • How can CASL researchers continue to contribute to the development and testing of basic theory (e.g., neuro-education) yet make research questions relevant to education settings? • How can we ensure CASL exploratory studies are methodological rigorous yet ecologically valid and lead to the development of feasible education interventions?

24. 2:45 – 3:15 Report Out:As you report out your group’s discussion of the 4 questions, consider how we might move forward by making Research relevant.

25. Problems and Solutions (3:15 – 4:00)Continue our Discussion of Last Year’s Topics Methodological Issues (measurement challenges, observational techniques, translating laboratory research to the classroom—effect size reduction from lab to school JREE) Logistical Issues (obtaining preK-12 samples; reducing attrition; maintaining school collaboration; ensuring feasibility & fidelity of implementation) DisseminationIssues (getting research into the hands of practitioners, sustaining the intervention after the grant ends) Future Grant Issues (Goal 1 vs Goal 2; moving from a Goal 2 to 3; applying to Ed Tech vs CASL) Others?

26. 4:00 – 4:30 Grant Monitoring Tips (Refresher & for New Grantees)- Grants Administration- Website - New Funding Opportunities

27. Grants Administration

28. Using the “e-grants” system Annual Reports – relationship to performance agreement Final Reports Requesting no-cost extensions Requesting other administrative changes

30. Annual Reports(see attached Tips Sheet) Cover Sheet Section A – Project Objectives Section B – Budget Information Section C – Additional Information

31. Cover Sheet Complete online Remember to send original signed cover sheet to: Director, Grants Administration US Department of Education/IES 555 New Jersey Ave., Room 508c Washington, D.C. 20208 FAX: (202) 219-2159

32. Section A – Project Objectives Select “PROJ” for “Measure Type.” Ignore all other “Performance Measure” boxes. Your Performance Agreement lists your project objectives for each year. List each objective in Section A, then cut and paste text beneath each objective to describe how you have met it (see the next slide).

33. Write your objective from the performance agreement here. PROJ Explanation of progress on this objective goes here. This is often several paragraphs long.

34. Section A – Project Objectives Describe status of all steps taken toward completion of your projects. Include details of what was done and how it was done. Describe what was completed. Describe any findings to date. Describe any work planned, but not undertaken.

35. Section B – Budget Information Provide an explanation if you did not expend funds at expected rate during the reporting period. Describe any significant changes to your budget resulting from modification of project activities. Describe any changes to your budget that affected your ability to achieve your objectives. Include a list or table of broad categories (e.g., Personnel, Travel, Subcontract) that compares the Award Amount for the given grant year vs. Actual Expenses.

36. Section C – Additional Information Attach a file the contains any additional information that you may want to include, such as: Changes you plan to make for next year that are consistent with scope and objectives. Anticipated changes to key personnel. Updated curriculum vitae for proposed personnel. Information on the measures that will be developed or have been used in your study Unanticipated outcomes or benefits from your project. List of grant-related presentations & publications. Copies of papers or posters related to the grant - ERIC. Revised/anticipated timeline for the upcoming year There is no limit as to how many pages you can include here, but it has to be one document.

37. Other Attachments Charts – attach the document that contains any figures or charts referenced in Section A. Tables – attach the document that contains any tables referenced in Section A. Program Specific Requirements – attach your revised IRB Certification here.

38. Final Reports Due 90 days after the end of your grant award. Include the same types of information as in the Annual report, described above, but summarize the entire project Must answer the final questions (1, 2, 3).

39. Final Report Question 1 Utilizing your evaluation results, draw conclusions about the success of the project and its impact. Describe any unanticipated outcomes or benefits from your project and any barriers you may have encountered.

40. Final Report Question 2 What would you recommend as advice to other educators that are interested in your project? How did your original ideas change as a result of conducting this project?

41. Final Report Question 3 If applicable, describe your plans for continuing the project (sustainability; capacity building) and/or disseminating the project results.

42. Administrative Changes (see NCE handout) No-cost extensions (why performance agreement updates are important) Changes to key personnel

43. Using e-grants http://e-grants.ed.gov

44. Change of Key Personnel listed on GAN – upload CV for new person No-Cost Extension (NCE)

45. IES Website:National Center for Education Research (NCER) Go to: http://ies.ed.gov/ncer

49. To learn about other CASL grantees’ work, pick a Year, Goal, or PI, then click on the project URL.