Worcester Polytechnic Institute Introduces

1. Worcester Polytechnic Institute Introduces Learn about research involving this online formative assessment system and hear about how it is used in classrooms. In this Webinar you will hear from: Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day. Neil Heffernan, a computer science professor about his research using this tool. Matt Militello, a professor of school leadership on how school leaders can use such a system. Cristina Heffernan, a professional developer who trains teachers to use the system and incorporate it into their classrooms. And Russ Rapose, District curriculum coordinator for Milbury Public Schools

2. If Old McDonald Had An Assessment… Matthew Militello Assistant Professor

3. The Press… The Push The Press: • More than ever we are TOLD, MANDATED, and SANCTIONED to use data to inform education decisions. • Specifically, assessment data. Means state tests. The Push: • These data are not timely, not informative for teachers; That is, these data points are autopsy data. • TMI: Too Much Information.

4. In Search of the Right Data! Central Office Student/School Growth School Leaders Teacher Effectiveness Teachers Student Cognition

5. Formative Assessment Systems (FAS) There is Demand…. Supply Followed Multi-Million Dollar Industry- Cost usually based on per student use Not all FAS are the SAME!

6. How Do Districts Select, Support, and Use FAS? • Three Districts Three FAS • Technical Features Implementation • Actual Use

7. Framework

8. Framework Fit is defined by the INTENT of the districts selection (e.g. teacher use (student-diagnostic data) or district use (student growth model data) and the PURPOSE of the assessment system—including validity.

9. Framework Readiness is definedas: (1) building capacity in regard to both knowledge of the assessment product and (2) the motivation to use the product in practice.

10. Framework Coherence is how educators used the assessment systems fit into the context and the standard operating procedures of the district. For instance, the creation of trigger mechanisms from review of data reports.

11. Woodbury Schools FAS- NWEA-MAP Good fit for Central Office, but not teachers Growth data forced on daily practice Teachers ill-equipped

12. Can’t Get Beyond FIT!

13. Can’t Get Beyond FIT!

14. Can’t Get Beyond FIT!

15. Needle in a Haystack:A FAS with Real-time, Student-Level Data! • Fit! Finally, Formative Assessments for TEACHERS • Technical Features Tutoring System for Real-Time Data and Diagnostic Data: PURPOSE • Teacher-Level Use- Desire For This Data: INTENT

16. Using Intelligent Tutoring Systems to Help Math Teachers in the Formative Assessment Process. Prof. Neil HeffernanWorcester Polytechnic Institute

17. Who Am I? • Teacher for America : teacher of middle school math and science in Baltimore City • PhD in Computer Science and Cognitive Science • Licensed to Carnegie Learning Inc. • Tenure professor at Worcester Polytechnic • Grant funded research on developing interventions

18. Quick Project History - 2003 Teacher are being asked to use data to inform instruction, but every minute in the classroom spent on assessment is a minute lost to instruction. US Department of Education (2003) Funded us to build a assessment system for teachers that did not take away teacher’s time.

19. Lindeman, Carolina Ruiz, George Heineman, Murali Mani, Ryung Kim and others 30 others

20. Definitions A diagnostic assessment is an assessment designed to uncover any misconceptions or gaps in knowledge or skills of individual students so that teachers can base their instruction on the current state of students’ knowledge, skills, or understanding. Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve students’ achievement of intended instructional outcomes.

21. Research Questions for US Dept of Ed 1. Does ASSISTments work as an assessment tool? • Reliably predict state test performance? • Reliably advise teachers & students on what knowledge to focus on? 2. Does ASSISTments help enhance student learning? • Does the System effectively teach as it assesses? • Can teachers use the advice of this System to lead to higher student learning?

22. The ASSISTment System • A web-based assessment system, designed to collect formative assessment data on student math skills • Usage has grown as our content has grown. • Currently, thousands of students use the system.

28. You Just Saw One ASSISTment • A question within this “tutor” is what we call an ASSISTment . • We have 2,000 such ASSISTments for 7th and 8th grade. (66 hours of instruction) • We have another 1,000 for 6th grade. • We have 1000 for 9-10th grade math, • There are Three other types of content. • Teacher created questions • Mastery Learning questions (track 130 skills in 8th grade math) • Connected Math matching units (500 ASSISTments with tutoring)

29. Many Different Types of Reports State Test Report Knowledge component reports (based upon 130 middle school skills) Item Report (e.g. for small formative assessments) Mastery Learning Report (which students have reached mastery?)

30. Names Obscured on Purpose

31. Report to a Teacher on Their Own State Test

35. 1. Does ASSISTments work as an assessment tool? Reliably predict state test performance? Reliably advise teachers & students on what knowledge to focus on? 2. Does ASSISTments help enhance student learning? Does the System effectively teach as it assesses? Can teachers use the advice of this System to lead to higher student learning? Research Questions for US Dept of Ed

36. Report to a Teacher on Their State

37. Predict Student Knowledge as Measured by State Test Scores Feng, M., Beck, J,. Heffernan, N. & Koedinger, K. (2008)  Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test? In Baker & Beck (Eds.). Proceedings of the 1st International Conference on Education Data Mining. Montreal, Canada.  pp.107-116. ISBN - 13: 9780615306292  Feng, M., Heffernan, N. T., & Koedinger, K. R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. ISBN:1-59593-332-9.  pp. 307-316. [Nominated for Best Student Paper]   Feng, M., Heffernan, N.T., & Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal of User Modeling and User-Adapted Interaction.Vol 19: p243-266. • We have state scores from 2004 to 2009 for thousands of students. • We showed our error in predicting the state test is as low as the test itself. • We showed we can even predict their 10th grade math score from their 8th grade data. • We have shown we can more reliably predict their state test score if we use how much assistance they need.

38. Research Questions for US Dept of Ed 1. Does ASSISTments work as an assessment tool? • Reliably predict state test performance? • Reliably advise teachers & students on what knowledge to focus on? 2. Does ASSISTments help enhance student learning? • Does the System effectively teach as it assesses? • Can teachers use the advice of this System to lead to higher student learning?

39. We have shown that our transfer model is good enough to improve assessment. • Using finer-grained skill model can improve accuracy of the assessment of student knowledge. • Feng, M., Heffernan, N. T., Mani, M., & Heffernan, C. (2007). Assessing students’ performance longitudinally: Item difficulty parameter vs. skill learning tracking.  The National Council on Educational Measurement 2007 Annual Conference, Chicago. • Feng , M, Heffernan, N., Heffernan, C. & Mani, M. (2009) Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models. Journal of Technology, Learning, and Assessment. (Featured Article of the year!) • Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (2007). The effect of model granularity on student performance prediction using Bayesian networks.  The International User Modeling Conference 2007.

44. Research Questions for US Dept of Ed 1. Does ASSISTments work as an assessment tool? • Reliably predict state test performance? • Reliably advise teachers & students on what knowledge to focus on? 2. Does ASSISTments help enhance student learning? • Does the System effectively teach as it assesses? • Can teachers use the advice of this System to lead to higher student learning?

45. Can teachers use the advice of this system to lead to higher student learning? • Quasi Experiment • Results: Student in schools that used ASSISTments did better on state test than students in control schools. • Results: Gaps were closed in achievement • Recruitment for a large 60 school field trial is being organized. $25 million dollar grant “i3” Investing in Innovation Fund grant line • If you are interested talk to me.

46. Mastery Learning Stories of Formative Assessment Online Exit Card Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day. Book Homework Online

47. The Formative Assessment Cycle Objective Teacher Student Lesson Assessment

48. The Cycle can move through two or three iterations depending on the situation. Objective Lesson Objective Lesson Objective Assessment Lesson Assessment Assessment

49. Online Exit Card Objective Lesson Assessment • Find the side of a triangle using the Pythagorean Theorem • Use Connected Math Book • Three question Problem Set online on ASSISTment

50. Assessment • Three question Problem Set online on ASSISTment