Roy B. Clariana, Penn State University & Ray Bernardi, Wyoming Area School District

1. The effects of progress reports and goal orientation on effort and achievement of students using an online individualized instructional program Roy B. Clariana, Penn State University & Ray Bernardi, Wyoming Area School District

2. Overview • This investigation examined the effects of advisement on 8th grade students’ (n = 194) effort and achievement. Advisement consisted of a progress report sent to parents of their child’s work on online computer-delivered lessons. • We asked, “Does advisement interact with achievement goal orientation and gender to affect effort (the number of online lessons completed) and achievement (PSSA performance)?”

3. Achievement goal orientation • Achievement Goal Theory (Pintrich & Schunk, 1996) proposes two broad classes of achievement goal orientations, performance or mastery (either approach or avoidance). • Learners with a performance goal orientation are ego-centered and need to validate or prove their ability through open performance. They may avoid difficult tasks when failure is possible. • In contrast learners with a mastery goal orientation seek to increase their knowledge and ability-- learning for the sake of learning.

4. Achievement goal orientation • Achievement goal orientation may relate to personal epistemological belief systems (Schommer-Aikins & Easter, 2006) and thus may be relatively stable. • It may be a state and/or a trait; but the learning context likely affects immediate achievement goal orientation. • Also, achievement goal orientation may shift over time based on experiences in the learning context (i.e., consider the likely effects of NCLB on achievement goal orientation).

5. Advisement • Advisement is not praise (Henderlong & Lepper, 2002). • Advisement is information that relates to required, actual, or anticipated lesson performance, and according to Smith (1988), addresses questions such as, "How am I doing in this lesson?" • Progress reports are one very common form of advisement (e.g., report cards, or a detailed report from the teacher showing every graded test and assignment within a date range). • Advisement is assumed to support effective instructional decision making, thus it directly relates to learners’ achievement goals.

6. Advisement in computer-mediated learning contexts • Milheim and Azbell (1988) report that advisement in computer-mediated settings improves posttest scores relative to no-advisement groups; increases the number of students that reach mastery; increases time on task; and thus increases instructional efficiency. • Clariana (1993) considered the effects of advisement on effort and achievement of at-risk 11th grade students working online. Progress reports were given to students at the end of their 1st and 3rd sessions (max. 5 sessions). • Analysis of the attendance data showed a significant difference between the advisement (M = 4.33 out of 5) and no-advisement (M = 3.44 out of 5) groups. • Analysis of the achievement data indicated that the advisement group did a little better (nsd) than the no- advisement group and advisement had a slightly larger effect on females’ mathematics achievement relative to males (p = .08). Similar gender differences for achievement goal orientation have also been observed (Dunn & Shapiro, 1999).

7. Voluntary homework • Providing students with structured learning opportunities outside school is a way to extend the school day. • The online lessons used in this investigation run in a browser delivered from a server on the internet. This means that the online lessons are available not only in school but anywhere that students can access the internet. • Which students (if any) will choose to do the online lessons as voluntary ‘homework’?

8. Participants • 8th grade students (total N = 218) in the Wyoming Area School District in northeastern Pennsylvania • 17 excluded because they did not participate in the online lessons. • 7 excluded because they did not have 2006 PSSA data. • Thus the final sample used in the data analysis consisted of n = 194 students. • The students attended classes in a facility that houses all 7th through 12th grade students (facility N = 1,315). • In this facility, 98% of students are Caucasian and 24% are on free or reduced lunch.

9. Setting and Treatment • Online IPs – CompassLearning lessons delivered over the internet that are aligned to state standards and examination anchors in mathematics and reading, and are unique for each student based on past performance measures. • Participants were randomly assigned to the Advisement or No-Advisement treatment that consisted of a progress report sent to parents in Week Three (in addition, gender and achievement goal orientation weretwo other independent variables).

10. Effort and achievement dependent measures • Effort – Number of online lessons completed • Achievement – PSSA Reading and Math scaled scores • Other descriptive data • homework completion data • Self-report survey measuring achievement goal orientation and internet access.

11. Achievement goal orientation survey • A ten item self-report survey to measure achievement goal orientation (from Hannah’s dissertation) • even numbered items – performance goal • odd numbered items – mastery goal • The self-report survey was completed by 164 students (84 females and 80 males), 30 students (14 females and 16 males) did not take the survey • Breakdown: • performance oriented (43%; n = 71 with 33 females and 38 males) • mastery oriented (41%; n = 67 with 32 females and 35 males) • neutral achievement goal orientation (15%; n = 26 with 19 females and 7 males). See the survey in Appendix A

12. mastery performance Achievement goal orientation analysis • Principalcomponents analysis with varimax rotation obtained two factors. • The ten items aligned as they should into mastery and performance categories. • Cronbach alpha for the 10-item achievement goal orientation: • All 10, α = 0.866 • Odd numbered (m), α = 0.789 • Even numbered (p), α = 0.837 coefficients sorted by size

13. Timeline of milestones… Changed to the Week of Mar 31 Week of Mar 13 Week of Dec 19 Student and teacher interviews Student self-report survey Letter to 8th grade parents Mar 31 Mar 21 Jan 9 Mar 17 August makeup Students work on online IPs PSSA Jan 15 Feb 15 Mar 15 PSSA data available Jan 27 Mar 20 Mail online IP progress report letter to parents of randomly selected group Print all final progress reports

14. Achievement: PSSA results 85% math 93% reading

15. PSSA Analysis The 2006 reading and mathematics PSSA scaled score data were analyzed by a 3-between x 2-within mixed ANOVA. • The between-subjects factors were Treatment (report or no report), Gender (female or male), and Achievement Goal (mastery, neutral, performance, and unknown). • The within-subjects repeated measure was Content (reading and mathematics).

16. Mixed ANOVA

17. PSSA: Content x Gender interaction reading math

18. PSSA: Content x Treatment x Gender interaction

19. Effort: Analysis of the number of lessons completed • The CompassLearning online IP total lessons completed data (combined reading and mathematics) were analyzed by a three factor ANOVA that included: Treatment (report or no report), Gender (female or male), and Achievement Goal (mastery, neutral, performance, and unknown). • Results: None of the main factors or interactions were significant except for the 3-way interaction of Treatment, Gender, and Achievement Goal (T * G * A), F(3,178) = 2.680, MSE = 394.182, p = .048

20. Lessons completed: Treatment x Gender x Achievement Goal interaction note (possible ceiling effect)

21. Did they do Homework? • The online lessons were not checked by teachers, and no daily grade was given for completion, so there was little or no pressure to do the online lessons outside school. • 5.7% of the students worked on the online IPs outside of school time. Most were females (9 females and 2 males). • ANOVA indicates that there was apparently an interaction (see next slide).

22. Possible interaction of Treatment x Gender for minutes of homework completed female 15 10 Homework minutes 5 male 0 no report report treatment Caution, small sample n=11

23. Findings In general terms, sending advisement home to parents … • relates to increased achievement for males in reading and math, for females in math, but a decrease (sig.) for females in PSSA reading scores. • did not positively influence females to do more online IP lessons. In fact, the presence of reports seemed to depress females’ online IP lesson completion (especially those with a performance achievement goal orientation). This is the opposite of the trend observed by Clariana (1993) that females may benefit more than males from progress reports.

24. Regarding praise • Deci (1975) proposed that males tend to be more sensitive to the informational aspect of praise and so perceive praise as an informational message that they are competent in the relevant task. This tends to enhance their intrinsic motivation for the task. • In stark contrast, he proposed that females tend to be more interpersonally sensitive and concerned with others’ evaluations of them and so perceive praise as a controlling intervention that diminishes their autonomy, thus decreasing their intrinsic motivation for the task. Also Katz, Assor, Kanat-Maymon, and Bereby-Meyer (2006) extended Deci’s hypothesis to include the effects of gender differences in reaction to advisement.

25. Possible explanations… • Katz et al. (2006) In 7th grade students with a moderate task interest, advisement had a positive effect on males’ and a negative effect on females’ self-report of intrinsic motivation. • Katz’s et al. findings are in line with the results of the present investigation for females with a performance orientation (see Figure on slide 20). • Similar to Deci’s hypothesis that females tend to be more interpersonally sensitive and concerned with others’ evaluations of them, performance-oriented females in the present investigation perhaps resented advisement as an attempt to control or manipulate their actions (which, of course, was exactly the case).

26. What would you do next time? • If you are the principal? • If you are the teacher? • If you are the investigator? • What effect did the age of the students have on the study?

27. Appendix A: Achievement Goal Orientation self-report items

28. Appendix B: Correlation table