1 / 21

Cluster 5 Spring 2005 Assessment Results

Cluster 5 Spring 2005 Assessment Results. Sociocultural Domain. Cluster 5 Course Requirement. Complete 1 course in each domain. Assessment Day Data Collection Scheme: Pre/Post.

gaston
Download Presentation

Cluster 5 Spring 2005 Assessment Results

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cluster 5 Spring 2005 Assessment Results Sociocultural Domain

  2. Cluster 5 Course Requirement • Complete 1 course in each domain

  3. Assessment Day Data Collection Scheme: Pre/Post Students in each cohort are tested twice on the same instrument – once as incoming freshmen (0 credit hours) and again in after they’ve completed 45-70 credit hours. COHORT 1 COHORT 2 Cohort 2: First time for Cluster 5 to have pre/post data since new objectives/instruments. COHORT 3

  4. Sociocultural Domain Instruments • Sociocultural Domain Assessment (SDA) • 3 different versions of test • Scenario-based multiple-choice exam created by faculty Administration error led to SDA1 & SDA2 being administered in Spring 2005 – because of this only 27 common items on both forms used to create total score

  5. Results • First: examine results for 398 2nd semester sophomores in Spring 2005 • Overall results • By JMU vs. Transfer/AP • By domain “completers” vs. “non-completers” • By which course completed • Rel. between course grade and SDA score • Second: examine pre/post results for the 264 students who took the SDA in both Fall 2003 and Spring 2005 • Overall pre/post difference • pre/post difference by domain “completers” vs. “non-completers” • pre/post difference by which course completed • Pre/post difference by item for “completers”

  6. Overall SDA Results: 2nd Semester Sophomores (N = 398) On average, students are answering almost 70% of the items correctly. Most scores are between 17 & 23.

  7. JMU (n = 369) vs.Transfer/AP Students (n = 29) • Are students who obtained transfer or AP credit for their courses scoring differently than students who completed their courses at JMU? • No. • JMU students: M = 18.63, SD = 3.83 • AP/transfer students: M = 18.55, SD = 3.68 • Difference was not statistically, t(335)=.11, p = .92, nor practically significant (d = .02).

  8. Domain “Completers” vs. “Non-Completers” More courses completed, the higher the mean. Differences among the means, however, are of small practical significance (d’s between .08 - .25)

  9. SDA Means by Course Completed“Completers” Only There were minor differences among the SDA means (d’s ~.01 - .24) GEIC101 not reported due to small N

  10. Relationship between SDA and Course Grade Positive and moderate relationships…as anticipated. r = .31 r = .23 r = .69 r = .36 w/o outlier

  11. Pre/Post Results (N = 264) Average increase of 8% in sociocultural domain knowledge • Spring 2005: M = 18.78, SD = 3.66 • Fall 2003: M = 16.62, SD = 3.38 • The difference between these means is of medium to large practical significance (d = .64)

  12. Pre/Post for “Non-Completers” (n = 33) vs. “Completers” (n = 199) Non-completers Completers “Completers” start off with higher means than “Non-completers” – but they both make similar gains over time (~2 pt. increase, d ~.55) Larger gain (~3 pts, d = 1.02) for the 32 students who completed 2 – 3 sociocultural domain courses.

  13. Pre/Post by Sociocultural Course Completed The magnitude of the change in knowledge is very similar for the three different courses (d ~ .75). N = 153 N = 98 N = 10 Spring 2005 average for GSOCI240 students may seem superior, keep in mind that these students were more knowledgeable coming into the course and that the averages for GSOCI240 are based on only 10 students.

  14. Pre/Post Difference in Items for “Completers” 7 items in Fall 2003 where less than half of the students are obtained the correct answer; there are only 2 such items in Spring 2005 Most of the gains are made for items of moderate to large difficulty.

  15. Items with Large Pre/Post Gains 7% or more students answered these items correctly in Spring 2005 than in Fall 2003 Majority of these items (67%) were written to assess goals 2 and 3. Goal 2:Identify implications of taking action regarding social/behavioral issue Goal 3:Use evidence to develop and evaluate positions regarding social/behavioral issue

  16. Items with Small (or No) Pre/Post Gains Items with no change, or change in the wrong direction are shown below for items that are of medium to extreme difficulty. 67% of these items are assessing either goals 1 and 4, which indicates that: 1) either more focus should be placed these goals in sociocultural domain courses or 2) that these items are of poor quality and either need to be revised or replaced Goal 1: Make plausible interpretations about behavior in social contexts Goal 4: Discriminate between ethical and nonethical practices in the social/behavioral sciences

  17. Conclusions2nd Semester Sophomore Data • 2nd semester sophomores scoring on average ~70% • AP/Transfer students are scoring no differently than JMU students • “Non-completers” of the sociocultural domain requirement are scoring no differently than “Completers” • Same average score regardless of what course student took to complete sociocultural domain requirement • There is a relationship between course grade and score, indicating that test content and course content are related

  18. ConclusionsPre/Post Data(1) • Average increase of 8% in sociocultural domain knowledge between 0 credit hours and 45-70 credit hours (large effect) • 62%  70% • “Completers” and “Non-completers” both make similar gains over time (~2 pt. increase, d ~.55) • Might be evidence that score gains are a result of maturation, not a result of taking sociocultural course • Note that “completers” come in with more knowledge. • Larger gains (~3 pts, d = 1.02)are seen for students completing 2 – 3 courses. • May need to collect larger sample of “non-completers” in future assessments

  19. ConclusionsPre/Post Data(2) • Same gains are being made on the test for students who fulfilled their requirement by taking different sociocultural courses • Students who take SOCI240 may be more knowledge coming into the course than students who choose to take other courses • 7 items where <50% of students obtaining correct response in Fall 2003 compared to 2 in Spring 2005 • Not seeing gains on easy items • Most gains are for items assessing Goals 2 & 3 • No gains (or gains in the wrong direction) for items assessing Goals 1 & 4

  20. Next Steps • Take a closer look at change in % of students correctly answering items over time • Use information to make inferences about program OR possibly to revise items • Set standard score on test in order to obtain richer information

  21. Setting Standards • May want to use standard setting procedures to establish “cut-offs” for proficiency on the test • Could be that students are gaining knowledge/skills over time, but are they gaining enough?

More Related