Magia G. Krause Ph.D. Candidate School of Information University of Michigan

1. Undergraduates in the Archives: Measuring the Impact of Archival Instruction Through Rubric-based Assessment Magia G. Krause Ph.D. Candidate School of Information University of Michigan Society of American Archivists Annual Meeting August 14, 2009

2. Instruction and Assessment in Academic Archives Krause, Magia G.(2008). Learning in the archives: A report on instructional practices, Journal of Archival Organization, 6(4) 233-268.

3. Overview of Teaching Experiment • Research Questions • Do undergraduate students learn from archival instruction? • What do undergraduate students learn from archival instruction? • Goals of study • Introduction of a reliable assessment tool to measure student learning • Introduction of archival literacy scale that might compliment a curriculum of undergraduate research education in archives

4. Methodology • Quasi-experiment in the field • Background • US history course in a large state university • 82 students total • Control and treatment groups • To compare students that did not receive archival instruction with students that did • Pre- and Post- Document Analysis Exercise • To provide a baseline comparison of the groups and look for an improvement in treatment group

5. Archival Instruction • Description of treatment (aka archival instruction) • Lecture/presentation to entire class by archivist • Hour-long session in the archives with 4 stations • Bibliographic Instruction • Critical Thinking • Photograph Analysis • Footnote/Citation Analysis

6. Photographs from the Instruction Session

7. Document Analysis Exercise • Divided into 3 sections • Adapted from NARA’s Teaching with Documents • Written Document Analysis • Photograph Analysis • Finding Aid Analysis • Difference between Pre- and Post- Tests • Alternative primary sources and finding aid

8. Assessing Student Learning • Rubric • Basic definition • Scoring guide that includes criteria and levels of performance • Types of rubrics • Holistic and analytic • Steps in creating a rubric • Criteria : Archival Literacy Skills • Observation • Interpretation/Historical Context • Evaluation/Critical Thinking • Research Skills

9. Rubric for this study

10. Using the Rubric • 3 raters • Two professional archivists with at least five years of experience teaching undergraduates and me • Important for increasing the reliability of the scores • Achieving kappa • Final results • Our Fleiss Kappa for the pretest was .80218 and .788 for the post-test, both of which are considered excellent strengths of agreement.

11. Statistical Tests • Non-parametric tests • Useful for analyzing ordinal and nominal data • Great for small samples • Tests used: • Mann Whitney U Test for Independent Samples • Results • The tests confirmed that the null hypothesis could be rejected with 95% confidence • Students in the treatment group did better on the post-test after participating in archival instruction. In fact, there is a statistically significant difference between the two groups’ scores for each of the criteria.

14. Discussion • The Research Skills Problem • Disappointing results • Possible factors • Students not actively engaged during instruction • Follow-up assignment involved pre-selected documents – no chance to practice searching for and identifying documents • Literature on undergraduate information-seeking behavior

15. Archival Literacy Scale • Elements • 4 categories of the rubric: • Observation • Interpretation • Evaluation • Research Skills • Testing for reliability • Results

16. Conclusions • Limitations to the generalizability of results – • Need to replicate the study to increase the reliability of the instruments • Wish for better instructional design for instruction component • Need for more communication and collaboration between archivists to develop and share curricula and standardized tools to assist in instruction and assessment

17. Thank You • For more information about the study described in this presentation, please contact: Magia Krause mghetu@umich.edu.