A Longitudinal Study of Student Skill Development

1. A Longitudinal Study of Student Skill Development Kathy Ray, University of the Pacific RUSA/MOUSS 10th Annual Reference Research Forum, June 2004

2. Why a Qualitative Longitudinal Study? • Libraries are expected to show evidence of their impact on student learning as a quality indicator within their institutions • Many libraries’ assessment studies measure the direct impact of instructional efforts; many are quantitative approaches • Few studies have tracked individual students over time to examine the development of research behaviors and skills; few are in-depth, qualitative approaches

3. Why a Qualitative Longitudinal Study?contd. • Explore student perspectives on research; see the process, the library through student eyes • Gain insight into student learning styles and other aspects of development • Explore implications for design of library instruction and other library services

4. Use of Cohort and Longitudinal Studies • Extensive use of cohort studies in human development and sociological research • Few cohort studies in library literature • Library-related longitudinal studies compare data sets but don’t follow individuals over time • Cohort studies are time consuming and require commitment over a multi-year period

5. Cohort Study at University of the Pacific • The study originated as part of the Library’s assessment plan, in an effort to answer this question: Do Pacific’s students have the desired level of information competency skills at graduation? • Information competency skill development is distributed across general education courses and is not centrally defined or assessed • Based on the outcomes of the study, should the university’s General Education program be revised to include specific information literacy goals?

6. Research Questions • Do students have the desired information competency skills at graduation? • How do they develop their skills? • What factors and experiences shape this development? • How can student behaviors and development be explained (using learning, development theories)? • What conclusions can be reached about the design of library instruction?

7. Research Design / Methodology • Qualitative: researcher as participant • Phenomenological approach: how do students experience the phenomenon of research, library use? • Population and sample: 19 student volunteers from a core GE course enrolling 700+ freshmen • Longitudinal data gathering: • structured interviews, recorded and transcribed; field notes; student papers (portfolio analysis); final competency quiz • Confidentiality of participants maintained • Coding of data and analysis • Pay!

8. Some Observations on the Research Design • Issues of validity: how students want to be perceived by librarians (saying the “right” thing) • End of semester fatigue level apparent in some interviews • We have a limited view of the great variety of factors that influence students • We must avoid generalization • Enables librarian to conceptualize instruction at the level of the individual

9. Coding and Analysis • An initial reading of the transcripts suggests categories, themes around which to organize data • In a second reading, student responses and statements are coded according to the agreed upon set of themes • Each student has his or her own unique experience • Data cannot be generalized to describe the “average student” • Themes help the researcher find both commonality and diversity of experience and begin to suggest meaning and interpretations of the experience

10. Why and how students pick research topics Where and how they look for information Research techniques: Innate vs.taught Factors affecting skill development Inputs – Student background characteristics Environment – Teachers; the curriculum; other students, friends; Library use, library instruction Learning styles Maturation (cognitive, ego, social, moral) Interview Questions/Themes

11. 19 students 11 female 8 male 18 first year students, 1 soph. Residence halls or Greek houses GPAs 2.0 to 3.8 Cohort Demographics

12. 2001 Business 3 Education 1 Engineering 4 Exploratory 5 Music 1 Pre-Pharmacy 2 Pre-Med 1 Psychology 1 Religious Studies 1 Sports Sciences English Sociology Graphic Design Communication Majors 20022003 5 4 1 1 2 2 1 0 1 1 11 11 1 1 1 1 1 1 1 1

13. Where and how they were taught to look for information Other databases Specialized encyclopedias Books E.A.I.* Web Assignment NoFT print journals FT read read copy *Expanded Academic Index print/email

14. Where and how they look for information Books WEB E.AE.A.I.*.I.* Other databases Assignment print out No FT copy & paste FT read print journal e-mail *Expanded Academic Index

15. Why do you go straight to the Internet when starting a research project? I definitely didn’t have time to go look for books or anything…I just needed to be right there on my computer, get it printed out, read it, write the paper. It’s just most convenient for me and usually I can find a lot of information. I’ve done papers where I’ve used only Internet sites and I’ve gotten decent grades…

16. Why do you go straight to the Internet when starting a research project? On the Internet, it pretty much gets straight to the point. If I were to go to the library and try to find a book on Roman marriages, then I have to read a few chapters…its so much easier just to read a few lines, jump to the next paragraph, get an idea of what they’re talking about…

17. Where is the first place you look for information? Well, actually, I like to check out books just because I know it’s the easiest thing: go in here, look at the library catalog and find your books. Once you go through them you can go online and look for keywords that were in the books to help you specify what you’re looking for. I find books helpful because you actually have something in your hand at 2:00 am. Sometimes Internet searches go bad.

18. Preliminary Findings 1st & 2nd year: Where and How They Look for Information • Web search - Yahoo, MSN, AOL, “whatever” • Continue habits from high school (use of the Internet) • Easy, convenient, “can always find something” • Expanded Academic Index • Taught how to use in library workshop; teachers encouraged use • Credible, scholarly articles; most full-text • Other databases “frustrating” and “useless” • Books avoided by most • Past papers; “things my friends or teacher gives me” • Barnes & Noble –innovative research strategy

19. Preliminary Findings 1st & 2nd year: Research Techniques • Keywords in basic search mode; avoidance of advanced search features • Some sophistication in narrowing a search • Start at top of list and work down • Follow links from one website to another • Ask Jeeves (natural language) • Ask for help: friends, professor, librarian • Browsing shelves (journals or books)

20. Preliminary Findings : Varying Methods of Making Judgments Regarding Sources • High preference for credible full-text articles • “Just Right” articles – not too brief, not too long, not too difficult to understand • Look at titles, abstracts, some skim article itself • Find enough to fulfill assignment requirements • No need to be comprehensive or look for the “best” articles • Use papers written by other students • Models for approximate length and complexity (and content?) • Look for credible, reliable sources • But judgment made on simple and usually superficial basis

21. Scholarly Journals I had a hard time sometimes. [The articles] wouldn’t mention if they were scholarly or not so I basically had to ask my professor. Most of the articles used bigger words than I do so they all seemed pretty scholarly to me… I figured if I had heard of the magazine, it wasn’t scholarly.

22. Conflicting Information A: One article said that emissions from hybrid cars were really low and another article said that the emissions were really high. Q: So what did you do? A: We used the one that supported our side. Q: Did you feel it was a better argument? Did you look at the other one and worry that it might be correct? A: Yeah, well, not really...

23. Preliminary Findings for 1st & 2nd year students:Influential Factors Affecting Skill Development • Habits developed in high school • Experience using the Web on his/her own • Use of high school or public libraries • Strategies & techniques learned by associating with other students • Library workshops where a database is demonstrated (associated with certain types of projects or topics) • Assignments; directions from teachers (although greatly varying and sometimes unsound) • Major course of study (sometimes)

24. Preliminary Findings:Not-Very-Influential Factors Affecting Skill Development • Curriculum • Few substantive research papers or assignments requiring use of scholarly resources • Use of the library home page • Library workshops • Reference librarians

25. Preliminary Conclusions & Implications for Library Instruction 1st & 2nd year students... • need to be introduced to a greater variety of sources more than they need to be taught search techniques • Some need instruction in specialized databases • Most use limit features to find scholarly or refereed sources and full-text articles • want a well-integrated research environment (SFX, etc) • need a basic orientation to where things are located • are not heavily invested in their research in the 2 years • But when a topic piques their interest and they master a body of material, intellectual growth is clearly evident.

26. Preliminary Conclusions & Implications for Library Instruction 1st & 2nd year students... • have a casual understanding of the ethics of use of information What I usually do is just minimize them (articles) so I can keep them right there. I’ll find a paragraph or a phrase that I want to incorporate into my paper and I’ll cut and paste – I’ll go to Word and then I’ll just have a whole page of different quotes that I’ll use. When I start writing my paper…I’ll cut and paste where I want them to go and then rewrite it. I’ll take parts of it and paste it into another document… I think it can be confusing because you’re just cutting and pasting and then you have to really decide what is plagiarism and what is not.

27. 3rd year students’ research strategies… • Think! • Books • Library databases • Internet • Talk to others • Original research

28. Preliminary Findings in 3rd year: Influential Factors Affecting Skill Development • Major course of study • Closer relationships with professors & higher expectations • More engagement in assignments/projects; more printing & highlighting, less cut-n-paste • More sophisticated understanding of various types of information • Books for overview, articles for specialized info, web for images • But sources needed to provide “evidence and stuff” • “Research paper” not a universal experience • Primary research, presentations, group projects

29. Preliminary Findings in 3rd year: Influential Factors Affecting Skill Development • Internet sufficient for many assignments • Often an opinion piece or analysis; outside info not required • Confident about ability to discern credible web sites • “I look for a logo or something professional” • Ethics well understood… • But time pressures cause them to rationalize short-cuts • Appeal of Barnes and Noble • All students rank higher than library

30. Barnes & Noble Strategy 2nd year student I got two of my best references from Barnes & Noble. I wasn’t able to get any books here because they were all checked out. So I borrowed money from my Dad and went and bought these books, and they were really, really helpful. The majority of this paper is from those books. I bought them and then I took them back. But don’t tell Barnes & Noble!

31. Barnes & Noble 3rd year student I usually go to bookstores and they have their own online catalog and at customer serviced they’ll say, we can order a book for you and if you like it you can buy it but you don’t have to, and usually I won’t but I can look at it and sit there and drink coffee and stuff… I’ll ask for a list of authors on nano technology and they print out a whole list of all these different authors. Once I have the names, I go to the library and then I have these authors to type in.

32. Case Studies3rd year students • Nora • English major, Film studies minor • Moki • Computer Engineering/Engineering Mngmt, double major; Physics minor • Shura • Sociology major, criminal justice emphasis

33. Final Goals • Conduct portfolio analysis based on works collected over four years • Compare student skill set in 4th year to desired competency level • Relate observations on student development to development and library literature • Disseminate results to campus faculty and profession • Reconsider approaches to group and individualized instruction