Analyzing Data, Getting Results Some practical, not-too-burdensome tips and tricks

1. Analyzing Data, Getting ResultsSome practical, not-too-burdensome tips and tricks Jenn Riley University of North Carolina at Chapel Hill

2. Evidence-driven decisions are a powerful guide for library operations. SCELC Research Day

3. “The plural of anecdote is not data.” SCELC Research Day Sometimes attributed to Frank Kotsonis. After a quote with the opposite meaning, by Raymond Wolfiger.

4. “There are three kinds of lies – lies, damned lies, and statistics.” SCELC Research Day Mark Twain, perhaps after Benjamin Disraeli.

5. Using data for planning library operations Designing web sites and other online resources Existence/hours of service points Materials to buy/license/accept/digitize/keep/preserve Effectiveness of/satisfaction with procedures/services SCELC Research Day Evaluating a pilot service or project Projecting future expenditures

6. Both cost and value are key ALCTS Heads of Technical Services in Large Research Libraries Interest Group, Task Force on Cost/Value Assessment of Bibliographic Control (2010) Proposes definitions of value for cataloging: SCELC Research Day Discovery success Use Display understanding Data interoperability Support for FRBR user tasks Throughput/timeliness Support administrative goals

7. Example studies • By Joyce Chapman, then at North Carolina State University • Benefits of manually enhanced metadata for images • Comparing effort to utility for specific EAD elements SCELC Research Day See Chapman, Joyce. “Metrics & Management: Cost & value of metadata workflows.” SAA 2011. http://www.academia.edu/1708422/Return_on_Investment_Metadata_metrics_and_management

8. Some common analyses Predicting impact of a change Cost per unit produced Change over time SCELC Research Day Identifying unmet needs Error/problem rate

9. Back to library scenarios SCELC Research Day

10. Existence/hours of service points • Who is using what and when? • How can we most effectively staff them? • Costs • Staff time • Facilities management costs • Benefits • Number and type of visitors, and how they use it • Service transactions completed • Specific services used at the location • Other data to collect • Usage by time of day • Calculate cost per transaction SCELC Research Day

11. Materials to buy/license/accept/digitize/keep/preserve • Should we acquire, make more accessible, or keep this? • Costs • Initial purchase/license • Ongoing license/maintenance • Staff for cataloging/processing/digitizing/ingesting/preserving • Software • Hardware/storage • Benefits • Current and predicted future use • Opportunity for transformative use SCELC Research Day

12. Evaluating a pilot service or project • Is the cost/benefit ratio appropriate? • What is the raw cost? • But it’s not all about cost/benefit: • Is the pilot achieving its aims? • Does this [whatever] do what we thought it would? • What collateral effects will it have? • Were the assumptions we made correct? • Data collection will be varied for this task SCELC Research Day

13. Designing web sites and other online resources • A/B testing • User-centered design • Satisfaction surveys with previous iterations, similar sites, or prototypes • Web stats for previous iterations or similar sites • Task-based usability testing • Don’t forget the cost of sustaining it once you have it up! SCELC Research Day

14. Effectiveness of/satisfaction with procedures/services • What parts of our current service are users most and least happy about? • What are the ineffieciences in our procedure for [whatever]? • Some data collection ideas • User surveys • Ratio of potential to actual users • Ratio of returning to non-returning users • Error/failure rates • Time from request to delivery • Time tracking during staff activity SCELC Research Day

15. Projecting future expenditures • Equipment • Define its lifecycle • Amortize purchase cost • Add in maintenance costs • Compare to use as context • Staff • Educated guess at raises, turnover, benefit costs changes • Consider: • Inflation • Past trends • Upcoming sea changes SCELC Research Day

16. Strategies for getting data that can be analyzed SCELC Research Day

17. Tracking use • Circulation • COUNTER/SUSHI • Physical visitors • Web hits • Social media engagement • Attendance at events/sessions SCELC Research Day

18. Tracking time • Can be effective when collected as a representative snapshot • Options for data collection • Clipboard next to a clock • Spreadsheet • Free time tracking apps • Make it as simple as possible SCELC Research Day

19. Calculating costs • Staff time • 2080 hours per year is full time • Standard benefit percentages • Materials (including software) • Initial purchase • Maintenance contracts for big-ticket items • Amortize big costs over time in service • Overhead • Universities typically have standard rates SCELC Research Day

20. Calculating error rates • Both objective and subjective criteria • Typically best when done as a sample • Consider both automated and manual means to locate errors for study SCELC Research Day

21. Categorization • Putting things into like groups • Compare size of groups to one another • Compare effort spent on one group to another • Compare priority/value of one group to another • Can be done at time of data collection, or afterwards • Good idea to have some sense of categories at the beginning of the study SCELC Research Day

22. Calculating benefit • Change in knowledge or status • Over time • After an interaction • Survey – ask about knowledge level before and after • Pre- and post-tests • Indirect measures • Number of people reached • Use SCELC Research Day

23. Additional data analysis strategies SCELC Research Day

24. Mechanics • Code qualitative data to make it processable • Make sure you pick a representative and consistent sample • Extrapolate based on known data when you need to • ALWAYS do a sanity check • Spreadsheets are your friend SCELC Research Day

25. More advice • Context is key • Don’t be paralyzed by a perceived need for perfection • Know your basic analysis plans before you collect/identify data • Utilize pilot projects to generate data where there is none • Use the right tool for the job • Document your assumptions • It’s OK to use “napkin math” SCELC Research Day

26. Get in the habit of collecting data.It will make your next decision easier. SCELC Research Day

27. Thank you!Questions and discussion SCELC Research Day jennriley@unc.edu