270 likes | 386 Views
Analyzing Data, Getting Results Some practical, not-too-burdensome tips and tricks. Jenn Riley University of North Carolina at Chapel Hill. Evidence-driven decisions are a powerful guide for library operations. “The plural of anecdote is not data.”. Sometimes attributed to Frank Kotsonis .
E N D
Analyzing Data, Getting ResultsSome practical, not-too-burdensome tips and tricks Jenn Riley University of North Carolina at Chapel Hill
Evidence-driven decisions are a powerful guide for library operations. SCELC Research Day
“The plural of anecdote is not data.” SCELC Research Day Sometimes attributed to Frank Kotsonis. After a quote with the opposite meaning, by Raymond Wolfiger.
“There are three kinds of lies – lies, damned lies, and statistics.” SCELC Research Day Mark Twain, perhaps after Benjamin Disraeli.
Using data for planning library operations Designing web sites and other online resources Existence/hours of service points Materials to buy/license/accept/digitize/keep/preserve Effectiveness of/satisfaction with procedures/services SCELC Research Day Evaluating a pilot service or project Projecting future expenditures
Both cost and value are key ALCTS Heads of Technical Services in Large Research Libraries Interest Group, Task Force on Cost/Value Assessment of Bibliographic Control (2010) Proposes definitions of value for cataloging: SCELC Research Day Discovery success Use Display understanding Data interoperability Support for FRBR user tasks Throughput/timeliness Support administrative goals
Example studies • By Joyce Chapman, then at North Carolina State University • Benefits of manually enhanced metadata for images • Comparing effort to utility for specific EAD elements SCELC Research Day See Chapman, Joyce. “Metrics & Management: Cost & value of metadata workflows.” SAA 2011. http://www.academia.edu/1708422/Return_on_Investment_Metadata_metrics_and_management
Some common analyses Predicting impact of a change Cost per unit produced Change over time SCELC Research Day Identifying unmet needs Error/problem rate
Back to library scenarios SCELC Research Day
Existence/hours of service points • Who is using what and when? • How can we most effectively staff them? • Costs • Staff time • Facilities management costs • Benefits • Number and type of visitors, and how they use it • Service transactions completed • Specific services used at the location • Other data to collect • Usage by time of day • Calculate cost per transaction SCELC Research Day
Materials to buy/license/accept/digitize/keep/preserve • Should we acquire, make more accessible, or keep this? • Costs • Initial purchase/license • Ongoing license/maintenance • Staff for cataloging/processing/digitizing/ingesting/preserving • Software • Hardware/storage • Benefits • Current and predicted future use • Opportunity for transformative use SCELC Research Day
Evaluating a pilot service or project • Is the cost/benefit ratio appropriate? • What is the raw cost? • But it’s not all about cost/benefit: • Is the pilot achieving its aims? • Does this [whatever] do what we thought it would? • What collateral effects will it have? • Were the assumptions we made correct? • Data collection will be varied for this task SCELC Research Day
Designing web sites and other online resources • A/B testing • User-centered design • Satisfaction surveys with previous iterations, similar sites, or prototypes • Web stats for previous iterations or similar sites • Task-based usability testing • Don’t forget the cost of sustaining it once you have it up! SCELC Research Day
Effectiveness of/satisfaction with procedures/services • What parts of our current service are users most and least happy about? • What are the ineffieciences in our procedure for [whatever]? • Some data collection ideas • User surveys • Ratio of potential to actual users • Ratio of returning to non-returning users • Error/failure rates • Time from request to delivery • Time tracking during staff activity SCELC Research Day
Projecting future expenditures • Equipment • Define its lifecycle • Amortize purchase cost • Add in maintenance costs • Compare to use as context • Staff • Educated guess at raises, turnover, benefit costs changes • Consider: • Inflation • Past trends • Upcoming sea changes SCELC Research Day
Strategies for getting data that can be analyzed SCELC Research Day
Tracking use • Circulation • COUNTER/SUSHI • Physical visitors • Web hits • Social media engagement • Attendance at events/sessions SCELC Research Day
Tracking time • Can be effective when collected as a representative snapshot • Options for data collection • Clipboard next to a clock • Spreadsheet • Free time tracking apps • Make it as simple as possible SCELC Research Day
Calculating costs • Staff time • 2080 hours per year is full time • Standard benefit percentages • Materials (including software) • Initial purchase • Maintenance contracts for big-ticket items • Amortize big costs over time in service • Overhead • Universities typically have standard rates SCELC Research Day
Calculating error rates • Both objective and subjective criteria • Typically best when done as a sample • Consider both automated and manual means to locate errors for study SCELC Research Day
Categorization • Putting things into like groups • Compare size of groups to one another • Compare effort spent on one group to another • Compare priority/value of one group to another • Can be done at time of data collection, or afterwards • Good idea to have some sense of categories at the beginning of the study SCELC Research Day
Calculating benefit • Change in knowledge or status • Over time • After an interaction • Survey – ask about knowledge level before and after • Pre- and post-tests • Indirect measures • Number of people reached • Use SCELC Research Day
Additional data analysis strategies SCELC Research Day
Mechanics • Code qualitative data to make it processable • Make sure you pick a representative and consistent sample • Extrapolate based on known data when you need to • ALWAYS do a sanity check • Spreadsheets are your friend SCELC Research Day
More advice • Context is key • Don’t be paralyzed by a perceived need for perfection • Know your basic analysis plans before you collect/identify data • Utilize pilot projects to generate data where there is none • Use the right tool for the job • Document your assumptions • It’s OK to use “napkin math” SCELC Research Day
Get in the habit of collecting data.It will make your next decision easier. SCELC Research Day
Thank you!Questions and discussion SCELC Research Day jennriley@unc.edu