An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas

1. 20 November 2013 An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas Prepared for the ULS Leadership Program by Luke Ferdinand, John Fudrow, Karen Calhoun and Jeff Wisniewski This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

3. Needs Assessment Defining and Addressing The Needs of an Organization and Its Audience Luke Ferdinand ULS Leadership Program 20 November 2013

4. Objectives • Define Needs in Organization • Define Needs Assessment • Understand the methods and tools used in Needs Assessment • Know potential risks in Needs Assessment methods ULS Research Methods Workshop

5. What is a Needs Assessment? "A systematic process for collecting information and making justifiable decisions" - Ryan Watkins Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naintroduction.html, ULS Research Methods Workshop

6. What is a need? • “Needs are the differences between your current achievements and your desired accomplishments.” Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naintroduction.html, ULS Research Methods Workshop

7. Needs Assessments Help Identify • Data that define your needs • Prioritization of your needs • Criteria for implementing solutions • Information necessary to justify selection of one or more activities to improve performance Ryan Watkins, 2013, “Needs = Gaps in Results”,http://ryanrwatkins.com/na/naessentials.html ULS Research Methods Workshop

8. Why? • “Provide a systematic process to guide decision-making in organizations • Provide justification for decisions before they are made. • Scalable for any size project, time-frame, or budget. • Offer a replicable model that can be applied by novices or experts. • Provide a systemic perspective for decision-makers. • Allow for interdisciplinary solutions for complex problems. “ Ryan Watkins, 2013, Needs = Gaps in Results”, http://ryanrwatkins.com/na/naessentials.html, ULS Research Methods Workshop

9. Where can we use a needs assessment? Strategic Action 9a : Ground Floor Needs Assessment • (obviously) • Opportunity to rethink the ground floor • Many assumptions and ideas for what's best • Gather information from stakeholders, relevant resources • Make an informed decision to best meet the needs of organization and community • Other opportunities? ULS Research Methods Workshop

10. 12 Steps Brought to you by the National Oceanic & Atmospheric Administration http://csc.noaa.gov/needs/ ULS Research Methods Workshop

11. Needs Assessment Steps • Confirm the Issues & Audience • Establish the Planning Team • Establish the Goals and Objectives • Characterize Your Audience • Conduct Information & Literature Search • Select Your Data Collection methods • Determine Your Sampling Scheme • Design and Pilot the Collection Instrument • Gather & Report Data • Analyze Data • Manage Data • Synthesize Data & Create Report ULS Research Methods Workshop

12. 1: Confirm Issues & Audience • Summary: • Establish purpose of assessment and summarize • Questions: • Is this a new issue or audience for the organization? • Is there agreement up the organizational chain this issue or audience needs to be addressed? • Risks: • Unknown stakeholders • Communication issues • LACK OF SUPPORT ULS Research Methods Workshop

13. 2. Establish the Planning Team • All steps should be conducted with planning team • Ideally members of stakeholder groups and experts • Communication is essential • Consider resource needs: time, expertise etc. • Questions: • Are the stakeholders new or well known? • How will geography impact planning? • Expertise within the team? • Risks • Sense of ownership • Communication • Planning can overtake action ULS Research Methods Workshop

14. 3. Establish Goals and Objectives • Summary • Identify the desired and actual levels of knowledge or skill • Identify cause(s) for the lack of knowledge or skill • Devise solution(s) • Questions: • Are the goals widely shared by the audience? • Are your objectives measurable? • Will the project be considered a success if the objectives are met? • Risks: • Potential to disengage by different team members who have different priorities • Not knowing if goals have been achieved and to what degree • Long term-support for doing needs assessments ULS Research Methods Workshop

15. Play Time

16. 4. Characterize Your Audience • Summary • Sample size • Skill & knowledge level • Educational Level • Attitudes and biases • Questions: • How long have you worked with the audience? • How much variation is there within the audience? • Risks: • Assumptions can backfire • Broadly generalizing and audience can lead to failure ULS Research Methods Workshop

17. Topics for Audience Characterization • Knowledge • Training • Tools & Techniques • Benefits • Attitudes & Biases • Ability to Attend or Access • Cultural Characteristics ULS Research Methods Workshop

18. 5. Information & Literature Search • Summary • Environmental Scan • Readings • Surveys to address questions raised in review • Questions: • Has this audience or issue been surveyed in the past? • What other ways might we find useful information about this audience or issue? • Risks: • Results may be outdated • Redundant effort, wasted time • Missing important information that may make results easier or better ULS Research Methods Workshop

19. 6. Select Your Data Collection methods • Summary • Observation • Personal Interviews • Surveys • Focus Groups • Questions: • Have all methods been considered? • Consider audience characteristics when drafting collection methods • How much expertise is there in-house? Seek help! • Risks: • Can be intrusive or upsetting • Time spent designing vs time spent analyzing ULS Research Methods Workshop

20. 7. Determine Your Sampling Scheme • Summary • Sample more than you think you need • Questions: • What is statistically recommended? • What is the population size of audience? • Risks: • Likely not scientifically sound • Too many or too few in sample ULS Research Methods Workshop

21. Exercise: Develop Your Goals Discuss how you would approach Audience Characterization Data Collection Methods Sample Size Measure of Success ULS Research Methods Workshop

22. 8. Design and Pilot Your Collection Instrument • Summary • Pilot first to identify weaknesses • Questions: • How will you pilot your instrument? • What kind of expertise is on your planning team? • How important is statistical precision? • How will data collection be standardized? • Risks: • Instrument will not be clear or gather necessary data • The audience will not be receptive to the survey instrument • Asking too many questions may irritate the respondents ULS Research Methods Workshop

23. 9. Gather and Record Data • Summary • Find means to incentivize respondents (survey prizes, etc) • Ensure anonymity if this is a priority to audience • Coordinate with assessment team • If audience is outside of library, make use of External Communications group • Questions: • Will you recruit people outside planning team to assist in data gathering? • Are you getting the desired response rate? • Risks: • Data will be biased • Language or vocabulary issues • Invalid Study ULS Research Methods Workshop

24. 10. Analyze Data • Summary • Keep findings and interpretation of findings separate in reporting • Questions: • Seek assistance in interpreting data if necessary • Risks: • Under or over-whelming audience with report • Missing trends and patterns • Not accounting for possible critical barriers • Letting bias slip into the process ULS Research Methods Workshop

25. 11. Manage Data • Summary • This step involves determining how data will be organized and archived. • The importance of this step is often not recognized until it is too late. • Questions: • Will this data ever need to be referred to again? • Will the raw data be stored? How will it be stored? • Can/should the data be shared? • Risks: • There may be unforeseen reasons that would necessitate the data being used again • Inadequate metadata ULS Research Methods Workshop

26. 12. Synthesize Data and Create Report • Always address your goals and objectives in synthesis. Report must include problems or errors with the design and the implementation of the survey. An executive summary is often helpful. • Questions: • Who is the audience for the report? • Did you address your objectives? • Risks: • Potential to disengage by different team members who have different priorities • Not knowing if goals have been achieved and to what degree • Long term-support for doing needs assessments ULS Research Methods Workshop

27. Gather your Data/Report ULS Research Methods Workshop

28. Thank you! Works Cited: http://www.cscnoaa.gov/needs http://ryanrwatkins.com/na (includes link to free NA eBook) Additional Reading: http://en.wikipedia.org/wiki/Needs_assessment ULS Research Methods Workshop

29. Survey Basics and Survey Monkey John Fudrow ULS Leadership Program 20 November 2013

30. When to Use a Survey • Explore a Topic • Discussion with Target Population • Gather Objective Data on Subject • Benchmarking of Service Levels ULS Research Methods Workshop

31. Survey Monkey Templates • Over 150 pre-made templates • Variety of topics • Community (2) • Customer Feedback (7) • Demographics (9) • Education (20) • Events (7) • Healthcare (14) • Human Resources (18) • Industry Specific (58) • Just for Fun (9) • Market Research (12) • Non-profit (5) • Political (11) ULS Research Methods Workshop

32. Surveys We Have Applied • ULS General Survey • Library Instruction Survey • E-book Survey • Event Feedback Surveys • Internal Planning ULS Research Methods Workshop

33. Types of Survey Questions • Open-Ended • Closed-Ended ULS Research Methods Workshop

34. Types of Survey Questions • Open-Ended • i. Single Textbox • ii. Multiple Textboxes • iii. Comment/Essay Box • iv. Numerical Textboxes • v. Demographic (U.S. or International) • vi. Date and/or Time ULS Research Methods Workshop

35. Types of Survey Questions • Close-Ended • i. Multiple Choice (Only One Answer Allowed) • ii. Multiple Choice (Multiple Answers Allowed) • iii. Rating • iv. Ranking • v. Matrix of Choices (Only One Answer Per Row) • vi. Matrix of Choices (Multiple Answers Per Row) • vii. Matrix of Drop-down menus ULS Research Methods Workshop

36. Likert Scales and Ratings • What is a Likert Scale? • A scaled response of a respondents feelings toward a topic based on a presented scale. ULS Research Methods Workshop

37. Likert Scales and Ratings • Clear wording of response choices. • Number of choices to relevancy • 5 to 7 choices allow for a consistent distribution. • The third choice should be the undecided or neutral decision. • Be careful not to force ranking by the wording of the base question. ULS Research Methods Workshop

38. Anonymity • IRB Authorization • Service Improvement • Anonymous Ratings • Do not store data publically ULS Research Methods Workshop

39. Survey Delivery • Target your audience • Multiple avenues for large samples • Create Multiple links to analyze effectiveness of delivery methods • DON’T SPAM • You cannot mass email ULS Research Methods Workshop

40. Statistical Relevance and You • We are not statisticians • We don’t have the resources nor time to run full analysis • Our reports are not statistically relevant • Our analysis is focused on report fulfillment • We don't properly sample • Our audience is often “expert” library users • This alters the influence of their input • We don't run via SPSS • Statistical Package for the Social Sciences ULS Research Methods Workshop

41. Using Survey Monkey (Free Version) • https://www.surveymonkey.com/ • Create a free account ULS Research Methods Workshop

42. Survey Monkey (Free Version) • Limitations • 10 Questions • 100 Responses • No Page Logic • 20 Less Templates • No Customization ULS Research Methods Workshop

43. Survey Question Creation • Create 2 questions you would like to ask of your department or patrons • Think about how you would use the results and write that idea in the page description. • We will look at several and discuss their efficacy. ULS Research Methods Workshop

44. Focus Groups: the Method and How to Use It Karen Calhoun ULS Leadership Program 20 November 2013

45. Learning objectives • Know the advantages and disadvantages of focus groups and when to use them • Be able to select and recruit focus group interviewees • Know how to prepare a focus group interview guide • Have information about: • The roles of the facilitator and recorder • How to analyze focus group interview data ULS Research Methods Workshop

46. Focus Groups, Defined • Qualitative social science research method • Feature open, interactive, in-depth small group discussion (typically 6 to 10 people), led by a trained moderator/facilitator • Participants carefully selected • Evaluate participant thoughts, opinions, practices, values, beliefs, feelings in a nonthreatening, semi-structured setting ULS Research Methods Workshop

47. Purpose of a focus group • IT IS TO: • Collect information and ideas on a pre-selected topic • IT IS NOT: • To achieve consensus • To answer participants’ questions (facilitator ≠ sage on stage) • To solve one or more problems • To make decisions • To provide a forum for participants to gripe or vent frustration (group therapy) ULS Research Methods Workshop

48. Generate insight into not just what people think or feel, but why Can be comparatively low cost method research method Interactive, not one way Can be used alone or in combination with other research methods Advantages of Focus Groups CC BY NC Francois Proulx http://www.flickr.com/photos/91569742@N00/404909051/ ULS Research Methods Workshop

49. Disadvantages/Risks of Focus Groups • Findings not representative of entire population of interest • Quality of results highly dependent on: • Appropriate preparation • Skill of moderator • Skill of recorder • Skill of analyst (preparation of report) • Can be challenging to evaluate responses to open-ended questions ULS Research Methods Workshop

50. Steps to Set Up Focus Group Interviews • Decide how many focus group interviews to hold (usually from two to a dozen) • Select appropriate facility (see next slide) and time(s) for the event(s) • Decide on participant incentives • Recruit participants (usually 6 to 10 people in each focus group) • Prepare interview guide/script • Assign moderator and recorder • Conduct the focus group(s) • Analyze and report results ULS Research Methods Workshop