DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT

1. DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT Kathleen A. Martin, D.P.E. Office of Institutional Research & Assessment Bucknell University

2. POINT OF CLARIFICATION • Terms “survey” and “questionnaire” are often used interchangeably • “Survey” refers to a descriptive research technique or methodology (also includes interviews, and focus groups) • “Questionnaire” refers to a paper and pencil instrument through which information is obtained • For the purposes of outcomes assessment, questionnaires are frequently developed to survey a population

3. QUESTIONNAIRES & ASSESSMENT • Useful for surveying groups of any size • Typically designed to determine opinions, attitudes or present practices • Limitation: rely on self report information v. observed behavior • Challenges to validity become a concern

4. VALIDITY CHALLENGES • Validity: the extent to which the test measures what it purports to measure • Internal Validity: Relates to instrument (questionnaire) quality. Are the appropriate questions asked clearly and logically? • External validity: can the results be generalized? • Measurement error: difference between the characteristics of your sample and the characteristics of the population • Non-response bias: answers of respondents do not mirror the characteristics of those who refused participation.

5. MAXIMIZING VALIDITY • Create a questionnaire that is clear and logical • Support content relevance and content representativeness • Content Relevance: are the questions relevant to the purpose of the questionnaire? Unimportant content should be eliminated. • Content Representativeness: Are the questions an adequate representation of the universe of possible questions? • Solicit the input of experts in scale development and the content of the questionnaire for an objective evaluation • Select a large, representative random sample from the population of interest or census the finite population • Increase response rate using formal follow up procedures

6. INITIAL CONSIDERATIONS • Is the development of a new questionnaire . . . • Necessary? • Feasible? • Select mode of administration • Web Based? • Paper and pencil? • Combination?

7. QUESTIONNAIRE PLANNING • 1. Determine the Objective • What do you want to ask and of whom? • Consider how the data will be analyzed. What information is needed? • 2. Determine the Mode of Administration • Paper and pencil • Cost effective, but low response rate • Web / E-mail based • Cost effective, higher response rate, some suspicion of the medium • Combination • Can be administratively cumbersome

8. QUESTIONNAIRE PLANNING • Determine the Sampling Methodology • To whom will you send the questionnaire? Who can supply the information? • Will everyone be surveyed? • ALL current students? ALL alumni? • If yes, this is a census of the finite population • The selection of a representative sample of the population may be more effective based on time and cost of administration • Decision depends on the size of the finite population

9. CONSTRUCTING the QUESTIONNAIRE • Consider what objective each question measures • Three to five questions per objective is recommended • How many questions? • Find a balance: long enough to obtain the necessary information but short enough so respondents won’t lose interest. • Questionnaires that are too long won’t be completed • Goal: 25-30 minutes MAX

10. DESIGNING THE QUESTIONNAIRE • Question Formats: Open Ended Questions • Allow respondents to express feelings and expand on ideas. • Question should be phrased to avoid a one word response. • “What aspects of the Sociology program best prepared you for your current position?” • Open ended questions at the beginning of the questionnaire may be discouraging to respondent – include one or two at the end • Analysis of responses can be challenging

11. QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Ranking • Forces the respondent to rank order responses based on a value judgment. • Rank the following items with regard to how you spend time at Bucknell from 1 (spend the most time) to 5 (spend the least time). _____ Studying _____ Socializing _____ Volunteering _____ Working at a part time job _____ Exercising

12. QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Checklist • A number of possible answers are provided and the respondent is asked to choose one or choose all that apply. • How did you finance your Bucknell University education? Choose all that apply. _____ Parental contribution _____ Student Loan _____ Work Study _____ Off campus employment _____ Other (please specify): _____________

13. Neutral / No Opinion Strongly Agree Strongly Disagree Disagree Agree QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Scaled items • Very common • Indicate strength of agreement or disagreement with a statement. • Numbers are assigned to each response on the continuum • The English curriculum at Bucknell University adequately prepared me for my current position. 1 2 3 4 5

14. QUESTIONNAIRE DEVELOPMENT • Question Formats: Closed Questions • Suggestions for writing scaled items • Use positively worded or negatively worded items consistently • Use an odd number of responses along the continuum to allow for a neutral response • Anchors may be different based on focus of the item (Strongly Agree – Strongly Disagree; Frequently – Never; Very Good – Very Poor) • Keep items short

15. QUESTIONNAIRE DEVELOPMENT • Question Formats: Categorical • Typical demographic questions: • Yes/No/?, Male/Female, F/S/J/S/G • Answers fall into certain categories – respondents pick the one that applies to them • Analyzed using frequency, proportion, percentages • 2 considerations: • Questions on a continuum should stay on a continuum if they will be directly analyzed • Example: Age – allow respondent to write in their age rather than check off a box for their age range • Categories must be collectively exhaustive & mutually exclusive (next slide)

16. Problematic: Number of hours spent studying per week: 18 – 21 21 – 24 24 – 27 Categories are not mutually exclusive Corrected: Number of hours spent studying per week: 18 – 21 22 – 25 26 – 29 QUESTIONNAIRE DEVELOPMENT

17. IMPORTANT CONSIDERATIONS • APPEARANCE AND DESIGN • Have clear directions that are age appropriate & easy to follow • Format, size, reproduction should facilitate completion • No less than 12 pt • Times, Arial, Courier • PILOT STUDY • Give the questionnaire to a few people to try it out • Have them evaluate readability and clarity of questions • Estimate time required to complete the questionnaire • Suggestions for revisions

18. SENDING IT OUT • INITIAL CONTACT / COVER LETTER • Short & to the point • Identify person by name • Convince respondent that participation is important • Ensure confidentiality • Deadline for return of questionnaire • SASE for return (mailed questionnaires only)

19. Institutional approval • Institutional Review Board • Research involving human subjects must be approved by the Bucknell University IRB • For more information, visit the IRB website at http://www.departments.bucknell.edu/inst_research/IRB/irb.htm • Campus Survey Coordination • The timing of all university-wide surveys must be coordinated through the Campus Survey Coordination Group. • For more information, visit the CSCG website at http://www.departments.bucknell.edu/inst_research/cscg.htm

20. FINAL THOUGHTS • FOLLOW UP PROCEDURES • Send to all participants to preserve confidentiality (“If you haven’t already returned the questionnaire…..”) • May include a second copy if available • Usually done in 2 week intervals • RESPONSE RATE FOR MAILED QUESTIONNAIRES • Typically low - 30% is considered good • Higher response rate with web-based questionnaires • Good follow-up increases response rate

21. SUMMARY OF STEPS • Determine the Objective (consider analyses) • Determine the Mode of Administration • Determine the Sampling Methodology • Construct the Questionnaire • Institutional Approval • Conduct the Pilot Study • Write the Initial Communication • Send the Questionnaire • Follow up • Analyze the Results

22. FOR MORE INFORMATION Contact Kathy Martin @ 71960 or kmartin@bucknell.edu