Survey Design & Item Construction

1. Survey Design & Item Construction Lindsay Couzens, M.S. UNLV’s 2009 Academic Assessment Workshop May 14th & 15th

2. Developing a Purpose • What is your overall research question? • What questions do you want your survey to answer? • Is there an existing survey that answers your questions? • Keep length of the survey in mind • Define target group • Identify Objectives UNLV Office of Academic Assessment

3. Question Objectives • Identify Objectives • Based on purpose/research question • Example: Fowler, 1995 • Develop definitions for objectives • All respondents should have a common understanding of the meaning of each question • Specify the kinds of answers needed to meet the objectives • Informs item construction UNLV Office of Academic Assessment

4. Types of Surveys • In person • Can clarify confusing statements • Immediate feedback • Limited to those who are present • Mail • Can reach large geographic areas • Depend on respondents’ motivation • Requires current address database • Email • Can reach large geographic areas instantly • Cost-efficient • Requires computer access • Online • Cost-efficient • Requires computer access • Interview • Can ask exploratory or probing questions • Can assist with unclear questions • Requires trained interviewers • Requires up-to-date telephone numbers UNLV Office of Academic Assessment

5. Issues to Consider • Timeline • Number of data collection points • Cutoff dates • Cost • Who will construct the survey? • How? • Survey Monkey • Elisten • WebCampus tool UNLV Office of Academic Assessment

6. Issues to Consider • How will the survey be distributed? • Who will analyze the survey? • Do you need special hardware or software? • Who will maintain the survey? • Who is responsible for updating, revising, editing? UNLV Office of Academic Assessment

7. Methods Pilot testing Focus groups How will the surveys be distributed? How will they be collected? UNLV Office of Academic Assessment

8. Research Questions and ObjectivesThomas, 1999 What is the research question? Do the objectives for the survey flow from the research question? Is each objective stated in concrete terms? Does each objective measure one idea or concept? UNLV Office of Academic Assessment

9. Item Construction

10. Question Content • Each question should be mapped to one of your identified and defined objectives • Consider the content, scope and purpose of each item • Response formats • Open-ended or closed-ended? • Survey layout • Items should be grouped by topic AND by response type UNLV Office of Academic Assessment 10

11. Response Formats • Things to Consider • Clarity of Directions • Levels of measurement • Nominal • Ordinal • Interval • Exhaustiveness • Response alternatives have sufficient range UNLV Office of Academic Assessment 11

12. Response Formats • Things to Consider • Exclusiveness • Response alternatives are mutually exclusive • Balancing Categories • There should be an even number of alternatives on both sides of neutral UNLV Office of Academic Assessment 12

13. Response Formats • Numerical Rating Scales • Likert-type scales • Rates attitudes or opinions • Horizontal rating scales • Used to indicate where a respondent’s attitude or opinion falls between two opposite attitude positions UNLV Office of Academic Assessment 13

14. Response Formats • Numerical Rating Scales • Semantic differential • Opposite adjectives rather than attitude positions • Vertical rating ladder • Ranking UNLV Office of Academic Assessment 14

15. Response Formats • Scores • Out of 10 • Respondents rate each item • Ranking • Items are ranked relative to other items in the set UNLV Office of Academic Assessment 15

16. Response Formats • Checklists • Respondent checks all that apply • Binary choice formats • Dichotomous questions • Ex: What is your sex? (M/F); Do you smoke? (Y/N) • Paired comparisons • Respondent chooses between two overlapping options UNLV Office of Academic Assessment 16

17. Response Formats • Multiple choice formats • Choice between multiple nominal categories • Marital status • Choice between ordinal categories • How often do you eat in the dining commons? UNLV Office of Academic Assessment 17

18. Response Formats • Multiple choice formats • Choice between ordered attitude statements • Which attitude statement most closely matches your own? • Numerical answers • Salary; how many people live in the household UNLV Office of Academic Assessment 18

19. Question Wording ChecklistdeVaus, 2002 Is the language simple? Can the question be shortened? Is the question double-barreled? Is the question leading? Is the question negative? UNLV Office of Academic Assessment 19

20. Question Wording ChecklistdeVaus, 2002 Is the respondent likely to have the necessary knowledge? Will the words have the same meaning for everyone? Is there a prestige bias? Is the question ambiguous? UNLV Office of Academic Assessment 20

21. Question Wording ChecklistdeVaus, 2002 Is the question too precise? Is the frame of reference for the question sufficiently clear? Does the question artificially create opinions? Is personal or impersonal wording preferable? UNLV Office of Academic Assessment 21

22. Question Wording ChecklistdeVaus, 2002 • Is the question wording unnecessarily detailed or objectionable? • Does the question have dangling alternatives? • Does the question contain gratuitous qualifiers? • Is the question a “dead giveaway”? UNLV Office of Academic Assessment 22