A-team Spring Session #2 Questionnaire Design

1. A-team Spring Session #2Questionnaire Design February 28, 2007

2. Questionnaire or Survey? • Questionnaire is an actual instrument: I.e. web questionnaire (Perseus) • Survey is actually a verb/method: “to study or examine comprehensively” • Your questionnaire is actually “surveying” • Often used interchangeably

3. Yields Different Types of Data: • Descriptive • Socioeconomic parameters to better understand the larger population represented by the sample. (e.g., income, age, college/school, major, class standing) • Behavioral • Patterns of use, recreation, entertainment, personal behavior. (e.g., # UGA bus rides per day/week) • Preferential • Opinions & preferences about socio-political issues. (e.g., opinion about new parking plan) Rea & Parker (1997)

4. Types of Responses/Measurement Scales • Nominal Scales • Used to categorize objects; “name” them • Object is in a category or it is not • No order implied along any dimension • Response sets that are nominal: • Yes/No (dichotomous) • Can be “Choose one,” “Choose all that apply,” “Choose one and specify,” or “Choose all and specify” from listing of characteristics Rea & Parker (1997); Schuh & Upcraft (2001)

5. Examples of Nominal Scales • Immediately after aerobic exercising I generally feel: ___Exhausted ___Invigorated ___Thirsty ___Sweaty ___Overheated ___Nauseated [Note: make sure (+) and (-) options offered in listing] • Indicate your sex: ___Male ___Female • Have you ever resided in Brumby Hall? ___Yes ___No Schuh & Upcraft (2001)

6. Measurement Scales (Cont’d) • Ordinal Scales(a.k.a. rank, order, rank-order) • Used to rank objects according to amount of characteristic the object possesses • Order reflects varying amounts or levels • Rank reflects range from high to low amounts • Ranking has no absolute zero • Intervals from one rank to next not the same • Likert scales are ordinal but sometimes treated as interval scales (judgment call) Schuh & Upcraft (2001)

7. Examples of Ordinal Scales • Order of finish in a horse race • Rank in class (e.g., achievement) • Highest degree earned • Order of preference • A higher number indicates a higher rank, e.g., “more” of characteristic possessed • Watch for (reverse) coding Schuh & Upcraft (2001)

8. Examples of Ordinal Scales • Rank-order your on-campus living preferences for the next academic year, with 1 = first priority, 2 = second priority, and so on: ___ Single Room ___One-bedroom Apartment ___ Double Room ___ Multiple-bedroom Apartment ___ Suite (2 double rooms separated by a bathroom) • Rank-order your reasons for attending this workshop, with 1=most influential reason, 2=second greatest influence, and so on: ___ Surveys are my life ___My boss sent me ___ To get the handouts ___ My thirst for knowledge ___ To earn CEUs ___ To get the free gift Schuh & Upcraft (2001)

9. Measurement Scales (Cont’d) • Interval Scales • A ranking/rating using interval score values • The difference between intervals is equal • The difference between 1 & 2 is the same as the difference between 4 & 5 • Still a focus on the amount of a characteristic an object possesses • Likert-type (pronounced Lick-ürt) scales often treated like interval scales (although considered ordinal): • 5=strongly agree, 4=agree, 3=no opinion, 2=disagree, 1=strongly disagree Schuh & Upcraft (2001)

10. Examples of Interval Scales • Likert Scale Example: Parking on campus should be free. __Strongly Agree __Agree __Neither Agree nor Disagree __Disagree __Strongly Disagree • Non-Likert Scale Example • When driving a UGA van, the safest following distance under ideal conditions (in seconds) is __1.5 __3 __4 __8 __10 __25 Schuh & Upcraft (2001)

11. Examples of Interval Scales • Non-Likert Scale Example (cont’d): Please rate your satisfaction with the following student activities on a scale of 1 to 5, with 1=very dissatisfied, 2=dissatisfied, 3=neither dissatisfied nor satisfied, 4=satisfied and 5=very satisfied. [could add not applicable or did not attend option]. __Welcome Week __Dawgs After Dark __Movie-O-Rama __Concert on the Quad __ Halloween “I Vant to Drink Your BlooooooDrive” Schuh & Upcraft (2001)

12. Measurement Scales (Cont’d) • Ratio Scales • Empirically meaningful zero/absolute zero; true absence of characteristic; (e.g., height, weight; more common in physical/biological sciences) • Have all characteristics of nominal, ordinal, and interval scales • Can be converted to ordinal scales • Can be converted to categories • Education examples: income, age, # years of education, # meetings with academic advisor Rea & Parker (1997); Schuh & Upcraft (2001)

13. Examples of Ratio Scales • Indicate the number of times you accessed the University Health Center in the last 30 days: ___ • Indicate your age: ___ Schuh & Upcraft (2001)

14. Likert Scales • Present question/item stem in a declarative sentence (one statement under consideration). • Response options represent varying degrees of agreement or endorsement of one statement. • Response options should be worded to represent approximately equal intervals; use equal # positive and negative possibilities. • The question stem doesn’t have to span the range of the construct (as in Thurstone or Guttman); response options infer levels of phenomena. DeVellis (1991, p. 68-70))

15. Likert Scales • Often 5, 7, or 9 response-options sets • A 6 response-options set is also common • Strongly disagree • Moderately disagree • Mildly disagree • Mildly agree • Moderately agree • Strongly agree DeVellis (1991, pp. 68-70)

16. Likert Scales • Midpoint often used but optional • What does midpoint wording imply? • Neither agree nor disagree: Apathy? • Agree and disagree equally: Strong paradox? • Common midpoint wording • Neither agree nor disagree • Agree and disagree equally • Neutral DeVellis (1991, pp. 68-70)

17. Likert Scales • Most used in surveys of opinions, beliefs, attitudes • Useful if statements are fairly strong (but not extremely) • Everyone can agree, have no opinion, or have little opinion about a mild statement • Write clear statements that reflect true differences of opinion DeVellis (1991, pp. 68-70)

18. Likert Scale Examples • Exercise is an essential component of a healthy life-style. 1=Strongly Disagree, 2=Moderately Disagree, 3= Mildly Disagree, 4=Mildly Agree, 5=Moderately Agree, and 6=Strongly Agree • Combating drug abuse should be a top national priority. 1=Completely True, 2=Mostly True, 3=Equally True and Untrue, 4=Mostly Untrue, and 5=Completely Untrue DeVellis (1991, p. 70)

19. Semantic Differential Scales • Response options consist of one but usually several adjective pairs • One adjective is negative, the other positive; each serves as the (-) or (+) end of a continuum that characterizes the stimulus DeVellis (1991, p. 70-71)

20. Semantic Differential Scales • Individual lines/points are placed between the two extremes (adjectives) • 7 or 9 lines/points are common • Respondents check/select lines or points closest to the adjectives if they hold extreme views • Respondents check/select lines or points toward the middle of the continuum if they hold more moderate views DeVellis (1991, pp. 70-71)

21. Semantic Differential Examples • Automobile Salesmen • Honest __ __ __ __ __ __ __ Dishonest • Quiet __ __ __ __ __ __ __ Noisy • Friendly __ __ __ __ __ __ __ Not Friendly • Fair __ __ __ __ __ __ __ Unfair • Trustworthy __ __ __ __ __ __ __ Untrustworthy DeVellis (1991, pp. 70-71)

22. Length of a Survey • Sufficient to capture needed data • Short enough to hold participants’ attention • Type of survey affects length • Types of questions affect length • Quantitative/Qualitative/Mixed approach affects length • Participant characteristics affect length

23. Measurement Scales: More Tips • Avoid providing categories/options that overlap; difficult or impossible to analyze • Frequently happens with age, income, class hours, years of service, hours worked, etc. • Example: • Select the category that best describes your annual, gross income: __$0-$10,000 __$10,000 – $30,000 __$30,000 - $60,000

24. Measurement Scales (Cont’d) • Be thoughtful with Use of “other” or “does not apply” or “not applicable” in listing of characteristics/options • Positive: Obtain option you may not have considered • Positive: Prevents forced responses • Negative: Can give response already listed or spurious data • Potential Negative: Adds to analysis time DeVellis (1991); Schuh & Upcraft (2001)

25. Analysis

26. Analysis (Cont’d) Schuh & Upcraft (2001); Upcraft & Schuh (1996)

27. References • DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park, CA: Sage. • Miller, T. K., (1999). CAS: The book of professional standards for higher education. Washington, DC: Council for the Advancement of Standards in Higher Education. • Payne, D. (1992). Measuring and evaluating educational outcomes. New York: Macmillan. • Rea, L. M. & Parker, R. A. (1997) Designing and conducting survey research (2nd Ed.). San Francisco: Jossey-Bass. • Schuh, J. H. & Upcraft, M. L. (2001). Assessment practice in student affairs: An applications manual. San Francisco: Jossey-Bass. • Upcraft, M. L. & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.