190 likes | 407 Views
Collective Wisdom. Designing Surveys for Use in Assessment. R. Ternes , October 2011. Defining the Problem. The most important consideration when creating a good survey is to identify and define your problem/question/goal. Vague problems lead to vague surveys which lead to vague answers.
E N D
Collective Wisdom Designing Surveys for Use in Assessment R. Ternes, October 2011
Defining the Problem • The most important consideration when creating a good survey is to identify and define your problem/question/goal. • Vague problems lead to vague surveys which lead to vague answers. • Chunk the problem • Break broad categories into manageable and measurable parts.
Examples • Too Broad • “What is our student learning environment like?” • Break it down. • How many hours do students study? • How many students study in groups? • How many papers do students write? • How many students attend office hours? • etc. etc.
Sampling and Generalizability • Sampling is linked to your goal. • Tackle sampling after defining your goal. • You must sample from the population that you want to make generalizations about. • If you want to generalize to all your students, you must sample from all your students. • Sampling English majors will tell you only about English majors.
Convenience Sampling • An ideal survey uses random sampling drawn from the population of interest. • Often, we have to make compromises to the way we sample. • We might sample our first-year students by surveying students in their first year writing course. • Understand the limitations of your choice!
Understanding Limitations • Consider the CLA, CAAP, MAAP etc. as a Gen Ed student learning measure. • For seniors, sample from everyone. • For freshmen, sample from students that enroll in your first year writing program (because it’s convenient). • If you do so, do you exclude your AP students? • Aren’t these some of your best students?
Writing Good Survey Questions • Good questions are: • Short • Simple • Clear • And they obviously relate to the goal or problem!
Writing Good Survey Questions • Binary or Likert scales are good standards • 5, 7, or 10 point scales are standard. • Right Hand Side = high numbers/pos. phrases • Use n/a only if you need to. • Always make it a separate option.
Writing Good Survey Questions • Use standard phrasing. • Strongly agree, agree, disagree, etc. • Don’t use words like ‘terrific’ or ‘terrible’. • Keep the language simple and plain. • Try to make the scale ‘balanced’. • One option for a negative weighting and 3 options for a positive rating ‘anchors’ the results towards the positive side.
Standard Format • The drinking age should be lowered from 21 to 18. • Strongly Agree • Agree • Neutral • Disagree • Strongly Disagree • This format naturally avoids several common pitfalls in survey design.
Use Non-Standard Formats Sparingly • Try to avoid non-standard question formats like: • ‘rank these in order of preference’ • ‘select all that apply’ • Non-standard formats • are more difficult to analyze statistically, • Are often harder to report, • and they can confuse participants. • Sometimes they can be very useful, but these questions are easy to do poorly.
Bad Question Overview • Leading Questions • Manipulating the Question Order • Double-Barrelled • ‘Either-or’ Fallacies • Lack of Variability • Lengthy/Complex Questions
Leading Questions • Leading questions steer respondents towards a particular answer. • Bad: “Do you think tuition is too high?” • Better: “How would you best classify your school’s tuition cost?” • Much higher than other comparable schools • Higher than other comparable schools • About the same as other comparable schools • Lower than other comparable schools • Much lower than other comparable schools
Question Order • Question order is important! • If you are polling opinions on government involvement in health care: • You are likely to illicit more negative responses if you first ask a dozen questions about government spending. • Put controversial questions last, if possible.
Double-Barreled Questions • Do not roll two questions into one. • “Do you like the flavor and texture of the cereal?” • If you get a ‘no’ response: • Is it the flavor they did not like? • Is it the texture they did not like? • Is it both?
Turn Either-Or Binaries into Likerts • Resist the temptation to create binary variables out of everything. • Are you a Democrat or a Republican? • Such a question ignores people that are Independent, Libertarian, Green Party, Tea Party, etc. • Researchers can lose valuable information by dichotomizing continuous variables. • Often binary questions can be easily turned into Likert type scales.
Lack of Variability • Design questions so that they give you information! • If everyone answers in the same way, then you haven’t gained much information. • Bad • Do you want more parking on campus? • Better • Do you think the university should create a $5 student fee to build more parking spots?
Avoid Lengthy Questions and Double Negatives • Don’t use double negatives. • Simplify, Simplify, Simplify. • Change: “At what convenience store were you or a member of your family most likely to purchase their gasoline from in the past year?” • To: “What gas station do you usually buy gasoline from?”