AIR 51st Annual Forum May, 2011

1. AIR 51st Annual Forum May, 2011 Estimating Behavior Frequency: Do Vague and Enumerated Estimation Strategies Yield Similar Results? Presented by: James Cole, Ph.D. Assistant Scientist Ali Korkmaz, Ph.D. Assistant Scientist Indiana University Center for Postsecondary Research

2. Estimating Behavior Frequency Probably most everyone in this room has collected self-reported data on behavioral frequency. Questions like . . . . How often did you meet with your advisor? How many hours per week do you study?

3. Estimating Behavior Frequency Probably most everyone in this room has collected self-reported data on behavioral frequency. Questions like . . . . How often did you meet with your advisor? How many hours per week do you study? You then examine your data. . . 34% said they met with their advisor “very often” 16% said they studied more than 15 hours per week

4. Estimating Behavior Frequency Probably most everyone in this room has collected self-reported data on behavioral frequency. Questions like . . . . How often did you meet with your advisor? How many hours per week do you study? You then examine your data. . . 34% said they met with their advisor “very often” 16% said they studied more 15 hours per week You wonder . . . “What does ‘very often’ mean?” “How did the student estimate 15 hours per week?”

5. Estimating Behavior Frequency The purpose of this presentation is to provide additional information regarding the use of vague and enumerated quantifiers in survey research. Specifically, do vague and enumerated responses produce consistent results?

6. Estimating Behavior Frequency Surveys of student behavior commonly collect data regarding estimations of behavior frequency in two ways: Vague quantifiers Beginning Postsecondary Students Longitudinal Study (NCES) includes items such as “how often respondent wrote essay answers as part of exams during the 2003-2004 academic year” with response categories “never,” “sometimes,” and “often.”

7. Estimating Behavior Frequency Surveys of student behavior commonly collect data regarding estimations of behavior frequency in two ways: Vague quantifiers Beginning Postsecondary Students Longitudinal Study (NCES) includes items such as “how often respondent wrote essay answers as part of exams during the 2003-2004 academic year” with response categories “never,” “sometimes,” and “often.” Enumerated (or tallied) quantifiers Beginning College Survey of Student Engagement (CPR), respondents are asked “During your last year of high school, about how many hours did you spend in a typical 7-day week doing each of the following?” and then provided with a list of activities and numeric response categories (0, 1-5, 6-10, 11-15, etc).

8. Estimating Behavior Frequency Referencing Versus Enumerating When Estimating Behavior Frequency The process of estimating behaviors using vague quantifiers is distinct from the process used to estimate behavior by trying to tally or count occurrences. Vague response sets invoke a process of comparison to estimate the behavior frequency (Pace & Friedlander, 1982; Schaeffer, 1991; Wanke, 2002). People compare their behavior to: similar behaviors they also engage in, or behavior of others engaged in same behavior.

9. Estimating Behavior Frequency Referencing Versus Enumerating When Estimating Behavior Frequency Enumerated or tallied responses involves one or two steps depending on the time frame, behavior frequency, and schedule (fixed or variable) (Brown, 2002; Schaeffer & Presser, 2003). Simple enumeration for low frequency behaviors regardless of the time frame or schedule As the frequency of the behavior increases and time lengthens estimation becomes a two-step process: enumeration and extrapolation

10. Estimating Behavior Frequency • Though enumerated responses have the appearance of accuracy, it is important to note that enumerated responses are not necessarily accurate (Brown, 2002; Schaeffer, 1991). • Satisficing • Accuracy of stored and recalled information(Ashcraft & Radvansky, 2009) • Good faith effort to accurately recall information (Brown & Sinclair, 1999).

11. Estimating Behavior Frequency Given the many difficulties with enumeration, some researchers have suggested that relative, vague judgments of behavior are preferred (Bradburn & Danis, 1984). • “Since behavioral frequency reports are error-prone anyway, why bother asking respondents for reports that suggest more precision than they can provide?” (Sudman, Bradburn and Schwarz, 1996 , p. 226).

12. Estimating Behavior Frequency Given the many difficulties with enumeration, some researchers have suggested that relative, vague judgments of behavior are preferred (Bradburn & Danis, 1984). • “Since behavioral frequency reports are error-prone anyway, why bother asking respondents for reports that suggest more precision than they can provide?” (Sudman, Bradburn and Schwarz, 1996 , p. 226). • However, Sudman, Bradburn and Schwarz (1996) go on to say: • “vague frequency expressions carry their own load of problems. . . different respondents use the same term to mean difference objective frequencies of the same behavior” (p. 226). • Thus, it is not surprising that researchers sometimes find group differences for meanings assigned to vague quantifiers (e.g., Wanke, 2002).

13. Estimating Behavior Frequency Group Differences in Estimating Behavior Due to the comparative process involved when a respondent interprets vague response sets students may interpret vague quantifiers differently based on their background, social groups, or past experiences. • “When the meaning of vague quantifiers differ by group, relative and absolute responses may lead to different conclusions” • (Schaeffer, 1991 p. 397). • This norming effect can become particularly problematic when examining groups differences using responses to vague quantifiers (Wright et al., 1994).

14. Estimating Behavior Frequency This study extends this line of research by investigating the following questions: • Is there a relationship between the frequency of the behavior and the time frame selected to estimate the frequency of the behavior? • For each behavior is there an increasing enumerated estimation with each increasing level of vague quantifier? • Are there group differences between vague and enumerated estimates of the same behaviors based on gender, type of high school (public/private), orcompletion of calculus (an indicator of academic achievement level)?

15. Estimating Behavior Frequency Data Source 2010 administration of BCSSE with 6 experimental items added

16. Estimating Behavior Frequency Sample Sample includes 30,964 entering first-year students enrolled at 81 institutions across the US. • Institution characteristics include: • 15% doctoral 30% baccalaureate • 48% masters 7% other Student characteristics include: 59% female • 8% African American 67% White • 8% Asian/Asian American 8% Other • 9% Latino/Hispanic 17% attended private high school 36% reported passing high school calculus

17. Estimating Behavior Frequency Data Analysis Student responses to the enumerated items were recoded so that all responses were on a per week basis. Students could report frequency based on five options (per day, week, month, academic term, and academic year). Week was taken as the baseline and responses with other time frames were adjusted by appropriate multipliers (day = 5, month = .25, academic term = .111, and academic year = .02778). There are 180 days in a typical high school academic year, 4 terms of 9 weeks.

18. Results • Is there a relationship between the frequency of the behavior and the time frame selected to estimate the frequency of the behavior?

19. Results • For each behavior is there an increasing enumerated estimation with each increasing level of vague quantifier?

20. Results Note - Superscript indicates non-significant differences between indicated level(s). No superscript indicates significant difference with other level(s).

21. Results Note - Superscript indicates non-significant differences between indicated level(s). No superscript indicates significant difference with other level(s).

22. Results • Are there group differences between vague and enumerated estimates of the same behaviors based on gender?

23. Results • Are there group differences between vague and enumerated estimates of the same behaviors based on completion of type of high school?

24. Results • Are there group differences between vague and enumerated estimates of the same behaviors based on completion of calculus?

25. Discussion Overall this study found that: The more frequent the behavior, the shorter the time frame the respondent uses when estimating the behavior. The implication for researchers is that time frame is an important factor when asking respondents to enumerate estimates of past behaviors.

26. Discussion Overall this study found that: The more frequent the behavior, the shorter the time frame the respondent uses when estimating the behavior. The implication for researchers is that time frame is an important factor when asking respondents to enumerate estimates of past behaviors. Vague quantifiers are associated with increasing enumerated responses for the same behavior. This provides additional reliability evidence for the use of vague quantifiers.

27. Discussion Overall this study found that: The more frequent the behavior, the shorter the time frame the respondent uses when estimating the behavior. The implication for researchers is that time frame is an important factor when asking respondents to enumerate estimates of past behaviors. Vague quantifiers are associated with increasing enumerated responses for the same behavior. This provides additional reliability evidence for the use of vague quantifiers. There were minimal group differences regarding these estimates. The implication of these results is that for these types of behaviors there are likely no meaningful differences between these groups of students and how they interpret vague and enumerated estimates of behavior.

28. AIR 51st Annual Forum May, 2011 • Any discussion, questions, or comments?

29. Thank you! Jim Cole colejs@indiana.edu Ali Korkmaz akorkmaz@indiana.edu