E ngaging in R igorous I nstrument D evelopment to I ncrease V alidity

1. Engaging in Rigorous Instrument Development to Increase Validity of Assessment Results Civic Learning Program in Judicial Affairs Jilliam N. Joe | Wendy M. Young | Mary M. Johnston Snapshot of Civic Learning Program • Identify Construct-of-Interest • General Help-Seeking 2. Establish Student Development Objective • As a function of the Civic Learning Program, students will be more likely to increase attitudes toward help-seeking and increase help-seeking behavior. 3. Research Available Instruments …And when an off-the-shelf instrument can’t measure up… 4. Select Instrument Development Team • Judicial Affairs Assessment Committee • Consultant from the Center for Assessment & Research Studies DeVellis, R.F. (2003). Scale development: Theory and applications, 2nd ed. Thousands Oaks,CA: Sage Publications. Example Help-Seeking Items 32 item-instrument Defining the Construct Help-Seeking Threat1: To an individual with a high level of help-seeking threat, the cost to seeking help is depreciation in self-esteem. “Because seeking help implies inadequacy, it is more inconsistent, and therefore more threatening, to individuals with high self-esteem than to individuals with low self-esteem” (Karabenick & Knapp, 1991, p.222). Help-Seeking Avoidance1: An individual with a high level of help-seeking avoidance will choose to not seek help even in the face of failure. Instrumental Help-Seeking1: An individual who engages in instrumental help-seeking does so when necessary and in order to gain the least amount of assistance possible in order to problem-solve (e.g., asks for a hint). Executive Help-Seeking1: An individual who engages in executive help-seeking, on the other hand, does so in order to minimize the cost (effort) of completing the task on his own (e.g., asks for direct answers). Informal vs. Formal Help-Seeking1: Help-seeking tendencies are directed toward either informal or formal sources. That is, an individual will seek help from different sources – sources that are formal (e.g. instructor) or informal (e.g. peer). Confidence (Perception of Help-Source)2: Help source perception is the degree to which an individual has confidence in the help source. Recognition of Need for Help2: Help-seeking recognition is the acknowledgement of the need for professional help in the face of a problem (personal or emotional). 1 2 3 4 5 6 7 Not at all Completely true true of me of me 1 Karabenick, S.A. (2003). Seeking help in large college classes: A person-centered approach. Contemporary Educational Psychology, 28, 37-58. Karabenick, S.A. (2004). Perceived goal achievement structure and college student help seeking. Journal of Educational Psychology, 96, 569-581. 2 Fischer, E.H. & Turner, J. L. (1970). Orientations to seeking professional help: Development and research utility of an attitude scale. Journal of Consulting and Clinical Psychology, 35(1), 79-90. Piloting Items Using Concurrent Think-Aloud Method Help-seeking tendencies… Results Discussion As shown in Figure 1, participants had a wide range of conceptualizations about “campus resource” (e.g., student assistant, roommate). In many ways, students interpreted these items in the way the developers intended. That is, the test developers wanted students to consider campus resource in a broad sense and not in a narrow sense (e.g., faculty members). Participants’ interpretation of “problem”, on the other hand, presented a challenge. How a respondent might answer items that include the word “problem” depends on the context. More often than not, that context was academic (i.e., within the classroom), which we believe was a result of the language of some of the items that may have implied a classroom context (e.g., task). Moreover, whenever an item was vague, that is, whenever the student said “it depends” on the context, he or she would use the middle category. Had we not conducted the think-aloud, we might have interpreted middle category responses as a neutral response and not as an “I don’t understand” or “it depends” proxy. Method Participants Participants (N= 3) used in this exercise where volunteers from a residence life program. The sample consisted of two (2) females and one (1) male. Procedures Three (3) think-aloud moderators were used for the exercise (two female and one male). All three were members of the assessment committee. Members were trained on the think-aloud process and materials. A mock think-aloud was conducted between the lead moderator and another moderator to make sure everyone had the same conceptualization of the process. Data collection A concurrent verbal report (think-aloud) method (Ericsson & Simon, 1980 &1993) was chosen for the purposes of this exercise. Moderators met with students one-on-one. Interviews lasted, on average, for about one half hour. Afterwards, moderators reconvened to debrief and provide general impressions of students’ responses. • Have been studied throughout a number of contexts (e.g., clinical, secondary and post-secondary education; Gould, 2006; Karabenick, 2004 & 2003; Deane, Wilson, & Ciarrochi, 2001; Karabenick & Knapp, 1991; Fischer & Turner, 1970). • Currently, however, there is no general measure of help-seeking in a college context. • The objective of the item development process was to modify current measures used in academic and clinical settings to a more generalized setting. • Throughout the item development process… • Members on the development panel noticed how some items may elicit different responses depending on the context to which students think the items refer (e.g., academic, personal/psychological). • As such, it was deemed necessary to pilot items on students in a way that will help us to understand how items might be interpreted. Analysis Each statement made by respondent were analyzed. Initial codes were developed using impressions drawn from a review of the data, as well as observers’ field notes. Coding is meant to be mutually exclusive; each statement can receive only one code. An analytic coding approach was used (Richards & Morse, 2006). Codes served as keywords. Each time a statement with “depends” for, example, was encountered, that statement was highlighted. Figure 1. Conceptual map of think-aloud analysis for the Getting Help from Others Questionnaire