Chapter 7: Types of Educational Measures

1. Chapter 7: Types of Educational Measures EDUC 502 November 7, 2005

2. Opening task • Form six groups of five students. • Have the first student hand copies of his/her abstract out to everyone in the group to read. • As you read through the abstract, make notes on any points that seem unclear or incorrect (BE BRUTAL!!) • Repeat until everyone has had their abstract evaluated.

3. Classifying educational measures • Cognitive or non-cognitive • Cognitive measures focus on what a person knows or is able to do mentally • Non-cognitive measures focus on affective traits or characteristics (e.g, personality traits, attitudes, values, interests, preferences, etc.)

4. Classifying educational measures • Commercially prepared or locally developed • Commercially prepared measures are developed for widespread use with a focus on technical merit • Locally prepared measures are developed by a researcher for specific situations with some, but not extensive, concern for technical characteristics

5. Classifying educational measures • Commercially prepared or locally developed • Commercially prepared measures are developed for widespread use with a focus on technical merit • Locally prepared measures are developed by a researcher for specific situations with some, but not extensive, concern for technical characteristics

6. Classifying educational measures • Self-report or observations by others • Self-report measures require the subjects to supply the response (e.g., tests, questionnaires, interviews, etc.) • Observations by others require subjects to be observed by others who record the data (e.g., observations, unobtrusive measures, etc.)

7. Four types of educational measures used in quantitative research • Tests • Questionnaires • Observations • Interviews

8. Tests • Norm-referenced - individual scores are interpreted relative to the scores of others in a well defined norming group (e.g., John's scores places him in the 95th percentile; Sally's score is in the bottom quartile) • Criterion-referenced - individual scores are interpreted in terms of the student's performance relative to some standard or criterion (e.g., Jeanne passed the Louisiana High School Graduate Exit Exam; Sammy did not make the cutoff for being promoted to the 7th grade)

9. Tests • Standardized tests have uniform procedures for administration, scoring, and interpreting test scores • Types of standardized tests • Achievement - tests of content knowledge or skills • Aptitude - tests which are used to predict future cognitive performance

10. Questionnaires • A questionnaire is an instrument containing statements designed to obtain a subject's perceptions, attitudes, beliefs, values, opinions, or other non-cognitive traits • Personality inventories • Psychological orientation (i.e., general psychological adjustment) • Educational orientation (i.e., traits such as self-concept or self-esteem that are related to learning and motivation)

11. Questionnaires • Likert scales • Response options require the subject to determine the extent to which they agree with a statement • An odd number of options provides for a middle or neutral response (e.g., strongly agree, agree, neutral, disagree, or strongly disagree) • An even number of options eliminates a response of neutral (e.g., strongly agree, agree, disagree, or strongly disagree)

12. Questionnaires • Checklists - responses require subjects to identify specific options from which they choose those options that appeal to them (e.g., which of the following dispositions are important to a first year teacher) • Ranked items - responses require students to place a limited number of items into sequential order (e.g., rank these dispositions in order of importance)

13. Problems with measuring non-cognitive traits • Difficulty clearly defining what is being measured (e.g., self-concept or self-esteem) • Response set - a tendency to respond the same way to all items (e.g., strongly agreeing with each statement) • Faking - a tendency to respond inaccurately (e.g., agreeing with statements because of the negative consequences associated with disagreeing)

14. Observations • Direct observation of behaviors • Natural or controlled settings • Structured or unstructured observations • Detached or involved observers

15. Observations • Inference • Low inference - involves little if any inference on the observers part • Children are in their seats • Teacher uses math manipulatives • High inference - involves high levels of inference on the observers part • Children are happy • Teacher lectures effectively

16. Observations • Laboratory observation • Specified environment • Use of structured forms and procedures • Concern with demand characteristics • Structured field observation • Natural setting • Use of structured forms and procedures • Frequency counts • Duration

17. Observations • Advantages of observations • Yields firsthand data without the contamination that can arise from tests, inventories, or other self-report instruments • Allows for the description of behavior as it occurs naturally • Allow for the consideration of contextual factors that can influence the interpretation and use of the results

18. Interviews • Interviews involve orally questioning subjects and recording their responses • Advantages • Establish rapport • Enhance motivation • Clarify responses through additional questioning • Capture the depth and richness of responses • Allow for flexibility • Reduce "no response" and/or "neutral" responses

19. Interviews • Disadvantages • Time consuming • Expensive • Small samples • Subjective • Types of questions • Structured • Semi-structured • Unstructured • Leading

20. Locating and evaluating educational measures • Reviews • Test authors and publishers • Computerized ERIC search for relevant articles • Reference sections of articles already read

21. Criteria for evaluating instruments • Validity evidence • Reliability evidence • Administration procedures • Norming information for norm-referenced tests (NRTs) • Standards for criterion-referenced tests (CRTs) • Avoidance of response problems in non-cognitive measures • Training observers and interviewers • High standards for observers using high inference observations

22. Homework Assignment • P. 174 (3, 5, 9, 10, 13, 14)