1 / 27

Test Development and Analysis

Test Development and Analysis. Session One - 14 May, 2014 What is measurement? Margaret Wu. Course content. Introduction to psychological measurement Test development Classical test theory Item response theory. Assessment. Assignment: Attendance 5% Weekly tasks 15%

deanna
Download Presentation

Test Development and Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Test Development and Analysis Session One - 14 May, 2014 What is measurement? Margaret Wu

  2. Course content • Introduction to psychological measurement • Test development • Classical test theory • Item response theory

  3. Assessment • Assignment: • Attendance 5% • Weekly tasks 15% • Design , administer and analyse a test. Prepare a written report 80%

  4. Overview of Week 1 Lecture • What is measurement? • Nominal, Ordinal, Interval, Ratio scales • Direct and indirect measurement • Latent variables • Some Key Concepts in Measurement • Validity • Reliability

  5. Psychological Measurement Measurement is about assigning numbers to represent an attribute. Levels of measurement • Nominal • Numerical values associated with a named attribute • Numbers on racing cars • Coding of gender (male=0, female=1) • Student identification codes in a survey • Classification of students with disability: hearing impaired, sight impaired, immobile etc. • Names (or labels)!

  6. Levels of Measurement • Ordinal • Numerical values that order with respect to an attribute • Position in which racing cars finish a race • Level of education attained • Codes for an item on attitude scale (eg Likert scale, strongly disagree=0, disagree=1, agree=2, strongly agree =3) • Orderings

  7. Levels of Measurement • Interval • Numerical values that give the amount of an attribute • Temperature • Time 5 mins to home time • Ratio • Numerical values that give the amount of an attribute and have an absolute zero • Height • Count of votes

  8. Summary: Levels of Measurement Source: http://trochim.human.cornell.edu/kb/measlevl.htm

  9. Latent Variables and Directness of Measurement • Students don’t wear their ability on their back • Ability must be indirectly measured through related observables • Indirect measurement needed • psychometrics (and educational measurement) is the science of indirect measurement • also has an extensive role in physical measurement

  10. Latent Variables and Manifest (Observable) Variables -- 1 1 2 Manifest variables, or “Items” Latent Variable 3 4 5 6 Not directly observable Directly observable

  11. An Example of a Latent Variable and Manifest Variables • Latent Variable: Eating disorder • Manifest Variables: symptoms • These can be observed. • Data can be gathered through a questionnaire.

  12. Example: Bulimia Check • Do you believe that you must continually be on a restrictive diet? • Do you eat in secret? • Do you experience feelings of guilt, shame or self-disgust in relation to your eating behaviour? • Do you binge eat more than twice a week? • Do you make yourself vomit? • Do you take fluid tablets (diuretics) in order to feel lighter? • Do you weigh yourself at least daily? • Do you abuse laxatives? • Do you believe that your life is evaluated by your weight, shape and size?

  13. Measurement of Latent Variables • Summing up the manifest variables • Scores on a questionnaire • Scores on a test • Scores reflect the “amount” of the latent variable • E.g., The number of “yes” answers reflects how serious the eating disorder is.

  14. Latent Variables and Manifest (Observable) Variables -- 2 1 2 Latent Variable 3 “Items” 4 5 6 A Bigger Idea Little Ideas

  15. One Item is Not Likely to be Sufficient -- 1 Latent Variable 1 ? = A Bigger Idea Little Idea

  16. One Item is Not Likely to be Sufficient -- 2 Latent Variable 1 1 Other stuff A Bigger Idea Little Idea

  17. Latent Variables and Manifest (Observable) Variables -- 3 1 1 2 2 Latent Variable 3 3 Other stuff 4 4 5 5 6 6 A Bigger Idea Little Ideas

  18. Implications of our model • Latent variable is central • Items can be changed • Latent variable “predicts” item responses • Items are reflections • Each item reflects the latent variable AND other stuff

  19. Measurement Validity & Reliability

  20. What is Validity? • A test is valid if it measures what it purports to measure. Allen, M.J. and Yen, W. M. (1979.) Introduction to Measurement Theory • A test is valid if it does what it is intended to do. Nunnally, J.C. (1978) Psychometric Theory

  21. What is measurement reliability?

  22. Latent Variables and Manifest (Observable) Variables -- 3 1 1 2 2 Latent Variable 3 3 Other stuff 4 4 5 5 6 6 A Bigger Idea Little Ideas

  23. Reliability • Is the instrument measuring something consistently & dependably • It pays no attention to the something • Reliability is a necessary, but not sufficient condition for validity. • Do repeated applications of the instrument under similar conditions yield consistent results?

  24. Consistency? • Would the test yield the same result tomorrow? • If you changed coders, would the test yield the same result. • If you choose another set of items from a content domain, would the test yield the same result

  25. Reliability or Validity?

More Related