Masters in Programme Evaluation

1. LECTURES 2006 Masters in Programme Evaluation Research Design for Programme Evaluation – Lecture 6

2. LECTURES 2008 Content • Measurement • Conceptualisation • Operationalisation • Levels of measurement • Reliability • Validity • Observations • Interviews

3. LECTURES 2008 Measurement • Consists of rules for the assignment of numbers to objects in such a way as to represent quantities of attributes (Nunnaly, 1978) • Rules: process for assigning of numbers explicitly stated and generally agreed upon by scientific community • Attributes of objects: objects – things in the world, individuals, groups, trees – which have attributes, features they may have – usually measure attributes of objects • Numbers to represent quantities: how much the object has of particular attribute – i) objects classified systematically ii) numbers manipulated by mathematical operations and inferences made about the objects

4. LECTURES 2008 Conceptualisation • Knowledge in form of text – relationships between constructs • First task to clarify conceptually what is to be measured – clear, explicit, and specific conceptual definitions of constructs • Developing conceptual definitions i) exploring everyday understanding of the construct ii) consult scholarly literature – if too little, consult related theory and develop working definition if too many, develop clear and unambiguous definition or borrow one iii) test definition mentally by applying to specific contexts and judging whether appropriate

5. LECTURES 2008 Operationalisation • Translating the linguistic meaning of a conceptual definition into observable indicators of the construct – linking the world of ideas to observable reality • The operational definition defines the construct in terms of specific operations, measurement instruments or procedures through which it can be observed – operational definition should correspond with conceptual definition • Elements before and after translation correspond with each other – attribute firmly grounded in theory through conceptual definition which in turn expressed in observable indicators thus grounded in reality

6. LECTURES 2008 Levels of measurement • Numbers abstract symbols that obey mathematical rules • Can be used to represent quantities of attributes • Relations between numbers in a mathe-matical system may not correspond with relations between attributes being quantitatively represented • Distinguish between 4 levels of measurement, corresponding with 4 characteristics of numbers

7. LECTURES 2008 Levels of measurement … • Nominal: numbers used to label categories – to distinguish between them – cannot use mathematical operations meaningfully on these numbers • Ordinal: categories differ and are ranked – can perform mathematical relations <,> • Interval: in addition to difference, rank also distances between numbers are meaningful – legitimately perform +,- • Ratio: all properties of interval plus true zero value – in addition can perform x,÷

8. LECTURES 2008 Reliability • Consistence with which a measuring instrument is measuring over repeated trials • Classical test theory: individual’s observed score (X) consists of a true score (t) plus a random error component (e): X = t + e • True score of an individual is his/her mean score over an infinite number of applications of the measuring instrument

9. LECTURES 2008 Types of reliability • Test-retest: applying the instrument to large sample of individuals and on different occasion apply same instrument to same sample and correlate the sets of scores • Parallel forms: if same test is used, and period too short carry-over effects, if period too long attribute may change – use parallel (equivalent) forms and correlate scores • Split-half: apply test, split items into 2 halves and correlate • Internal consistency: degree to which all items in instrument correlate with the instrument as a whole – Cronbach’s alpha coefficient (α)

10. LECTURES 2008 Validity • Degree to which a measuring instrument measures what it is supposed to measure • Measure good degree of fit between conceptual and operational definitions of construct and instrument should be usable for the particular purpose for which it was designed • Reliability a necessary but not sufficient condition for validity

11. LECTURES 2008 Kinds of validity • Criterion-related validity: extend to which an instrument is related to some or other standard/criterion of a construct – correlations between scores on a measuring instrument and a criterion measure (e.g. skills test and work performance) – 2 types depending on when criterion data available i) predictive validity: criterion data available in future ii) concurrent validity: criterion data available at same time • Content validity: the extent to which a measuring instrument reflects a particular domain – important in measures of knowledge • Construct validity: extent to which an instrument measures a construct – convergent and discriminate

12. LECTURES 2008 Interviews • Natural form of interaction • Often used in interpretive/qualitative research • Job interviews also very popular • How structured: i) highly structured – straightforward information – essentially list of standard questions, like questionnaire ii) in depth information – unstructured – difficult to interpret quantitatively iii) semi-structured – list of key topics - interview schedule – interpret in terms of ratings or behaviourally anchored rating scales (BARS)

13. LECTURES 2008 Observations • Direct contact with behaviour that is studied (interviews and questionnaires depend on recall) • Kinesics – study of body movements • Proxemics – the study of space when people interact • Unobtrusive measurement • Structured (more or less) – using standardised rating scales to record – time samples – checklists of particular kinds of behaviour – observations by independent observers

14. LECTURES 2008 Data collection • Basic material with which researchers work • Based on observations – numeric (quantitative) linguistic (qualitative) • Should be sound: capture the meaning of what is being observed • Validity the extent to which the operational definition captures the meaning of the conceptual definition

15. LECTURES 2008 Data analysis • Aim is to transform data into an answer to the research question • When planning the research should have a very clear idea of the type of data to be gathered and how they are going to be analysed • Careful consideration of the data analysis strategies will ensure that the design is coherent as the researcher matches the analysis with the type of data, the purpose of the research and the research paradigm

16. Assignment 3 LECTURES 2008 Refer to the article and answer the questions below: • Wesson, M.J. & Gogus, C.I. (2005). Shaking hands with a computer: An examination of two methods of organizational newcomer orientation. Journal of Applied Psychology, 90(5), 1018 - 1026. • Deadline: 4 April

17. Assignment 3 LECTURES 2008 • Discuss the researchers’ sampling strategy critically. • How would you improve on the researchers’ sampling strategy if necessary? • Discuss the researchers’ measurement of the dependent variable(s) critically. • If you could improve on their measurement of the dependent variable(s), how would you do it?