1 / 35

La présentation sera en anglais/Presentation will be in English

deiondre
Download Presentation

La présentation sera en anglais/Presentation will be in English

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Unités de recherche éducationnelle/Educational Research UnitsL’adulte et l’apprentissage en milieu de travail/Adult and Workplace LearningMesure et évaluation des apprentissages/Measurement, Evaluation and AssessmentSéminaire d’après-midi/Afternoon Seminar30 janvier 2013/January 30th , 2013De 16h à 17h15/4:00 to 5:15 p.m. LMX 388 La présentation sera en anglais/Presentation will be in English Des rafraichissements seront servis/Refreshments will be served RSVP avant le 28 janvier à : RSVP before January 28th at: claire.duchesne@uottawa.ca Tips on how to develop measurement tools and collect data Ivana Pavic, Maurice Taylor, Sait Atas and David Trumpower

  2. Tips on how to develop measurement tools • Involve your learners and instructors in the development (Ivana) • Ensure the tool has a strong conceptual development (Maurice) • Pilot test the tool to obtain evidence of reliability, validity, utility (Sait) • Revise, as necessary, following psychometric analysis (David)

  3. Maurice Taylor & Ivana Pavic Let’s talk about the networks in our lives

  4. Talking about the social networks in our lives is very much related to a term called SOCIAL CAPITAL

  5. Social capital relates to the connections we have with people • …..connections to family • …..connections to friends • …..connections to the workplace • …..connections to the neighbourhood • …..connections to the local community

  6. Why are these connections important? 1) they help us advance our interests in the society we live in 2) they help us co-operate within groups 3) they promote trust, goodwill and a desire to act in a supportive manner

  7. Some examples • The Ottawa East Community Association • Immaculata Parent Council • The Art of Running Club • The Ottawa Newcomers Club • Sandy Hill Community Health Centre Volunteer • One Parent Families Association • Jamaican Ottawa Community Association

  8. What are the benefits of social capital? • Glue- holds society together by facilitating co-operation within groups of people • Oil- keeps the economic and social wheels of society rolling smoothly

  9. Social capital is like: • a resource that people can accumulate and use to their benefit and advantage • having a bank account of all of the networks in our lives

  10. The lack of social capital also can hold back people preventing them from achieving their goals which can affect the quality of life

  11. Social capital is an important part of the learning process for 1) individuals 2) the training outcomes of a program 3) the community that we live in

  12. How do we know if we’ve got social capital? • How do we measure it?

  13. Conceptual development of the tool • How robust is the concept of social capital? • OECD: 2000-2012 – framework for social outcomes of learning • Australian Bureau of Statistics (2004) – indicators for a social capital framework • A concerted effort to empirically understand adult literacy and social capital outcomes in Scotland, the UK, Australia and New Zealand

  14. Conceptual development • Interest on Canadian soil • Canadian Institute for Policy Research Initiatives; National Centre for Literacy • Is the timing right for developing a measure • Do you have access to a target population?

  15. Conceptual development • 1st Subscale: Network Qualities (NQ): including sub-concepts of trust levels, efficacy, diversity and inclusiveness • 2nd Subscale: Network Structure (NS): including sub-concepts of size, communication mode and power relationships • 3rd Subscale: Network Transactions (NTS): including sub-concepts of sharing support and sharing knowledge • 4th Subscale: Network Types (NTY): including sub-concepts of bonding, bridging, and linking

  16. Lets talk about the reliability, validity, and utility of the Social Capital Inventory

  17. Sample Items from SCI

  18. Reliability • …refers to the stability or consistency of scores (over time, across raters, across items). • When a test is reliable, it provides dependable and consistent results.

  19. Methods for Estimating Reliability

  20. Let’s have a look at the Social Capital Inventory’s reliability • To test SCI’ s reliability internal consistency reliability was determined. • *The SCI was found to have adequate reliability.

  21. Is reliability good enough for a test to be used A reliability coefficient only indicates whether the attribute measured by the test— whatever it is—is being assessed in a consistent way. Whether the test is actually assessing what it was designed to measure is addressed by an analysis of the test's validity.

  22. Validity • The term validity refers to whether or not the test measures what it claims to measure. • On a test with high validity the items will be closely linked to the test’s intended focus.

  23. SCI Validity • Three major types: Involves systematic examination of the test content to determine whether it covers a representative sample of the domain to be measured Typically conducted by panel of subject matter experts

  24. SCI Validity • Three major types: Involves demonstrating that a measurement tool varies systematically with another measure of the same construct But, requires the existence of another valid measure…

  25. SCI Validity Involves empirical testing of hypotheses about the construct, using the test as a measure of the construct e.g., if we know that one group should have higher levels of construct X than another group, then comparison of the mean level of construct X (as measured by the test) for the two groups should reveal a significant difference • Three major types:

  26. Ongoing research • N=115 adult learners seeking literacy and essential skills in formal, non-formal, and informal learning settings • 1 Adult high school (Ontario) • 6 Workplace programs (4 in Manitoba, 2 in Nova Scotia • Mixed methods study (SCI, SDLRS, interviews)

  27. Ongoing research - results • Relatively high SCI means are consistent with qualitative data obtained from instructors and learners • Those with average to above average scores on the SCI have significantly higher scores on the SDLRS • Disproportionately higher percentage of females than males had average to above average scores on the SCI, if they do not belong to any clubs

  28. Take-home Tips • 1. Start with theory • -Read, read, read!

  29. Take-home Tips • 2. Seek feedback from Subject Matter Experts • -Teachers • -Colleagues • -Committee members?!

  30. Take-home Tips • 3. Pilot test • -Small, representative sample • -Use multiple raters (if applicable) • -Evaluate descriptive statistics (frequency distributions, means, SDs) • -Evaluate reliability (Cronbach alpha, item- total correlations, inter-rater agreement)

  31. Take-home Tips • 4. Revise, if necessary • -Delete unreliable items • -Clarify wording • -Modify scale • -Consider difficulty/discriminability

  32. Take-home Tips • 5. Include validity checks • -Collect demographic data • -Include other measures related to construct • -Compare with prior studies/theory

  33. To Summarize: • Read • Listen to your committee • Pilot test • Revise • Include validity checks

More Related