1 / 21

IE 486 Work Analysis & Design II

IE 486 Work Analysis & Design II. Instructor: Vincent Duffy, Ph.D. School of Industrial Eng. & Ag.& Bio Eng. Lab 3 – Evaluation of Questionnaire Data Friday, February 23, 2007. Layout of Data Sheet. People Observed Score (for each question) Total (Yi.) p1 7 6 13

roxy
Download Presentation

IE 486 Work Analysis & Design II

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IE 486 Work Analysis & Design II Instructor: Vincent Duffy, Ph.D. School of Industrial Eng. & Ag.& Bio Eng. Lab 3 – Evaluation of Questionnaire Data Friday, February 23, 2007

  2. Layout of Data Sheet People Observed Score (for each question) Total (Yi.) p1 7 6 13 p2 3 5 8 p3 3 3 6 p4 4 3 7 p5 5 5 10 p6 5 3 8 p7 6 6 12 p8 5 3 8 p9 5 5 10 p10 1 2 3 p11 7 6 13 p12 2 2 4 p13 4 5 9 Total ( Y.j ) 57 54 111 (Y..)

  3. k is the number of items in the group. s2res is the variance of residual components, which can not be controlled. s2p is the variance component for person. * The alpha coefficient is interpreted as the ratio of true score variance to observed score variance. Cronbach’s Alpha as a measure of Internal Consistency

  4. Cronbach’s Alpha as a measure of Internal Consistency Cronbach’s Alpha Coefficient Example: Number of people : N=13 Number of questions: k=2 i =1..a, j = 1..b, where a=13, b=2. SS(total) = (Yij2) – Y..2/ ab = 72+62+32+…+52 – (1112)/26 = 67.1154 SS(people) = (Yi.2) / b – Y..2/ ab = (132+82+….+92)/2 – (1112)/26 = 58.6154 SS(question) = (Y.j2) /a– Y..2/ ab = (572+542)/13 – (1112)/26 = 0.3462 SS(residual) = SS(total) – SS(people) – SS(question) = 67.1154 – 58.6154 – 0.3462 = 8.1538

  5. Cronbach’s Alpha Coefficient Example (cont’) MSp = SSp/(a-1) = 58.6154/12 = 4.8846 MSr = SSr/(ab-a-b+1) = 8.1538/12 = 0.6795 The estimated variance of person effect : S2p= [MSp – MSr ] / k = (4.8846 – 0.6795) / 2 = 2.1026 (variance component for person) The estimated variance of residual effect: s2r = MSr = 0.6795 (variance component for residual) The Alpha coefficient is calculated as: (k2 * s2p )) / (k2 * s2p + k * s2p) = 4*2.1026/(4*2.1026 + 2*0.6795) = 0.86089

  6. Exercise • Suppose we collected additional data as follows: • What is Cronbach’s alpha coefficient? • Can we conclude that our questionnaire has internal consistency.

  7. IE 486 Work Analysis & Design II Instructor: Vincent Duffy, Ph.D. School of Industrial Eng. & Ag.& Bio Eng. Lab 3 – Evaluation of Questionnaire Data Friday, February 23, 2007

  8. k is the number of items in the group. s2res is the variance of residual components, which can not be controlled. s2p is the variance component for person. * The alpha coefficient is interpreted as the ratio of true score variance to observed score variance. Cronbach’s Alpha as a measure of Internal Consistency

  9. Cronbach’s Alpha as a measure of Internal Consistency • For example, suppose 13 people were asked to rate a pair of questions on a 7-point scale. • The pair of questions look different but they are testing the same item. • For example • How much do you like the weather today? • How do you feel about the weather today?

  10. Layout of Data Sheet People Observed Score (for each question) Total (Yi.) p1 7 6 13 p2 3 5 8 p3 3 3 6 p4 4 3 7 p5 5 5 10 p6 5 3 8 p7 6 6 12 p8 5 3 8 p9 5 5 10 p10 1 2 3 p11 7 6 13 p12 2 2 4 p13 4 5 9 Total ( Y.j ) 57 54 111 (Y..)

  11. Cronbach’s Alpha as a measure of Internal Consistency Cronbach’s Alpha Coefficient Example: Number of people : N=13 Number of questions: k=2 i =1..a, j = 1..b, where a=13, b=2. SS(total) = (Yij2) – Y..2/ ab = 72+62+32+…+52 – (1112)/26 = 67.1154 SS(people) = (Yi.2) / b – Y..2/ ab = (132+82+….+92)/2 – (1112)/26 = 58.6154 SS(question) = (Y.j2) /a– Y..2/ ab = (572+542)/13 – (1112)/26 = 0.3462 SS(residual) = SS(total) – SS(people) – SS(question) = 67.1154 – 58.6154 – 0.3462 = 8.1538

  12. Cronbach’s Alpha Coefficient Example (cont’) MSp = SSp/(a-1) = 58.6154/12 = 4.8846 MSr = SSr/(ab-a-b+1) = 8.1538/12 = 0.6795 The estimated variance of person effect : S2p= [MSp – MSr ] / k = (4.8846 – 0.6795) / 2 = 2.1026 (variance component for person) The estimated variance of residual effect: s2r = MSr = 0.6795 (variance component for residual) The Alpha coefficient is calculated as: (k2 * s2p )) / (k2 * s2p + k * s2res) = 4*2.1026/(4*2.1026 + 2*0.6795) = 0.86089

  13. Cronbach’s Alpha as a measure of Internal Consistency SAS code for Cronbach’s alpha data one; input q1 q2; cards; 7 6 3 5 3 3 : : : 4 5 ; proc corr alpha; var q1 q2; run; Output from SAS: Raw value of Coefficient : 0.860892 Same as the result from hand calculation.

  14. Interpreting Cronbach’s Alpha results How to interpret the result? • The higher the correlation coefficient, the higher the internal consistency of the test. • The acceptable range for Cronbach’s alpha coefficient is usually between 0.7 – 1.0.

  15. Interpreting Cronbach’s Alpha results • What to do if the coefficient is low, such as 0.5? Check the following: • Are the questions ambiguous or not? • Are the scales sensitive enough to detect difference? Assumption: Question pairs are asked in the same direction

  16. Exercise We wanted to measure a concept: ‘the user satisfaction about a web site’. To measure that concept, we made one pair of questions. • How do you like this web site? 1) strongly dislike … 4) neutral … 7) strongly like • I’m very pleased with this web site when using it. 1) strongly disagree … 4) neutral … 7) strongly agree

  17. Exercise • Suppose we collected additional data as follows: • What is Cronbach’s alpha coefficient?

  18. Exercise • Step 1: calculate total (Yi., Y.j, Y..)

  19. Exercise • Step 2: calculate SS (a = 4, b = 2) • SS(total) = (62+72+32+22+52+72+42+52) – (392)/8 = 22.875 • SS(people) = (132+52+122+92)/2 – (392)/8 = 19.375 • SS(question) = (182+212)/4 – (392)/8 = 1.125 • SS(residual) = 22.875 – 19.375 – 1.125 = 2.375 Step 3: calculate MS • MS(people) = SS(people) / (a-1) = 19.375/(4-1) = 6.458 • MS(residual) = SS(residual)/ (ab-a-b+1) = 2.375/(8-4-2+1) = 0.792

  20. Exercise • Step 4: calculate the estimated variance of person effect and residual effect (k = b = 2) S2(people) = [MS(people) – MS(residual)] / k = (6.458 - 0.792) / 2 = 2.833 variance component for person S2(residual) = MS(residual) = 0.792 variance component for residual

  21. Exercise • Step 5: calculate the Cronbach’s alpha coefficient • Alpha coefficient = [ k2 S2(people) ] / [k2 S2(people) + kS2(residual) ] = (22 * 2.833) / (22 * 2.833 + 2 * 0.792) = 0.877 • Alpha coefficient > 0.7 • We may conclude that our questionnaire has internal c\onsistency.

More Related