1 / 27

Informant Accuracy

This study explores the accuracy of cognitive reports in social network data and examines if accuracy changes based on the structural property being described. It analyzes four different data sets and concludes that informants are not very accurate in reporting their communication patterns.

wendysimpson
Download Presentation

Informant Accuracy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Informant Accuracy Christopher McCarty

  2. The Problem • Social network data often rely on cognitive reports such as ‘Name the three people you interact with most in this group’ • How closely do these cognitivereports represent behavior? • Does accuracy change based on the structural property a researcher is trying to describe?

  3. Informant Accuracy Study 1Peter D. Killworthand H. R. Bernard (1976) Informant Accuracy in Social Network Data. Human Organization 35 (3): 269-286 • 21 people from a regional group of deaf respondents who relied on teletype to communicate with other group members • Number of lines to and from each member were recorded (behavior) • Respondent ranked all 31 members in order of their perceived (cognitive) • The reported most communicated with was only in rank 1-4 52% of the time

  4. Only 24% of the time will a person who ranks j 5th will they actually talk to them 2nd-8th

  5. Informant Accuracy Study 2H. R. Bernard and Peter D. Killworth (1977) Human Communication Research 4 (1): 3-18 • Four data sets • Deaf – New data set of teletype data from two groups of 30 deaf respondents in Washington DC. Pre-cognitive and post-cognitive data • Hams - 44 Ham radio operators with recorded lines transmissions, and 0-9 scale assessment of interaction with each of 54 users • Office – Cognitive data from 40 people in an office with observed (behavioral) data about 44. Observers walked through office every 15 minutes for five hours every workday • Tech – 37 peopel in a graduate technology program at WVU were observed (behavioral) and reported interaction (cognitive)

  6. Conclusions Study 2 • People do not know, with any accuracy, those with whom they communicate • Post-cognitive data more accurate than pre-cognitive, but not significantly • Except for first ranked, no difference between a typical week and last week • No difference between data sets • No demographics that explain inaccuracy (e.g. women are no more accurate than men) • Keeping logs does not improve accuracy • Adjusting by respondents’ confidence in their assessment makes no difference

  7. Analysis of BK data in UCINET

  8. Informant Accuracy Study 3Peter D. Killworth and H. Russell Bernard (1979) Informant Accuracy in Social Network Data III: A Comparison of Triadic Structure in Behavioral and Cognitive Data, Social Networks 2: 10-46

  9. Number/proportion of triads reported correctly

  10. Assumption that behavior changes because cognition changes is dangerous • Researcher returning at time 2 would likely find different triads and conclude something had changed • Change could be from inaccurate report from time 1 to an inaccurate report at time 2

  11. Conclusions Study 3 • A lot of significant structure in both the behavioral and cognitive data (what triadic structures don’t exist?) • Structures from ranked behavioral data are similar to ranked cognitive data; and scaled behavioral data are similar to scaled cognitive data; but structures produced by ranking and those produced by scaling are different • A 76% level of scatter exists between behavioral and cognitive structures when compared triad by triad • Null hypothesis that cannot be rejected: Social structure never changes; only reports of it by informants changes due to informant inaccuracy

  12. Informant Inaccuracy Study 4H. R. Bernard, Peter D. Killworth and Lee Sailer (1979) Informant Accuracy in Social Network Data IV: A Comparison of Clique-Level Structure in Behavioral and Cognitive Network Data, Social Networks 2: 191-218 • Compared the same four data sets using four clique-finding algorithms: • Factor Analysis • CONCOR • COMPLT

  13. Never more than 50% concordance (Hams on COMPLT) between recalled and observed cliques

  14. Conclusions Study 4 • At dyadic level 50% inaccuracy found (Study 2) • At triadic level 76% inaccuracy found (Study 3) • At the clique level 160% inaccuracy was found (Study 4) • “We must therefore recommend unreservedly that any conclusions drawn from data gathered by the question “Who do you talk to?” are of no use in understanding social structure or communication.”

  15. Informant Accuracy Study 5H.R. Bernard, Peter D. Killworth and Lee Sailer (1982) Informant Accuracy in Social-Network Data V. An Experimental Attempt to Predict Actual Communication from Recall Data, Social Science Research 11:30-66 • Examination of recall of communication over time • Are respondents more accurate depending on the time period reported on? • Does a respondent’s position within the network determine accuracy?

  16. Data • Electronic Information Exchange System (EIES), the precursor to e-mail • Quotes Page 35 and 36 • 57 paid users reported on their communication in each of 37 windows, including their “last-on” • Respondents also rated their confidence in their rating

  17. -Of those recalled, 30% were not communicated with (T1P)-Two thirds of the people a respondent received messages from were forgotten (T2P)-79% of those recalled by a respondent are wrong in some way (T12AB)-More than 52% of the time respondents chose the wrong most frequent (TOP1)-More than 40% of the top 3 should not be in the top 3 (TOP3)

  18. Conclusion to Study 5 • Respondents are inaccurate at reporting no matter the lag in time • Respondents were not accurate at communicating with nobody • However, while individual respondents were not accurate, en masse reports match behavior about broad facts, such as the person spoken to most frequently

  19. Linton C. Freeman, A. Kimball Romney and Sue Freeman (1987) Cognitive Structure and Informant Accuracy, American Anthropologist, 89: 310-325 • Data collected from colloquium series at UCI, 9 sessions • 33 people interviewed about whether they had attended the last session, and to name the others who attended • Observers recorded who actually attended

  20. Results • 141 recall errors out of 272 opportunities (51.8%) (very close to BKS results) • However reports were better at describing the long-term pattern than the detail about individual attendees • Errors were biased in the direction of long-term patterns, similar to BKS findings that en masse the group gets some things right • Some people were better than others at describing the pattern, presumably those who attended regularly and were part of the in-group • Used Consensus Analysis to analyze results

  21. Informant Accuracy Overall Conclusions • When asking respondents who they talk to in a bounded (whole) network, when they talk to them and how frequently they talk to them, one should be careful • Are people more accurate at reporting their personal network interactions?

More Related