1 / 67

Guillaume Cabanac guillaumebanac@univ-tlse3.fr

Guillaume Cabanac guillaume.cabanac@univ-tlse3.fr. Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics http:// bit.ly /rguCabanac2012. March 28th, 2012. Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac. Outline of these Musings.

billie
Download Presentation

Guillaume Cabanac guillaumebanac@univ-tlse3.fr

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Guillaume Cabanac guillaume.cabanac@univ-tlse3.fr Musings at the Crossroads ofDigital Libraries, Information Retrieval, and Scientometricshttp://bit.ly/rguCabanac2012 March 28th, 2012

  2. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences

  3. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences

  4. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Question DL-1How to transpose paper-based annotations into digital documents? DL IR Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity SCIM Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Collective annotation: Perspectives for information retrieval improvement.”RIAO’07 : Proceedings of the 8th conference on Information Retrieval and its Applications, pages 529–548. CID, may 2007.

  5. 1541 1630 1790 1830 1881 1998 Annotated bible (Lortsch, 1910) Fermat’s last theorem (Kleiner, 2000) Annotations from Blake, Keats… (Jackson, 2001) Les Misérables Victor Hugo US students (Marshall, 1998) Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac From Individual Paper-based Annotation … • Characteristics of paper annotation • Secular activity: older than 4 centuries • Numerous applicative contexts: theology, science, literature … • Personal use: “active reading”(Adler & van Doren, 1972) • Collective use: review process, opinion exchange …

  6. Web servers a discussion thread Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac … to Collective Digital Annotations hardcopy Hard to share  ‘lost’ author 87% (Ovsiannikov et al., 1999) 13% reader > 20 annotation systems(Cabanac et al., 2005) ComMentor … iMarkup … Yawas … Amaya … Annotation server 1993 2005

  7. a reader’s comment discussionthread Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Digital Document Annotation: Examples • W3C Annotea / Amaya(Kahan et al., 2002) • Arakne, featuring “fluid annotations”(Bouvin et al., 2002)

  8. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Collective Annotations • Reviewed 64 systems designed during 1989–2008 • Collective Annotation • Objective data • Owner, creation date • Anchoring point within the document. Granularity: all doc, words… • Subjective information • Comments, various marks: stars, underlined text… • Annotation types: support/refutation, question… • Visibility: public, private, group… • Purpose-oriented annotation categories Annotation remark Annotation reminder Annotation argumentation Personal Annotation Space

  9. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Question DL-2How to measure the social validity ofa statement according to the argumentative discussion it sparked off? DL IR Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity SCIM Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Social validation of collective annotations : Definition and experiment.” Journal of the American Society for Information Science and Technology, 61(2):271–287, feb. 2010, Wiley. DOI:10.1002/asi.21255

  10. Social Validation Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Social Validation of Argumentative Debates • Scalability issue  • Which annotationsshould I read? • Social validation = degree of consensus of the group

  11. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Social Validation of Argumentative Debates • Informing readers about how validated each annotation is Before Annotation magma After Filtered display

  12. case 1 case 2 case 3 case 4 Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Social Validation Algorithms • Overview • Two proposed algorithms • Empirical Recursive Scoring Algorithm (Cabanac et al., 2005) • Bipolar Argumentation Framework Extension • based on Artificial Intelligence research works (Cayrol & Lagasquie-Schiex, 2005) A A B B validity – 1 socially refuted 1socially confirmed 0socially neutral

  13. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Social Validation Algorithm • Example • Computing the social validity of a debated annotation

  14. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Validation with a User-study • Aim: social validation vs human perception of consensus • Design • Corpus: 13 discussion threads= 222 annotations + answers • Task of a participant • Label opinion type • Infer overall opinion • Volunteer subjects 119 53

  15. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Experimenting the Social Validation of Debates • Q1 Do people agree when labeling opinions? • Kappa coefficient (Fleiss, 1971; Fleiss et al., 2003)Inter-rater agreement among n > 2 raters • Weakagreement, with variability subjective task Fair to good agreement Value of Kappa Poor Debate Id

  16. HP – SV Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Experimenting the Social Validation of Debates • Q2 How well SV approximates HP? • HP = Human Perception of consensus • SV = Social Validation algorithm 1. Test whether PH and VS are different (p < 0.05) Student’s paired t-test: (p = 0,20) > (a = 0,05) 2. Correlate HP et SV  Pearson’s coefficient of correlationrr(HP, SV) = 0.48 shows a weak correlation Density y = p(HP – SV) example: HP = SV for 24 % of all cases Density

  17. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Question DL-3How to harness a quiescent capital present in any community: its documents? DL IR Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity SCIM Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Organization of digital resources as an original facet for exploring the quiescent information capital of a community.”International Journal on Digital Libraries, 11(4):239–261, dec. 2010, Springer. DOI:10.1007/s00799-011-0076-6

  18. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Documents as a Quiescent Wealth • Personal Documents • Filtered, validated, organized information… • … relevant to activities in the organization • Paradox: profitable, but under-exploited • Reason 1 –  folders and files are private • Reason 2 –  manual sharing • Reason 3 –  automated sharing • Consequences • People resort to resources available outside of the community • Weak ROI  why would we have to look outside when it’s already there?

  19. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac How to Benefit from Documents in a Community? • Mapping the documents of the community • SOM [Kohonen, 2001] Umap [Triviumsoft] TreeMap [Fekete & Plaisant, 2001]… • Limitations  Find the documents with same topics as D  Find documents that colleagues use with D  concept of usage: grouping documents⇆keeping stuff in common

  20. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac How to Benefit from Documents in a Community? • Organization-based similarities • inter-folder • inter-document • inter-user

  21. community Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac How to Help People to Discover/Find/Use Documents? • Purpose: Offering a global view of • … people and their documents • Based on document contents • Based on document usage/organization • Requirement: non-intrusiveness and confidentiality • Operational needs • Find documents • With related materials • With complementary materials • Seeking people ⇆seeking documents • Managerial needs • Visualize the global/individual activity • Work position  required documents

  22. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Proposed System: Static Aspect 4 views = {documents, people}  {group, unit} 1. Group of documents • Main topics • Usage groups 2. A single document • Who to liaise with? • What to read? 3. Group of people • Community of interest • Community of use 4. A single people • Interests • Similar users (potential help)

  23. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences

  24. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Question IR-1Is document tie-breaking affecting the evaluation of Information Retrieval systems? DL IR Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators SCIM Guillaume Cabanac, Gilles Hubert, Mohand Boughanem, Claude Chrisment. “Tie-breaking Bias : Effect of an Uncontrolled Parameter on Information Retrieval Evaluation.” M. Agosti, N. Ferro, C. Peters, M. de Rijke, and A. F. Smeaton (Eds.) CLEF’10 : Proceedings of the 1st Conference on Multilingual and Multimodal Information Access Evaluation, volume 6360 de LNCS, pages 112–123. Springer, sep. 2010. DOI:10.1007/978-3-642-15998-5_13

  25. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Measuring the Effectiveness of IR systems • User-centered vs. System-focused[Spärck Jones & Willett, 1997] • Evaluation campaigns • 1958 Cranfield, UK • 1992 TREC (Text Retrieval Conference), USA • 1999 NTCIR (NII Test Collection for IR Systems), Japan • 2001 CLEF (Cross-Language Evaluation Forum), Europe • … • “Cranfield” methodology • Task • Test collection • Corpus • Topics • Qrels • Measures : MAP, P@X ... using trec_eval [Voorhees, 2007]

  26. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Runs are Reordered Prior to Their Evaluation • Qrels = qid, iter,docno, rel Run = qid, iter,docno, rank,sim, run_id relevant[1 ; 127] (N, 0.8), (R, 0.8), (N, 0.5) Reordering by trec_evalqid asc, sim desc, docno desc (R, 0.8), (N, 0.8), (N, 0.5) Effectiveness measure = f (intrinsic_quality, )MAP, P@X, MRR…

  27. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Consequences of Run Reordering • Measures of effectiveness for an IRS s • RR(s,t) 1/rank of the 1st relevant document, for topic t • P(s,t,d) precision at document d, for topic t • AP(s,t)average precision for topic t • MAP(s)mean average precision • Tie-breaking bias • Is the Wall Street Journal collection more relevant than Associated Press? • Problem 1 comparing 2 systems AP(s1, t) vs. AP(s2, t) • Problem 2 comparing 2 topics AP(s, t1) vs. AP(s, t2) Sensitive to document rank Ellen Chris

  28. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac What we Learnt: Beware of Tie-breaking for AP • Poor effect on MAP, larger effect on AP • Measure bounds APRealisticAPConventionnalAPOptimistic • Failure analysis for the ranking process • Error bar = element of chance  potential for improvement padre1, adhoc’94

  29. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Question IR-2How to retrieve documents matching keywords and spatiotemporal constraints? DL IR Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators SCIM Damien Palacio, Guillaume Cabanac, Christian Sallaberry, Gilles Hubert. “On the evaluation of geographic information retrieval systems: Evaluation framework and case study.”International Journal on Digital Libraries, 11(2):91–109, june 2010, Springer. DOI:10.1007/s00799-011-0070-z

  30. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Geographic Information Retrieval • Query = “Road trip around Aberdeen summer 1982” • Search engines • Topic term  {road, trip, Aberdeen, summer} • spatial  {AberdeenCity, AberdeenCounty…} • Geographic temporal  [21-JUN-1982 .. 22-SEP-1982] • term  {road, trip, Aberdeen, summer} •  1/6 queries = geographic queries • Excite (Sanderson et al., 2004) • AOL (Gan et al., 2008) • Yahoo! (Jones et al., 2008) •  Current issue worth studying

  31. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac The Internals of a Geographic IR System • 3 dimensions to process • Topical, spatial, temporal • 1 index per dimension • Topic bag of words, stemming, weighting, comparing with VSM… • Spatial spatial entity detection, spatial relation resolution… • Temporal temporal entity detection… • Query processing with sequential filtering • e.g., priority to theme, then filtering according to other dimensions • Issue: effectiveness of GIRSs vs state-of-the-art IRSs? • Hypothesis: GIRSs better than state-of-the-art IRSs

  32. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Case Study: the PIV GIR System • Indexing: one index per dimension • Topical = Terrier IRS Spatial = tilingTemporal = tiling • Retrieval • Identification of the 3 dimensions in the query • Routing towards each index • Combination of results with CombMNZ [Fox & Shaw, 1993; Lee 1997]

  33. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Case Study: the PIV GIR System • Principle of CombMNZ and Borda Count

  34. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Case Study: the PIV GIR System • Gain in effectiveness

  35. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Question IR-3Do operators in search queries improve the effectiveness of search results? DL IR Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators SCIM Gilles Hubert, Guillaume Cabanac, Christian Sallaberry, Damien Palacio. “Query Operators Shown Beneficial for Improving Search Results.” S. Gradmann, F. Borri, C. Meghini, H. Schuldt (Eds.) TPDL’11 : Proceedings of the 1st International Conference on Theory and Practice of Digital Libraries, volume 6966 de LNCS, pages 118–129. Springer, sep. 2011. DOI:10.1007/978-3-642-24469-8_14.

  36. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Search Engines Offer Query Operators Information need “I’m looking for research projects funded in the DL domain” Regular query Query with operators • Various Operators • Quotation marks, Must appear (+), boosting operator (^),Boolean operators, proximity operators…

  37. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Our Research Questions

  38. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Our Methodology in a Nutshell VN . . . V4 V3 V2 V1: Query variant with operators Regular query        

  39. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Effectiveness of Query Operators • TREC-7 per Topic Analysis: Boxplots • ‘+’ and ‘^’

  40. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Effectiveness of Query Operators • Per Topic Analysis: Box plot 0.4 Query variant highest AP AP of TREC’s regular query AP (Average Precision) 0.2 0.1 0.3 Query variant lowest AP Topics 32

  41. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Effectiveness of Query Operators • TREC-7 Per Topic Analysis • ‘+’ and ‘^’ MAP  = 0.1554 MAP ┬ = 0.2099 +35.1%

  42. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences

  43. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Question SCIM-1How to recommend researchers according to their research topics and social clues? DL IR Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences SCIM Guillaume Cabanac. “Accuracy of inter-researcher similarity measures based on topical and social clues.”Scientometrics, 87(3):597–620, june 2011, Springer. DOI:10.1007/s11192-011-0358-1

  44. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Recommendation of Literature (McNee et al., 2006) • Collaborative filtering • Principle: mining the preferences of researchers •  those who liked this paper also liked… •  Snowball effect / fad •  Innovation? •  Relevance of theme? • Cognitive filtering • Principle: mining the contents of articles profile of resources (researcher, articles) citation graph • Hybrid approach      ????

  45. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Foundations: Similarity Measures Under Study • Model • Coauthors graph authors  auteurs • Venues graph authors  conferences / journals • Social similarities • Inverse degree of separationlength of the shortest path • Strength of the tie number of shortest paths • Shared conferences number of shared conference editions • Thematic similarity • Cosine on Vector Space Model di = (wi1, … , win) built on titles (doc / researcher)

  46. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Computing Similarities with Social Clues • Task of literature review • Requirement topical relevance • Preference social proximity (meetings, project…) •  re-rank topical results with social clues • Combination with CombMNZ (Fox & Shaw, 1993) • Final result: list of recommended researchers Degree of separation CombMNZ Social list Strength of ties  Shared conferences CombMNZ TS list Topical list

  47. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Evaluation Design • Comparison of recommendations and researchers’ perception • Q1 : Effectiveness of topical (only) recommendations? • Q2 : Gain due to integrating social clues? • IR experiments: Cranfield paradigm (TREC…) • Does the search engine retrieve relevant documents? relevance judgments{0, 1} binary[0, N] gradual Doc relevant? corpus search engine x assessor topic input qrels trec_eval Effectiveness measuresMean Average PrecisionNormalized Discounted Cumulative Gain improvement +12.3 % significativity p < 0.05 (paired t-test)

  48. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Evaluating Recommendations • Adaptation of the Cranfield paradigm (TREC…) • Is the search enginerec. sys. Retrieving relevant documentsresearchers? recommender system Top 25 « With whom would you like to chat for improving your research? » relevance judgments{0, 1} binary[0, N] gradual doc relevant ? researcher name of a researcher corpus search engine x topical + social assessor topical topic input qrels trec_eval #subjects Effectiveness measures Mean Average PrecisionNormalized Discounted Cumulative Gain improvement +12.3 % significativity p < 0.05 (paired t-test)

  49. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Experiment • Features • Data dblp.xml (713 MB = 1.3M publications for 811,787 researchers) • Subjects 90 researchers-contacts contacted by mail 74 researchers began to fill the questionnaire. 71 completed it • Interface for assessing recommendations   

  50. Musings at the Crossroads of DL, IR, and SCIMGuillaume Cabanac Experiments: Profile of the Participants • Experience of the 71 subjects Mdn = 13 years • 74 • Productivity of the 71 subjects Mdn = 15 publications Number of participants Seniority Number of participants Number of publications

More Related