1 / 57

Digging Deep for Hidden Information in the Web

Digging Deep for Hidden Information in the Web. Part 1: Automated blog analysis Part 2: Automated hyperlink analysis. Part 1 Automated Blog Analysis. Analysing Public Science Debates through Blogs and Online News Sources. Part 1 Contents. Background Blogs Online news sources RSS

rkreps
Download Presentation

Digging Deep for Hidden Information in the Web

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Digging Deep for Hidden Information in the Web Part 1: Automated blog analysis Part 2: Automated hyperlink analysis

  2. Part 1 Automated Blog Analysis Analysing Public Science Debates through Blogs and Online News Sources

  3. Part 1 Contents • Background • Blogs • Online news sources • RSS • Tracking public science debates • Detecting public science debates

  4. Background Blogs, public opinion, online news, RSS

  5. Background • There are millions of bloggers • Bloggers are almost normal human beings • Automatically tracking bloggers’ postings may give insights into public opinion

  6. Blog tracking companies • IBM • WebFountain • Intelliseek • BlogPulse • “Monitor, measure and leverage consumer-generated media” • Others growing…

  7. RSS Format • Rich Site Syndication/Really Simple Syndication • XML technology • Used for frequently updated information sources (blogs, news, academic journals) • RSS Readers • Users subscribe to the RSS feeds of favourite blogs/sites/journals/searches • Notified when updates available • User-controlled ‘push’ technology

  8. Tracking Public Science Debates

  9. Blog keyword searches • Technorati “Searches weblogs by keyword and for links” • Stem cell research • Blogdigger • stem cell research • IceRocket • Allows Advanced searches • Allows genuine date range search (Google only allows “last updated” date range searches)

  10. Track evolution over time • What is changing about interest in Stem cell research/GM food? • Are experts good at identifying changes in public interest? • How can experts be sure/can they be supported with quantitative information? • Can blogs be used to generate time series reflecting changes in “public interest”?

  11. Free science debate graphs • Solves the trend identification problem? • Blogpulse Offers free automatic blog searches and keyword-generated click-search graphs • Stem cell research • GM food • Mobile phone radiation

  12. Research graphs • Time-consuming to collect data • Give control over the data source

  13. Detecting Public Science Debates

  14. How to detect a new debate? • Heuristic methods • E.g. Read papers, scan relevant blogs • Automatic methods • E.g. look for sudden increase in usage of science-related words in blogs?

  15. Free hot topic searches • Blog keyword search (sort by date) • Technorati “Searches weblogs by keyword and for links” • Stem cell research • Blogdigger blog search • Hot topic searches • Blogdex – top contagious information • Bloglines – today’s hot topics (most popular links) • Searches find the really big science debates?

  16. Specialist research tools • Commercial software • Intelliseek/IBM • Mozdeh RSS monitor • Generates sub-collections • Generates word time series • Allows keyword searches • Identifies hot topics

  17. Mozdeh Science Concern Corpus • A collection of blog postings containing a fear word AND a science word • Trend detection used to identify hot “science fear” topics • Data cleaning to remove spam • Need manual scanning of list of words experiencing biggest usage increase

  18. Classification of top 5 words

  19. Classification of top 200 words The words come from multiple stories 7.5% of top 200 Words Represent new public fears of Science stories E.g. new medical cure

  20. Unexpected results? • Social science research • Sudden burst of discussion over fears of the economic theories of Karl Rove, an influential advisor to George Bush • Computer security • Concern over spyware features in a software vendor’s products • Research showing that consumers’ pin numbers could be revealed by poor printing

  21. Conclusions Many free tools support exploration of Consumer Generated Media Also room for specialist research tools

  22. References • http://www.blogpulse.com/ • http://www.blogpulse.com/www2006-workshop/ • http://www.creen.org/ • Thelwall, M., Prabowo, R. & Fairclough, R. (2006, to appear). Are raw RSS feeds suitable for broad issue scanning? A science concern case study. Journal of the American Society for Information Science and Technology.

  23. Acknowledgement • The work was supported by a European Union grant for activity code NEST-2003-Path-1. It is part of the CREEN project (Critical Events in Evolving Networks, contract 012684, http://www.creen.org/)

  24. Part 2: Automated hyperlink analysis Link analysis as a social science technique

  25. Link Analysis Manifesto • Links are: • A wonderful new source of information about relationships between people, organisations and information • An easy to collect data source • But: • Results should be interpreted with care

  26. Part 2 Contents • Academic link analysis –mainly from an information science perspective • A general social science link analysis methodology • Commercial applications

  27. Why Count Links? • Individual hyperlinks may reflect connections between web page contents or creators • Counts of large numbers of hyperlinks may reflect wider underlying social processes • Links may reflect phenomena that have previously been difficult to study • E.g. informal scholarly communication

  28. Why Count University Links? • To map patterns of communication between researchers in a country • Which universities collaborate a lot? • Which universities collaborate with government or industry? • Which universities are using the web effectively?

  29. Counting links • Search engines will count them for you! • Yahoo! advanced queries, e.g. Links from Wolves Uni. to Oxford Uni. Or back • domain:ox.ac.uk AND linkdomain:wlv.ac.uk • domain:wlv.ac.uk AND linkdomain:ox.ac.uk • Google link queries • Find links to specific URLs, e.g. links to the University home page link:www.wlv.ac.uk

  30. Counting links • Can use a special purpose web crawler or robot • Visits all the pages in a web site • Counts the links in the site • Can use “advanced” counting methods

  31. Some Inter-University Hyperlink Patterns Mainly for the UK and Europe

  32. Links to UK universities against their research productivity The reason for the strong correlation is the quantity of Web publication, not its quality This is different to citation analysis

  33. Most links are only loosely related to research • 90% of links between UK university sites have some connection with scholarly activity, including teaching and research • But less than 1% are equivalent to citations • So link counts do not measure research dissemination but are more a natural by-product of scholarly activity • Cannot use link counts to assess research • Can use link counts to track an aspect of communication

  34. UK universities tend to link to their neighbours

  35. Universities cluster geographically

  36. Language is a factor in international interlinking • English the dominant language for Web sites in the Western EU • In a typical country, 50% of pages are in the national language(s) and 50% in English • Non-English speaking extensively interlink in English

  37. Patterns of international communication Counts of links between EU universities in Swedish are represented by arrow thickness.

  38. Counts of links between EU universities in French are represented by arrow thickness.

  39. Which language???

  40. Which language???

  41. Which language? Who is isolated?

  42. International link patterns • The next slide is a (Kamada-Kawai) network of the interlinking of the “top” 5 universities in AEAN countries (Asia and Europe) with arrows representing at least 100 links and universities not connected removed.

  43. The rich get richer on the web • Link creation obeys the ‘rich get richer’ law • Sites which already have a lot of links attract the most new links • Some sites have a huge number of links: most have one or none

  44. Rich get richer example: Links from Australian university pages The anomalies are also interesting

  45. Part 3: A General Social Science Link Analysis Methodology • A general framework for using link counts in social sciences research • For research into link creation or • Together with other sources, for research into other online or offline phenomena • Applicable when there are enough links relevant to the research question to count • For collections of large web sites or • For large collections of small web sites

  46. Nine stages for a research project • Formulate an appropriate research question, taking into account existing knowledge of web structure • Conduct a pilot study • Identify web pages or sites that are appropriate to address the research question

More Related