1 / 51

Web Content Mining

Web. Knowledge. Web Content Mining. 17.3.2016. Najlaa Gali and Pasi Fränti. Web. Web Mining. Goals: Automatic extraction of useful information from web Challenging task: heterogeneity and lack of structure Three types: Web usage mining : Discovery of user patterns from web usage logs

kellit
Download Presentation

Web Content Mining

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Web Knowledge Web Content Mining 17.3.2016 Najlaa Gali and Pasi Fränti

  2. Web

  3. Web Mining Goals: • Automatic extraction of useful information from web • Challenging task: heterogeneity and lack of structure Three types: • Web usage mining: Discovery of user patterns from web usage logs • Web structure mining: Discovery from the structure of links. • Web content mining: Discovery from content: text and images.

  4. Applications 1 • To gather, categorize, organize and provide the best possible information available on the WWW to the user requesting the information. • Keyword (or term) based association analysis. • Topic classification. • Similarity detection • Cluster pages by a common author • Cluster pages containing information from common source

  5. Applications 2 • Sequence analysis: predicting a recurring event, discovering trends. • Event detection and tracking. • Help to understand the users behavior. • Anomaly detection: find information that violates usual patterns. • Discovery of frequent phrases. • Text segmentation. • Produce a higher quality of information to the user based upon the requests made through examining images, content, formats and web structure (improving quality of search results). • Businesses can maximize the use of this text mining to improve marketing of their sites as well as the products they offer.

  6. Content of Web PageHypertext Markup Language (HTML, XHTML) Logo image Navigation bar Title Keywords Images Text

  7. Components of Web page • Web page is created from : Hypertext Markup Language (HTML), Cascading Style Sheets (CSS) and Java script (JS). • HTML: describes the structure of a website. • CSS: define the look and layout of text and other material. • Script such as Java script: affect the behavior of HTML web pages.

  8. HTML • A standard markup language used to create web pages. • Its elements consist of tags enclosed in angle brackets (like <html>) • The elements form the building blocks of all websites. • The HTML allows images and objects to be embedded and can be used to create interactive forms. • Can embed scripts and styling sheets. <!DOCTYPE html> <html> <head> <title>This is a title</title> </head> <body> <p>Welcome to DAA++</p> </body> </html>

  9. Example of HTML elements • Title: <title>The Title</title> • Heading: <hx>Heading level</hx>, x= 1…6 • Paragraph: <p>Paragraph</p> • Link: <ahref="https://www.daa.com/">A link to DAA++!</a> • Line break: <p>This <br> is a paragraph <br> with <br> line breaks</p> • Image: <img src=“/documents/PasiFranti.jpg” style=“width: 80px; height: 104px;”>

  10. CSS • Designed primarily to enable the separation of document content from document presentation, including aspects such as the layout, colors, and fonts. • Often used to set the visual style of web pages and user interfaces written in HTML and XHTML. • Enable multiple HTML pages to share formatting by specifying the relevant CSS in a separate .css file, so that to reduce the complexity and repetition in the structural content with every node.

  11. How to style using CSS • HTML presentational attributes (without CSS) <h1><font color="red"> DAA++</font></h1> • CSS style properties <h1style="color:red">DAA++</h1> • Link to external styling sheet <link href="path/to/file.css" rel="stylesheet"> • Internal styling <html> <head> <style> #xyz { color:red } </style> </head> <body> <pid="xyz"> Hello DAA++</p> </body> </html>

  12. Javascript • One of the three essential technologies (HTML,CSS, JS) of WWW content production. • Adds client-side behavior to HTML pages such as animation of page elements, Interactive content (games), and playing audio and video. • Validating input values of a web form to make sure that they are acceptable before being submitted to the server. <script> document.body.appendChild (document.createTextNode('Hello World!')); var h1 =document.getElementById('header'); // holds a reference to the <h1> tag h1 =document.getElementsByTagName('h1')[0]; // accessing the same <h1> </script> <noscript>Your browser either does not support JavaScript, or has it turned off. </noscript>

  13. DOM

  14. DOM Concept • DOM makes all components of a web page accessible • HTML elements • their attributes • text • They can be created, modified and removed with JavaScript. html head body meta meta h1 title p ul a li li

  15. DOM objects • DOM components are accessible as objects or collections of objects • DOM components form a tree of nodes • relationship parent node – children nodes • Attributes of elements are accessible as text • Browsers can show DOM visually as an expandable tree

  16. Example of DOM html INTERSPORT DW SPORTS Unit 49a The Circus, Cabot Circus BS1 3BD BRISTOL, United Kingdom <div> <h3>INTERSPORT DW SPORTS</h3> < p> <span>Unit 49a TheCircus, CabotCircus </span><br> <span>BS1 3BD</span> <span>BRISTOL, United Kingdom</span> </p> </div> DOM tree div h3 p INTERSPORT DW SPORTS span span span BS1… BRISTOL… Unit 49 a…

  17. Text nodes in DOM • Text node • Can only be as a leaf in DOM tree • it’s nodeValue property holds the text • innerHTML can be used to access the text <p> This is text <a href="/path/page.html">link in it</a>. </p> p This is text a Link in it

  18. Goal

  19. Summary Extraction • Title “Rosso restaurant”, “City pharmacy” • Keywords “restaurant, food, lunch, dinner” • Representative Image • Short description ma-pe: 16.00-22.00 la: 12.00-22.00 puh. 013 227 874

  20. Web Page Title <title>Wentworth House Hotel Bath Hotels - Cheap Hotels in Bath, Somerset, UK</title> • Title Tag (91 %) • Logo image (89 %) • Web page body (93 %)

  21. Title and Meta Tags • The obvious source for titling • But includes also additional information • <title> Piato Restaurant– 123 Blues Point Road, McMahons Point, Sydney | Visit Piato and experience the life & flavour of Europe. North Sydney Functions. North Sydney Restaurants.</title> • <title> Joensuu Keskusta | Intersport - Sport to the people </title> • Segmentation is needed! Joensuu Keskusta Intersport Sport to the people

  22. Work flow https://www.jdwetherspoon.com/pubs/all-pubs/england/london/the-coronet-holloway The coronet

  23. Extract Content of Title and Meta tags <title>The Coronet, Holloway | Our Pubs | J D Wetherspoon</title> <meta name="keywords" content="The Coronet" />

  24. Segmentation by delimiters <title>Sydney Waterfront Restaurant | Restaurant Milsons Point -Aqua Dining</title> <title>SIGNORELLI GASTRONOMIA - Pyrmont Italian Restaurant - EAT • DRINK • SHOP • COOKItalian Restaurant Pyrmont Sydney –Signorelli Gastronomia</title> <title>Neutral Bay Club | Tennis, Bowls, Bistro & Functions | Sydney</title> <title>The Coronet, Holloway | Our Pubs | J D Wetherspoon</title> Pre-defined delimiter patterns:

  25. Candidate Segments <title>The Coronet, Holloway | Our Pubs | J D Wetherspoon</title> <meta name="keywords" content="The Coronet" /> • Candidates • The Coronet • Holloway • Our Pubs • J D Wetherspoon

  26. Scoring-Position in Title and Meta Tags <title>The Coronet, Holloway | Our Pubs | J D Wetherspoon</title> • Appear first or last either in Title or Meta gets 0.1 0.1 0.0 0.0 0.1 0.1 <meta name="keywords" content="The Coronet" /> • Candidates • The Coronet • Holloway • Our Pubs • J D Wetherspoon 0.1 0.0 0.0 0.1

  27. Popularity Among Header Tags <h1 class="banner-inner__title">The Coronet</h1> <h2 class="venue-finder__title-text">Find a pub or hotel</h2> <h2 class="venue-finder__title-text">Our Pubs</h2> <h2 class="venue-finder__title-text" ng-hide="isPubName">Check out your nearest pub or hotel</h2> <h3 class="feature-panel__title">Discover our food menu</h3> <h3 class="feature-panel__title">Our drinks selection</h3> <h4 class="tab__title">Nearby J D Wetherspoons</h4> • Candidates • The Coronet 1  6 = 6 • Holloway • Our Pubs 1  5 = 5 • J D Wetherspoon 1  3 = 3 Frequency Weight

  28. Scoring-Position in Web Link Domain Path File name https:// www.jdwetherspoon.com/ pubs/all-pubs/england/london/ the-coronet-holloway 1 3 1.5 Dice similarity measure • Candidates • The Coronet • Holloway • Our Pubs • J D Wetherspoon 3 × 0.70 = 2.1 3 × 0.58 = 1.74 1.5 × 0.00 = 0.00 1 × 1.00 = 1.00

  29. Rank Segments Normalizing

  30. Impact of criteria • Criteria 1 has the lowest impact (0.65) • More generic words such as home and welcome are often placed at the beginning • Either the slogan, address or general information about the web page is placed at the end of the title. • Criteria 2 has slightly higher impact (0.68) • Heading tags are not always used, and even when existing, the correct title is not always there. • Criterion 3 is statistically significant in comparison with criteria 1 and 2.

  31. Web Page Body Content of text nodes N-grams (n=1…6) Filter by part-of-speech (POS) patterns

  32. Construct DOM tree Body div div h2 h1 p div Aqua Dining Sydney Waterfront Restaurant |… Aqua Dining offers a… div h3 h5 a Feeling social?.. a facebook Navigation

  33. Extract text nodes Navigation Feeling Social? Find us on Facebook Sydney Waterfront Restaurant Restaurant Milsons Point Aqua Dining offers a quintessential Sydney dining experience with unrivalled harbour views that sweep from Luna Park to the world famous Sydney Harbour Bridge and the Sydney Opera House.

  34. Apply POS tagging Navigation Feeling Social? Find us on Facebook Sydney Waterfront Restaurant Restaurant Milsons Point Aqua Dining offers a quintessential Sydney dining experience with unrivalled harbour views that sweep from Luna Park to the world famous Sydney Harbour Bridge and the Sydney Opera House. NNP=Proper noun, singular NNPS=Proper noun, plural NN=Noun, singular or mass VBG=Verb, gerund VB=Verb, base form PRP=Personal pronoun DT=Determiner CC=Coordinating conjunction JJ=Adjective NNP VBG VB NNP PRP IN NNP NNP NN NNP NNPS NNP NNP JJ NN NN NNP NNP VBZ DT NNP IN NNS IN NN NN DT NNP JJ WDT IN NNP NN JJ NNP CC DT NNP NNP NNP NNP NNP

  35. Extract potential phrases Navigation Feeling Social? Find us on Facebook Sydney Waterfront Restaurant Restaurant Milsons Point Aqua Dining offers a quintessential Sydney dining experience with unrivalled harbour views that sweep from Luna Park to the world famous Sydney Harbour Bridge and the Sydney Opera House. NNP=Proper noun, singular NNPS=Proper noun, plural NN=Noun, singular or mass VBG=Verb, gerund VB=Verb, base form PRP=Personal pronoun DT=Determiner CC=Coordinating conjunction JJ=Adjective NNP VBG VB NNP PRP IN NNP NNP NN NNP NNPS NNP NNP JJ NN NN NNP NNP VBZ DT NNP IN NNS NN NN IN NNP JJ DT WDT IN NNP NN JJ NNP CC DT NNP NNP NNP NNP NNP

  36. Feature extraction • Similarity with the link of the web page • Appearance in title tag • Appearance in meta tag • Popularity on the web page (frequency) • Appearance in heading (h1, h2…h6) tags • Capitalization • Capitalization frequency • Independent appearance • Phrase length

  37. Similarity with web link https://www.aquadining.com.au/

  38. Appearance in Title tag <title>SydneyWaterfront Restaurant | Restaurant Milsons Point - AquaDining</title>

  39. Appearance in Meta title tag <metaproperty="og:title" content="SydneyWaterfront Restaurant | Restaurant Milsons Point - AquaDining" />

  40. Appearance in Header tags <h1class="site-title">AquaDiningSydney Restaurant</h1> <h2>AquaDining</h2> Weight= 6 Weight= 5

  41. Popularity on the web page <title>SydneyWaterfront Restaurant | Restaurant Milsons Point - AquaDining</title> <metaproperty="og:title" content="SydneyWaterfront Restaurant | Restaurant milsonspoint - AquaDining" /> <h1class="site-title">AquaDiningSydney Restaurant</h1> <h2>AquaDining</h2>

  42. Capitalization

  43. Capitalization frequency <title>Sydney Waterfront Restaurant | Restaurant Milsons Point - AquaDining</title> <metaproperty="og:title" content="Sydney Waterfront Restaurant | Restaurant milsonspoint - AquaDining" /> <h1class="site-title">aquaDiningSydney Restaurant</h1> <h2>aquaDining</h2>

  44. Independent appearance <title>Sydney Waterfront Restaurant | Restaurant Milsons Point - Aqua Dining</title> <meta property="og:title" content="Sydney Waterfront Restaurant | Restaurant milsons point - Aqua Dining" /> <h1 class="site-title">aqua Dining Sydney Restaurant</h1> <h2>aqua Dining</h2>

  45. Length

  46. Normalization

  47. Classifiers • Naive Bayes • Support Vecotr Machines (SVM) • Clustering • K-nearest neighbors (k-NN)

  48. Results with Titler corpus Extracted titles

  49. Results with Mopsi Services Annotated titles

  50. Logo Image • ~89 % of web pages have their title within a logo image • Needs to detect logo image • Apply OCR • Challenging !!!

More Related