1 / 13

A Fully Automated Object Extraction System for the World Wide Web

A Fully Automated Object Extraction System for the World Wide Web. a paper by David Buttler, Ling Liu and Calton Pu, Georgia Tech. Why’d they do it?. Identifying object regions and boundaries has been done manually and with some automation mostly relying on syntactic knowledge (ie HTML).

dillan
Download Presentation

A Fully Automated Object Extraction System for the World Wide Web

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Fully Automated Object Extraction System for the World Wide Web a paper by David Buttler, Ling Liu and Calton Pu, Georgia Tech

  2. Why’d they do it? • Identifying object regions and boundaries has been done manually and with some automation mostly relying on syntactic knowledge (ie HTML). • Embley, Jiang & Ng (Hmmm… must be some famous scientists in Germany) developed a pretty sweet heuristics-based automatic object extraction system, which we want to copy but throw out the ontology heuristic – and maybe throw in a few ideas of our own. 

  3. Omini (not the book after Jarom) • Fully-automated extraction • Parses a page into a tree structure • Locates smallest subtree with all objects • Reduces possibilities for next step • Finds correct object separator tags • Contributions to IE • A few algorithms for subtree extraction and object extraction • Most the other stuff is already known

  4. Some Terms & Definitions • Well-Formed Web Document • No brackets besides tags • ALL tags are paired (even <br>,<hr>, etc.) • Attribute values in a tag are in quotes • Nested tags do not overlap • Well-Formed Doc  Tag Tree

  5. System Architecture

  6. Phase 2, Part A: Subtree Extraction • 3 Heuristics used to find the minimal subtree containing all objects of interest • Fanout • Content Size • Tag Count

  7. Phase 2, Part B: Object Separator Extraction • Combination of 5 Heuristics • SD (Standard Deviation) & RP (Repeating Pattern) are taken from BYU. • SB (Sibling tag), PP (Partial Path) are new. • IPS (Identifiable Path Separator) is an extension of BYU’s IT (Identifiable Tag).

  8. Phase 2, Part B Continued: Object Separator Heuristics • SD – Distance between consecutive occurrences of a candidate tag. (Objects usually the same size.) • RP – Absolute value of difference between pairs of tags together and alone. (Pattern of tags usually means just one thing.) • IPS – Ranks tags according to a table of common object separators.

  9. Phase 2, Part B Continued: Object Separator Heuristics • SB – Pairs of tags that are immediate siblings of minimal subtree. (ie <p><a>…</a><b>…</b><c>…</c></p> (# object separators should = # objects) • PP – Counts occurrences of same path of tags from a node. (Multiple instances of object should have same object structure.)

  10. Phase 2, Part B Continued: Object Separator Heuristics • Combining Heuristics • Probability that tag <tr> is an object separator if 3 heuristics say 78%, 63% and 85%: 99% • 78+63+85-78*63-78*85-63*85+78*63*85 = 99% • Combination of all 5 heuristics is best.

  11. Phase 3: Object Extraction • Candidate Object Construction • Uses Object Separator Tag from Phase 2 • Object Extraction Refinement • Removes objects that may not be of the same structure, too big or too small

  12. Results • Ran Omini on 1,500 pages across 25 sites • Using the combination of all 5 heuristics: • 94% of Object Separators picked correctly • 100% Precision and 98% Recall • vs BYU • Omini as good if not better in all tests • Over 5 websites in March 2000: • BYU: 59% success rate • Omini: 93% success rate

  13. Criticism of BYU System • IT (Identifiable Tag) vs IPS (Identifiable Path Separator): • IPS changes tag table based on the node at which the minimal subtree is anchored. • PP (Partial Path) vs HC (Highest Count): • By itself, HC not very successful • In combination with other heuristics, HC can actually make the total accuracy worse! • PP just like HC on some websites • Ontology approach uses human intervention – if goal is fully automated, this won’t do.

More Related