Integration of Friendly Data Islands on the Web. Information Extraction.

Integration of Friendly Data Islands on the Web.Information Extraction.

Roadmap • Introduction • What extraction rules are • Generating extraction rules • A couple of systems • Conclusions

User Interface Controller Data Layer The theory • A wrapper is a building block that provides an ad-hoc, message-based API to an app • They interface apps at one or more layers, but, more often than not, they must deal with the user interface or the data layer Business Logic Data AccessLayer

The Da Vinci Code Dan Brown Doubleday, 2006 15.95 € Robert Langdon is a Harvard Professor of Symbology… Buy The problem

Features of current web documents • Trillions of documents • Generated on demand by software applications • Change continuously • Require navigation from search forms • Written in telegraphic language • Formatted according to HTML templates

The solution Wrapping

Wrapping in a nutshell • Goals • Endow data islands with APIs • Ease implementing software applications • Implications • Form filling • Navigation • Info extraction • “Ontologisation”

Look out! • Information extraction has driven most research efforts • Few wrapping systems are complete • Wrapping is usually mistaken for information extraction • This talk is about engineering information extraction for enabling information integration

Document Extraction rules B1 The Da Vinci Code A1 Dan Brown Templating/ Ontologisation rules 15.95 € 2006 The Da Vinci Code P1 Doubleday Doubleday 2006 Dan Brown 15.95 € Robert Langdon… Robert Langdon… Message ID: MUC-0001 Message Template: Court resolution Date of Event: April, 30 2007 Charge: Terrorist attack Perpetrator: Salahuddin Amin Perpetrator: Anthony Garcia Perpetrator: Waheed Mahmood Perpetrator: Omar Khyam … Attributes Ontology instances Templates How IE works Information extractor

Roadmap • Introduction • What extraction rules are • Generating extraction rules • A couple of systems • Side by side comparison • Conclusions

Running example

Running example <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 Text: blah, blah </li> <li> Reviewer: Alan Wohl Rating: 8 Text: yeah, yeah </li> </ul></body></html> <!–- Sample #3 --><html><body> Book name: W4F explained Reviews: <ul> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 Text: cough, cough </li> </ul></body></html>

Kinds of extraction rules • Regular expressions • First-order logic rules • Pointers into DOM tree • Context-free grammars • Tag trees

RoadRunner $FileName<html><body> Book name: $BookTitle Reviews: <ul> (( <li> Reviewer: $ReviewerName Rating: $Rating Text: $Text </li> )+)? </ul></body></html> Regular expressions TSIMMIS [Root, get("page.html"), "#"] [BookReview, Root, "<body>#</body>"] [BookName, BookReview, "# "] [Tmp, Rook, "<ul>#</ul>"] [Reviews, Tmp, "split(Tmp, '<li>')"] [ReviewerNames, Reviews, "Reviewer:# "] [Ratings, Reviews, "Rating:# "] [Text, Reviews, "Text:# "]

First-order logic rules SRV bookTitle(X) :- prev(X, "Bookname:"), next(X, " "). reviewerName(X) :- prev(X, "name:"),next(X, " "), !bookTitle(X). rating(X) :- isNatural(X), length(X, 1), inList(X). text(X) :- prev(X, "Text:"),next(X, "</li>").

Pointer into the DOM tree WebOQL select x’.Text, y’.Text, y’’’’.Text, y’’’’’’’.Text from x, y in browse("page.html") where x.Text = "Book name:" and y.Text = "Reviewer:"

Context-free grammars Minerva Page ::= $FileName <html><body> Review </body></html> Review ::= Book name: $BookName Reviews: <ul> (<li> Reviewer Rating Text <li>)* </ul> Reviewer ::= Reviewer: $Reviewer Rating ::= Rating: $Rating Text ::= Text: $Text

Tag trees DEPTA li b br b br b

Classification • Hand-crafted • Supervised induction • Little-supervised induction • Unsupervised induction

Hand-crafted • Techniques • Natural intelligence • Systems • TSIMMIS • Minerva • WebOQL • W4F • XWrap The pattern to extract the title is “…”

Raw documents Labelled documents Automated induction Supervised induction • Techniques • Bottom-up ILP • Top-down ILP • Ad-hoc algorithms • Systems • SRV • RAPIER • WIEN • WHISK • NoDoSE • SoftMealy • STALKER • DEByE

Automated induction Little-supervised induction • Techniques • String alignment • Tree alignment • Systems • OLERA • Thresher Raw document Record and attribute labelling

Raw documents Automated induction Pattern interpretation Unsupervised induction • Techniques • String alignment • Tree alignment • Statistical roles • Systems • DeLa • RoadRunner • EXALG • DEPTA • IEPAD

Roadmap • Introduction • What extraction rules are • Generating extraction rules • A couple of systems • RoadRunner • SRV • Conclusions

Token matching <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> String mistmatch $1

...and matching… <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> Tag match $1<html>

...and matching… <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> Tag match $1<html><body>

...and matching… <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> Tag match, string match, … $1<html><body> Book name:

...and matching… <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> String mismatch, tag match $1<html><body> Book name: $2

...and matching… <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> … $1<html><body> Book name: $2 <ul> <li> Reviewer: $3 Rating: $4 Text: $5 </li>

Stop: lists and optionals $1<html><body> Book name: $2 <ul> <li> Reviewer: $3 Rating: $4 Text: $5 </li> <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> Tag mismatch

Stop: lists and optionals $1<html><body> Book name: $2 <ul> <li> Reviewer: $3 Rating: $4 Text: $5 </li> <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html>

Stop: lists and optionals $1<html><body> Book name: $2 <ul> (<li> Reviewer: $3 Rating: $4 Text: $5 </li>)+ <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html>

…and matching finishes <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 … </li> <li> Reviewer: Alan Wohl Rating: 8 … </li> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 … </li> </ul></body></html> $1<html><body> Book name: $2 <ul> (<li> Reviewer: $3 Rating: $4 Text: $5 </li>)+ </ul></body></html>

Just union-free grammars!

Roadmap • Introduction • What extraction rules are • Generating extraction rules • A couple of systems • RoadRunner • SRV • Conclusions

Exercise • Support predicates: next(x,y), previous(x,y) • Try to explain isCorD(X) abcabdab bbcaabda

Exercise • Support Predicates: next(x,y), previous(x,y) • Now, try to Explain isCorDorE(X) abcabdabee bbcaabdaee

Define target predicates Target Predicates title: #PCDATA. reviewer: #PCDATA. rating: #PCDATA. text: #PCDATA.

Instantiate target predicates <!–- Sample #1 --><html><body> Book name: Ontologies Reviews: <ul> <li> Reviewer: John Doe Rating: 7 Text: blah, blah </li> <li> Reviewer: Alan Wohl Rating: 8 Text: yeah, yeah </li> </ul></body></html> <!–- Sample #3 --><html><body> Book name: W4F explained Reviews: <ul> </ul></body></html> <!–- Sample #2 --><html><body> Book name: SPARQL in action Reviews: <ul> <li> Reviewer: Dan Smith Rating: 9 Text: cough, cough </li> </ul></body></html>

Instantiate target predicates Positive Samples Negative Samples title("Ontologies"). title("SPARQL in action"). title("W4F Explained"). reviewer("John Doe"). reviewer("Alan Wohl"). reviewer("Dan Smith"). rating("7"). rating("8"). rating("9"). text("blah, blah"). text("yeah, yeah"). text("cough, cough"). !title("Book name:"). !reviewer("Book name:"). !rating("Book name:"). !text("Book name:"). !title("Reviews:"). !reviewer("Reviews:"). !rating("Reviews:"). !text("Reviews:"). !title("Reviewer:"). !reviewer("Reviewer:"). !rating("Reviewer:"). !text("Reviewer:"). !title("Rating:"). !reviewer("Rating:"). !rating("Rating:"). …

Define support predicates Support Predicates prev: #PCDATA, #PCDATA. next: #PCDATA, #PCDATA. length: #PCDATA, #PCDATA. isNatural: #PCDATA.

On Negative Samples prev("Book name:", ""). next("Book name:", ""). length("Book name:", 10). !isNatural("Book name:"). prev("Reviews:", ""). next("Reviews:", ""). !isNatural("Reviews:"). prev("Reviewer:", ""). next("Reviewer:", ""). !isNatural("Reviewer:"). prev("Rating:", ""). next("Rating:", ""). !isNatural("Rating:"). … Instantiate support predicates On Positive Samples prev("Ontologies", ""). next("Ontologies", " "). length("Ontologies", 10). !isNatural("Ontologies"). prev("SPARQL in action", ""). next("SPARQL in action", " "). length("SPARQL in action", 16). !isNatural("SPARQL in action"). prev("W4F explained", ""). next("W4F explained", " "). length("W4F explained", 16). !isNatural("W4F explained"). …

Top-down induction title(X) :- . (3, 14) title(X) :- prev(X, X). (0, 0) title(X) :- prev(X, ""). (0, 5) title(X) :- !prev(X, X). (3, 14) title(X) :- !prev(X, ""). (3, 9) title(X) :- prev(X, Y). (3, 14) title(X) :- prev(X, ""). (3, 9) title(X) :- !prev(X, Y). (?, ?) title(X) :- !prev(X, ""). (0, 5) title(X) :- next(X, X). (0, 0) … title(X) :- !next(X, X). (3, 14) title(X) :- next(X, Y). (3, 14) title(X) :- !next(X, Y). (?, ?) title(X) :- length(X, X). (0, 0) …

Rule selection Combined covering New covering Old covering p0 = # positive bindings of R n0 = # negative bindings of R p1 = # positive bindings of R&A n0 = # negative bindings of R&A t = # positive bindings of both R and R&A

Induction goes on… title(X) :- . (3, 14) title(X) :- prev(X, Y). (3, 14) title(X) :- prev(X, Y), X = Y. (?, ?) title(X) :- prev(X, Y), X != Y. (?, ?) title(X) :- prev(X, Y), prev(X, X). (?, ?) title(X) :- prev(X, Y), !prev(X, X). (?, ?) title(X) :- prev(X, Y), prev(X, Z). (?, ?) title(X) :- prev(X, Y), !prev(X, Z). (?, ?) title(X) :- prev(X, Y), prev(Y, X). (?, ?) …

…and on… title(X) :- . (3, 14) title(X) :- prev(X, Y). (3, 14) title(X) :- prev(X, Y), Y = "". (?, ?) title(X) :- prev(X, Y), Y = "", prev(X, X). (?, ?) title(X) :- prev(X, Y), Y = "", !prev(X, X). (?, ?) title(X) :- prev(X, Y), Y = "", prev(Y, Y). (?, ?) title(X) :- prev(X, Y), Y = "", !prev(Y, Y). (?, ?) title(X) :- prev(X, Y), Y = "", prev(X, Z). (?, ?) title(X) :- prev(X, Y), Y = "", !prev(X, Z). (?, ?) …

Integration of Friendly Data Islands on the Web. Information Extraction.

Integration of Friendly Data Islands on the Web. Information Extraction.

Presentation Transcript

Integration of Information Extraction with an Ontology

Information Extraction and Integration

Web Data Extraction

Scalable Information Extraction and Integration

Extraction of Opinions on the Web

Information Network Analysis and Extraction Extraction and Integration of the Semi-Structured Web

Integration of Information Extraction with an Ontology

Data Integration on the Semantic Sensor Web

Open Information Extraction from the Web

Integration on the Web

The Semantic Web-Week 22 Information Extraction and Integration (continued)

Information Extraction on the Web

Data Extraction and Integration from Imprecise Web Sources

Chapter 15: Data Integration on the Web

Web scale Information Extraction

Information Extraction from Multimedia Content on the Social Web

Webs of Data and Data on the Web, the Deep Web, Data Discovery, Data Integration

Web Scraping ,Data Scraping,Web Extraction,Data Extraction - USA

Extraction of Product Specifications on the Web

Information Extraction and Integration

Web data extraction tool