Searching in an XML Corpus Using Content and Structure INEX 2003, Germany

1. Searching in an XML CorpusUsing Content and StructureINEX 2003, Germany Yiftah Ben-Aharon, Sara Cohen, Yael Grumbach, Yaron Kanza, Jonathan Mamou, Yehoshua Sagiv, Benjamin Sznajder, Efrat Twito The Hebrew University of Jerusalem

2. Approach • IR techniques were extended in the context of XML corpus • The granularity of the retrieval is refined: fragments of document (and not necessarily whole document) are considered as potential results • The additional information provided by the structure of the document, and of the query, is exploited when retrieving results

3. Approach (cont’d) • An extensible system was built • E.g., new ranking techniques can be added easily • The system was implemented in a short time • E.g., topics are translated into XSL stylesheets • Programming language: Java • Operating System: Windows XP

4. Topic • Only the title of the topic, denoted T, is used for retrieval • We denote • T+ the list of terms in T that are preceded by a + sign • T- the list of terms that are preceded by a - sign • To the list of optional terms • We have implemented our retrieval system only for CO and SCAS topics (not VCAS)

5. Topic Processor Filter Extractor Ranker Merger Topic Result Fragments augmented with ranking scores Relevant documents Relevant fragments Ranker 1 Indices Ranker 2 IEEE Digital Library Ranker 3 Ranker 4 Ranker 5 Indexer

6. Preprocess • XML documents and topics • Terms are stemmed (using Porter stemmer) • Stopwords are eliminated • Indices are built

7. Topic Processor Filter Extractor Ranker Merger Topic Result Fragments augmented with ranking scores Relevant documents Relevant fragments Ranker 1 Indices Ranker 2 IEEE Digital Library Ranker 3 Ranker 4 Ranker 5 Indexer

8. Index • Inverted Keyword Index • Associates each term with the list of documents (id’s) containing it • Keyword-Distance Index • Stores information about distance between two terms over all the sentences in all the documents of the corpus

9. Index • Tag Index • Associates to each tag a weight, according to the “importance” of its content • E.g., the information provided by the front matter is more important than the information provided by a subsection • Inverse Document Frequency Index • Associates to each term its IDF, classical in IR • IDF is the fraction of documents in the corpus containing the term

10. Topic Processor Filter Extractor Ranker Merger Topic Result Fragments augmented with ranking scores Relevant documents Relevant fragments Ranker 1 Indices Ranker 2 IEEE Digital Library Ranker 3 Ranker 4 Ranker 5 Indexer

11. Filter • Documents not containing all the terms of T+ are considered as irrelevant • Documents containing all the terms of T+ are extracted from the corpus

12. Topic Processor Filter Extractor Ranker Merger Topic Result Fragments augmented with ranking scores Relevant documents Relevant fragments Ranker 1 Indices Ranker 2 IEEE Digital Library Ranker 3 Ranker 4 Ranker 5 Indexer

13. Relevant Fragments • Relevant fragments from each document that passed the filtering are extracted • Relevant fragments • CAS: determined by the topic title • CO: the system determines potentially relevant fragments • whole document • front matter • abstract • any section • any subsection

14. Extracting Relevant Fragments from a Document • An XPath processor is not suitable, since the syntax of CAS topics is more general than that of XPath. • The relevant fragments are extracted by means of an XSL stylesheet that is generated from T • For CAS topics, the stylesheet also checks that the returned fragments satisfy the predicates of the title • The implementation of the translator of topics to XSL stylesheets, is fast

15. Topic Processor Filter Extractor Ranker Merger Topic Result Fragments augmented with ranking scores Relevant documents Relevant fragments Ranker 1 Indices Ranker … IEEE Digital Library Ranker … Ranker … Ranker n Indexer

16. An Overview of the Ranking Process • n different rankers give scores based on the structure and the content of the fragments • In our implementation, 5 rankers • For some rankers, the weights of tags are incorporated into the score • Each ranker gives scores to all the fragments returned by the extractor • For each result, the scores of all the relevant fragments are aggregated

17. Word-Number Ranker • This ranker counts the number of terms from T- and To appearing in the fragment • The score is • increased when the number of terms from To is increased • decreased when the number of terms from T- is increased

18. IDF Ranker • We measure the “rarity” of a term using the classical formula of IDF • The score is • increased when the number of rare terms from To isincreased • decreased when the number of rare terms from T- is increased

19. TFIDF Ranker • It is an extension of the Vector Space Model to XML documents • TF counts the number of occurrences of a term in the fragment (and not the whole document) • Each occurrence is multiplied by the weight of its tag • TFIDF = TF * IDF • The score of a fragment is computed by • adding the TFIDF of terms from T+ andTo • subtracting the TFIDF of terms from T-

20. Proximity Ranker • This ranker is based on the correlation between pairs of words from T+ and Toappearing in a single phrase in a sliding window containing 5 terms • Such a pair is called lexical affinity (LA) • The score of a fragment is computed by counting the number of LA’s • The score is increased when a LA appears under “important” tags

21. Similarity Ranker • Idea: If two terms appear frequently in the same sentence in the corpus, they should be considered as related • It is a sort of blind query refinement • The score of a fragment is based on • Distance between the terms of the query and the terms of the fragment • Increases when the pair appears under “important” tags

22. Topic Processor Filter Extractor Ranker Merger Topic Result Fragments augmented with ranking scores Relevant documents Relevant fragments Ranker 1 Indices Ranker 2 IEEE Digital Library Ranker 3 Ranker 4 Ranker 5 Indexer

23. Merger • The scores of the various rankers are merged into a single rank • The main problem is how to determine the relative weight of each ranker • The scores of the 5 rankers are lexicographically sorted as follows • An order among the rankers is determined • A tuple of the 5 scores is created for each result • The tuples are lexicographically sorted

24. Merger (cont’d) • Our submitted results use different orderings of the rankers • E.g., • Word Number • Idf • Similarity • Proximity • TFIDF

25. Conclusion • Our system builds and uses indices • It combines different rankers • The rankers use both the content and the structure • Thesystem is extensible • The implementation uses configuration files • New rankers can be added easily • The system can be easily adapted to changes in the formal syntax of queries

26. Future Works • We still need to experiment thoroughly with the system • Modify the merger by using a single formula to combine the scores of the different rankers • How to determine the relative weight of each ranker? • Add and modify rankers

27. Thank You. Questions?