1 / 37

MG4J: Managing Gigabytes for Java Exercise

MG4J: Managing Gigabytes for Java Exercise. Ida Mele. Document. Indexing in MG4J is centered around documents Package: it.unimi.di.big.mg4j.document The object document, which is the instance of the class Document , represents a single document that can be indexed

semah
Download Presentation

MG4J: Managing Gigabytes for Java Exercise

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MG4J: Managing Gigabytes for JavaExercise Ida Mele

  2. Document • Indexing in MG4J is centered around documents • Package: it.unimi.di.big.mg4j.document • The object document, which is the instance of the class Document, represents a single document that can be indexed • Different documents have different number and type of fields. • For example, • E-mail: from, to, date, subject, body • HTML page: title, url, body MG4J - exercise

  3. Document • Summary of methods: MG4J - exercise

  4. DocumentCollection • Package: it.unimi.di.big.mg4j.document • DocumentCollectionis a randomly addressable lists of documents MG4J - exercise

  5. FileSetDocumentCollection • Package: it.unimi.di.big.mg4j.document • The main method of FileSetDocumentCollection allows to build and serialize a set of documents specified by their filenames MG4J - exercise

  6. Document Factory • Package: it.unimi.di.big.mg4j.document • The factory turns a pure stream of bytes (file) into a document made by several fields (title and text) MG4J - exercise

  7. Standard MG4J Document Factories • CompositeDocumentFactory • HtmlDocumentFactory • IdentityDocumentFactory • MailDocumentFactory • PdfDocumentFactory • ReplicatedDocumentFactory • PropertyBasedDocumentFactory • TRECHeaderDocumentFactory • ZipDocumentCollection.ZipFactory MG4J - exercise

  8. Query • Package:it.unimi.di.big.mg4j.query • To query the index we can use the main method of the class Query • We can submit queries by using: • command line • web browser • QueryEngine: The query engine receives the query and returns the ranked list of results • HttpQueryServer: A simple web server for query processing MG4J - exercise

  9. Indexing and querying: exercise • TECHNICAL REQUIREMENTS: • UNIX Operating System • Java (>=6) • Document collection and the libraries are available at: • http://www.dis.uniroma1.it/~mele/WebIR.html MG4J - exercise

  10. Set the classpath • Download and extract htmlDIS.tar.gz • Download and extract lib.zip • Download the file set-classpath.sh • Edit the first line of the file set-classpath.sh: replace your_directory with the path of the folder containing all the .jar files (lib folder) • Set the CLASSPATH: source set-classpath.sh MG4J - exercise

  11. Building the collection of documents (1) • Help: • java it.unimi.di.big.mg4j.document.FileSetDocumentCollection --help • Create the collection: • find htmlDIS -iname \*.html | java it.unimi.di.big.mg4j.document.FileSetDocumentCollection -f HtmlDocumentFactory -p encoding=UTF-8 dis.collection • findreturns the list of files, one per line. This list is provided as input to the main method of the FileSetDocumentCollection MG4J - exercise

  12. Building the collection of documents (2) • We need also to specify a factory (the -f option) and the encoding as a property • The name of the collection is dis.collection • The collection does not contain the files, but only their names • Deleting or modifying files of htmlDIS directory may cause inconsistence in the collection MG4J - exercise

  13. Building the index • Help: java it.unimi.di.big.mg4j.tool.IndexBuilder --help • Create the index: • java it.unimi.di.big.mg4j.tool.IndexBuilder --downcase -S dis.collection dis • --downcase: this option forces all the terms to be downcased • -S : specifies that we are producing an index for the specified collection. If the option is omitted, Index expects to index a document sequence read from standard input • dis: basename of the index • If you have memory problem, you can use -Xmx for allocating more memory to Java: • java -Xmx512M it.unimi.di.big.mg4j.tool.IndexBuilder --downcase -S dis.collection dis MG4J - exercise

  14. Index files (1) • dis-{text,title}.terms: contain the terms of the dictionary. One term per line • more dis-text.terms • dis-{text,title}.stats: contain statistics • more dis-text.stats • dis-{text,title}.properties: contain global information • more dis-text.properties MG4J - exercise

  15. Index files (2) • dis-{text,title}.frequencies: for each term, there is the number of documents with the term (-code) • dis-{text,title}.globcounts: for each term, there is the number of occurrence of the term (-code) • dis-{text,title}.offset:for each term, there is the offset (-code) MG4J - exercise

  16. Index files (3) • dis-{title,text}.sizes: contain the list of the document sizes. The document size is the number of words contained in each document (- code) • dis-{text,title}.batch<i>: temporary files with sub-indices (-code). Use the option --keep-batchesto not delete temporary files • dis-{text,title}.index:contain the index (-code) MG4J - exercise

  17. Web server • Help: • java it.unimi.di.big.mg4j.query.Query --help • Querying the index: • java it.unimi.di.big.mg4j.query.Query -h -i FileSystemItem -c dis.collection dis-text dis-title • Command line: {text, title} > computer • Web browser: http://localhost:4242/Query MG4J - exercise

  18. Query (1) • Search one word: The result is the set of documents that contain the specified word • Example: computer • AND: more than one term separated by whitespace or by AND or &. The result is the set of documents that contain all the specified words • Example: computer science • Example: computer AND science • Example: computer & science MG4J - exercise

  19. Query (2) • OR: more than one term separated by OR or |. The result is the set of documents that contain any of the given words • Example: conference | workshop • NOT: the operator NOT or ! is used for negation • Example: conference & ! workshop • Parentheses: the parentheses are used to enforce priority in complex queries • Example: university & (rome | california) MG4J - exercise

  20. Query (3) • Proximity restriction: the words must appear within a limited portion of the document • Example: (university rome)~6 • Phrase: using “ ” we can look for documents that contain the exact phrase • Example: “university of rome la sapienza” • Ordered AND: more than one term separated by < • Example: computer < science < department MG4J - exercise

  21. Query (4) • Wildcard (*): wildcard queries can be submitted appending * at the end of a term • Example: infor* • Index specifiers: prefixing a query with the name of an index followed by : you can restrict the search to that index • Example: title:computer • Example: text:computer science AND title:FOCS MG4J - exercise

  22. Sophisticated queries (1) • MG4J provides sophisticated query tuning • To use this features, we must use the command line interface • $ --- to get some help on the available options • Some examples: • $mode--- to choose the kind of results • Example: >$mode short • $selector --- to choose the way the snippet or intervals are shown • Example: > $selector 3 40 MG4J - exercise

  23. Sophisticated queries (2) • Other examples: • $mplex --- when multiplexing is on, each query is multiplexed to all indices. When a scorer is used, it is a good idea to use multiplexing • Example: > $mplex on • $score --- to choose the scorer • Example: > $score VignaScorer • $weight --- to change the weight of the indices. This is useful when multiplexing is on • Example: >$weight text:1 title:3 MG4J - exercise

  24. Scorer (1) • Scorer are important for ranking the documents result of a query. • Default: BM25Scorer and VignaScorer • ConstantScorer. Each document has a constant score (default is 0) • >$score ConstantScorer • CountScorer. It is the product between the number of occurrences of the term in the document and the weight assigned to the index • >$score CountScorer MG4J - exercise

  25. Scorer (2) • TfIdfScorer. It implements TF/IDF • TF is the term frequency of the term t for the document d: c/l; where c is the number of occurrences of t in d and l is the length of d • IDF is the inverse document frequency of the term t in the collection: log(N/f); where N is the number of documents in the collection and f is the number of documents where t appears • >$score TfIdfScorer MG4J - exercise

  26. Scorer (3) • DocumentRankScorer. The scores of documents are stored in a text file • >$score DocumentRankScorer nameFile MG4J - exercise

  27. Virtual fields (1) • A virtual field produces pieces of text that refer to other documents (possibly belonging to the collection) • Referrer: the document that is referring to another document • Referee: the document to which a piece of text of the Referrer is referring to • Intuitively, the Referrer gives us information about the Referee • The Referrer produces in a virtual field a number of fragments of text, each referring to a Referee • The content of a virtual field is a list of pairs made by the piece of text (called virtual fragment) and by some string that is aimed at representing the Referee (called the document spec) MG4J - exercise

  28. Virtual fields (2) • In the case of the HTML document: • the document spec is a URL (as specified in the href attribute) • the virtual fragment is the content of the anchor element and some surrounding text (anchor context) • The HTMLDocumentFactoryproduces the pairs (document spec, virtual fragment) MG4J - exercise

  29. Virtual fields (3) • Create the list of URL of the documents in the collection: • java it.unimi.di.big.mg4j.tool.ScanMetadata -S dis.collection -u dis.urls • Create the document resolver. It is able to map the document spec produced by some document factory into actual references to documents in the collection • Given a document spec, the resolver will decide whether the spec really refers to a document in the collection or not, and in the first case it will find out to which document the spec refers to: • java it.unimi.di.big.mg4j.tool.URLMPHVirtualDocumentResolver -o dis.urls dis-anchor.resolver MG4J - exercise

  30. Virtual fields (4) • Building the index: • java it.unimi.di.big.mg4j.tool.IndexBuilder -a -v anchor:dis-anchor.resolver --downcase -S dis.collection dis • Querying the index: • java it.unimi.di.big.mg4j.query.Query -h -i FileSystemItem -c dis.collection dis-text dis-title dis-anchor • {text, title, anchor} > anchor:conference • {text, title, anchor} > title:combinatorial algorithms AND anchor:conference • {text, title, anchor} > text:RoboCup AND anchor:info MG4J - exercise

  31. Virtual gap (1) • All the virtual fragments that refer to a given document of the collection are like a single text, called virtual text • Virtual fragments coming from different anchors are concatenated, and they are in a text file • This may produce false positive results • For example, the query • anchor:(computer AND science) • produces as result a list of documents that contain both the words in some of their anchors, but not necessarily in the same anchor MG4J - exercise

  32. Virtual gap (2) • To avoid such kinds of false positives, we can use virtual gaps • The virtual gap is a positive integer, representing the virtual space left between different virtual fragments • For example, if the virtual gap is 64 (the default), anchors are concatenated by leaving 64 “empty words” between subsequent fragments • We can submit the query: • >anchor:(computer AND science)~64 • and we will be sure that only documents containing both the term in the same anchor are retrieved MG4J - exercise

  33. Virtual gap (3) • If the anchor is longer than 64 characters, we can still have false positives • In the indexing phase, it is possible to specify a different virtual gap • For example, we can use: • java it.unimi.di.big.mg4j.tool.IndexBuilder -a -g anchor:100 -v anchor:dis-anchor.resolver --downcase -S dis.collection dis • It uses 100 characters for the virtual gap MG4J - exercise

  34. Term map (1) • A simple representation of a dictionary is the term list (the file .terms): a text file containing the whole dictionary, one term per line, in index order (the first line contains the term with index 0, the second line the term with index 1, etc.) • A more efficient representation is based on a monotone minimal perfect hash function: it is a very compact data structure that is able to answer to the question "What is the index of the term XXX?” • You can build such a function from a sorted term list using: • java it.unimi.dsi.sux4j.mph.MinimalPerfectHashFunction titles.mph dis-title.terms MG4J - exercise

  35. Term map (2) • Monotone minimal perfect functions have a serious limit: they can answer correctly to the question "What is the index of the term XXX?”but only if the term appears in the dictionary • To solve this problem, we can use a signed function • For terms not in the dictionary, the function will answer with a special value (-1) that means "the word is not in the dictionary” • java it.unimi.dsi.util.ShiftAddXorSignedStringMap titles.mph titles.map mycollection-title.terms MG4J - exercise

  36. Term map (3) • Wildcard searches require the use of a prefix map • A prefix map is able to answer correctly to the question "What are the indices of terms starting with the characters YYY?” • If terms are lexicographically sorted, the answer is a pair of integers, representing the first and the last index of terms satisfying the property • We can build a prefix map by using: • java it.unimi.dsi.util.ImmutableExternalPrefixMap -b4Ki -o dis-title.terms dis-title.dict MG4J - exercise

  37. Homework Read the MG4J (big) manual: http://www.dis.uniroma1.it/~mele/teaching/WebIR/manual-mg4j.pdf Repeat the exercise Create your own document collection, build the inverted index (with or without virtual fields), then submit some queries and try the different scorers MG4J - exercise

More Related