1 / 8

Hitting a Moving Target: Historical Language Change in Information Retrieval

Hitting a Moving Target: Historical Language Change in Information Retrieval. Miles Efron Graduate School of Library and Information Science University of Illinois, Urbana-Champaign mefron@illinois.edu http:// people.lis.illinois.edu/~mefron. The Setting: Digitized Book Repositories.

jeneva
Download Presentation

Hitting a Moving Target: Historical Language Change in Information Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hitting a Moving Target: Historical Language Change in Information Retrieval Miles Efron Graduate School of Library and Information Science University of Illinois, Urbana-Champaign mefron@illinois.edu http://people.lis.illinois.edu/~mefron

  2. The Setting: Digitized Book Repositories Google Books: 15 million books Hathi Trust: 8.4 million books Project Gutenberg: 20,000 books … Growing quickly General research problem: How to support information retrieval over such corpora? Specific problem: How to support information retrieval in corpora that see substantive linguistic change?

  3. The Setting: Query by Passage When April with its sweet showers Even a fool, when he holds his peace is counted wise. Many hands make light work.

  4. The Setting: Historically Diverse Corpora When April with its sweet showers… April It’s April WhenApril with his showers sweet with fruit April comes & April goes Whan that Aprille with his shouressote April 8 Like Aprillshowre so stream the trickling teares sootestormis of Aprillevntoδe rote

  5. Two Strategies for Resolving Language Mismatch When April with its sweet showers… Whan that Aprille with his shouressote Spelling correction Machine Translation Problems Spelling: creature wight Translation: out of vocabulary knowledge

  6. Combining Approaches via the Noisy Channel Model For a modern word m, we have the probability that the “correct” old word is o given spelling correction evidence so and translation (i.e. dictionary) evidence do. Build a query model based on these conditional probabilities

  7. Example Query: sweet showers Query Model: 0.428 swete 0.369 sweet 0.024 sote 0.061 showrs 0.0592 shoures 0.0592 showres Top Results: ...OF CAUNTERBURY. Whan that Aprille with his SHOURES soteThe droghte of Marche hath perced... ...like this: Whan that Aprille with his SHOURES sote: Whanthat Apreele with 'is shoores... ...by grace; Gifts from heaven fall in SHOW'RS, Cheering dark and lonely hours, By our pathway bloom SWEETflow'rs, When we're saved by grace. E'en...

  8. Interested? Contact Me. Miles Efron Graduate School of Library and Information Science University of Illinois, Urbana-Champaign mefron@illinois.edu http://people.lis.illinois.edu/~mefron

More Related