a (corny) ending

1. a (corny) ending

2. Course Outcomes • After this course, you should be able to answer: • How search engines work and why are some better than others • Can web be seen as a collection of (semi)structured databases? • If so, can we adapt database technology to Web? • Can useful patterns be mined from the pages/data of the web? What did you think these were going to be?? REVIEW

3. Main Topics • Approximately three halves plus a bit: • Information retrieval • Information integration/Aggregation • Information mining • other topics as permitted by time REVIEW

4. Adapting old disciplines for Web-age Information (text) retrieval Scale of the web Hyper text/ Link structure Authority/hub computations Databases Multiple databases Heterogeneous, access limited, partially overlapping Network (un)reliability Datamining [Machine Learning/Statistics/Databases] Learning patterns from large scale data Social Networks REVIEW

5. Clustering (2) Text Classification (2) Filtering/Recommender Systems (1.5) Computational Advertising (1) XML and handling semi-structured data + Semantic Web standards (3) Information Extraction(2.5) Information/data Integration (1.5) Topics Covered • Introduction (2) • Text retrieval; vectorspace ranking (3) • Indexing; tolerant retrieval; (1) • Correlation analysis & Latent Semantic Indexing (3) • Link Analysis in Web Search (A/H; Pagerank) (4) • Crawling; Map/Reduce (2) Social Networks (3)

6. Big Idea 1 Finding“Sweet Spots” in computer-mediated cooperative work • It is possible to get by with techniques blythely ignorant of semantics, when you have humans in the loop • All you need is to find the right sweet spot, where the computer plays a pre-processing role and presents “potential solutions” • …and the human very gratefully does the in-depth analysis on those few potential solutions • Examples: • The incredible success of “Bag of Words” model! • Bag of letters would be a disaster ;-) • Bag of sentences and/or NLP would be good • ..but only to your discriminating and irascible searchers ;-) Giving “pointers” where results can be found and letting users do the “reading” is okay for simple queries But for aggregate queries, it becomes tiresome Here read these 700 employee files to figure out the average employee salary.

7. Collaborative Computing AKA Brain Cycle StealingAKA Computizing Eyeballs Big Idea 2 • A lot of exciting research related to web currently involves “co-opting” the masses to help with large-scale tasks • It is like “cycle stealing”—except we are stealing “human brain cycles” (the most idle of the computers if there is ever one ;-) • Remember the mice in the Hitch Hikers Guide to the Galaxy? (..who were running a mass-scale experiment on the humans to figure out the question..) • Collaborative knowledge compilation (wikipedia!) • Collaborative Curation • Collaborative tagging • Paid collaboration/contracting • Many big open issues • How do you pose the problem such that it can be solved using collaborative computing? • How do you “incentivize” people into letting you steal their brain cycles?

8. Tapping into the Collective UnconsciousAKA “Wisdom of the Crowds” Big Idea 3 • Another thread of exciting research is driven by the realization that WEB is not random at all! • It is written by humans • …so analyzing its structure and content allows us to tap into the collective unconscious .. • Meaning can emerge from syntactic notions such as “co-occurrences” and “connectedness” • Examples: • Analyzing term co-occurrences in the web-scale corpora to capture semantic information (today’s paper) • Analyzing the link-structure of the web graph to discover communities • DoD and NSA are very much into this as a way of breaking terrorist cells • Analyzing the transaction patterns of customers (collaborative filtering)

9. If you don’t take Autonomous/Adversarial Nature of the Web into account, then it is gonnagetcha.. Big Idea 4 • Most “first-generation” ideas of web make too generous an assumption of the “good intentions” of the source/page/email creators. The reasonableness of this assumption is increasingly going to be called into question as Web evolves in an uncontrolled manner… • Controlling creation rights removes the very essence of scalability of the web. Instead we have to factor in adversarial nature.. • Links can be manipulated to change page importance • So we need “trust rank” • Fake annotations can be added to pages and images • So we need ESP-game like self-correcting annotations.. • Fake/spam mails can be sent (and the nature of the spam mails can be altered to defeat simple spam classifiers…) • So we need adversarial classification techniques • Sources may export untrustworthy/made-up data • So we need SourceRank?

10. Interactive Review

13. Anatomy may be likened to a harvest-field. • First come the reapers, who, entering upon untrodden ground, cut down great store of corn from all sides of them. These are the early anatomists of Europe • Then come the gleaners, who gather up ears enough from the bare ridges to make a few loaves of bread. Such were the anatomists of last. • Last of all come the geese, who still contrive to pick up a few grains scattered here and there among the stubble, and waddle home in the evening, poor things, cackling with joy because of their success. Gentlemen, we are the geese. --John Barclay English Anatomist

14. Information Integration on Web still rife with uncut corn • Unlike anatomy of Barclay’s day, Web is still young. We are just figuring out how to tap its potential • …You have great stores of uncut corn in front of you. • …… go cut some of your own!