1 / 42

Torsten Suel Associate Professor CSE Department Polytechnic Institute of NYU suel@poly

Networking. Databases. Distributed Systems. Web Search Engines: Technologies, Applications, and Opportunities. Computer Security. Software Engineering. Torsten Suel Associate Professor CSE Department Polytechnic Institute of NYU suel@poly.edu. Computer Graphics.

misty
Download Presentation

Torsten Suel Associate Professor CSE Department Polytechnic Institute of NYU suel@poly

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Networking Databases Distributed Systems Web Search Engines: Technologies, Applications, and Opportunities Computer Security Software Engineering Torsten Suel Associate Professor CSE Department Polytechnic Institute of NYU suel@poly.edu Computer Graphics

  2. What this talk is about: - web search engines - how they work - underlying technologies - applications and impact - what is next? - opportunities and education

  3. The Basics: • type in some words • get back results (usually 10 at a time) • hopefully good results among these • if not, change your query, try again

  4. The Basics: • type in some words • get back results (usually 10 at a time) • hopefully good results among these • if not, change your query, try again

  5. Web Search Technology • as the web grew, we needed a way to find sites • has grown into many-billion-$ industry at heart of the web • search engines: largest supercomputers of the world • 100’s of millions of queries per day, 0.1 sec latency/query • evaluated over tens of billions of documents • the majors: Google, MSFT Bing, Yahoo!(?), Baidu, Yandex • but many other businesses use similar technologies, or • rely on search engines for their business

  6. Overview of this Lecture: • 1. Search engines in a nutshell: • - How does the web work? • - How do search engines work? • - Basic search architecture • - Historical background • 2. Technical challenges and opportunities: • - link analysis • - computational advertising • 3. Education and opportunities • - the search landscape • - search engines and the curriculum

  7. 1. Search Engines in a Nutshell The Web: text … A lot of text … >100 billion pages of text and other stuff …

  8. What is the web? (another view) • pages containing (fairly unstructured) text • images, audio, etc. embedded in (hanging off) pages • structure defined using HTML • (Hypertext Markup Language) • hyperlinks between pages! • over 100 billion pages • over 3 trillion hyperlinks • a giant graph!

  9. How the web is organized: site structure Web Server (Host) Web Server (Host) www.cnn.com www.poly.edu • pages reside in servers • sites often contain related pages • site/host structure • local versus global links Web Server (Host) www.irs.gov

  10. How Browsing Works Fetching “www.cnn.com/world/index.html” give me the file “/world/index.html” Web Server Desktop (with browser) www.cnn.com here is the file: “...”

  11. GET /world/index.html HTTP/1.0 User-Agent: Mozilla/3.0 (Windows 95/NT) Host: www.cnn.com From: … Referer: … If-Modified-Since: ... HTTP: desktop or crawler web server HTTP/1.0 200 OK Server: Netscape-Communications/1.1 Date: Tuesday, 8-Feb-99 01:22:04 GMT Last-modified: Thursday, 3-Feb-99 10:44:11 GMT Content-length: 5462 Content-type: text/html <the html file>

  12. Basic structure of a search engine: mining & indexing Crawler Index disks look up Search.com Query: “computer”

  13. Crawling Crawler disks • crawler: also called spider, web robot • fetches pages from the web • starts at set of “seed pages” • parses fetched pages for hyperlinks • then follows those links (e.g., BFS) • until all pages fetches (i.e., never)

  14. Data Mining: • data fetched by the crawler is analyzed • many different tasks: • - link analysis (later) • - detection of spam pages and dangerous pages • - analyzing data about past queries (clicks etc.) • - data extraction (products, people, locations, …) • mining becoming increasingly important • needs experts in statistics & machine learning • large-scale mining platforms: • mapReduce (Google), Hadoop, Pig (Yahoo!), Dryad (MSFT), …

  15. Indexing aardvark 3452, 11437, ….. . . . . . arm 4, 19, 29, 98, 143, ... armada 145, 457, 789, ... armadillo 678, 2134, 3970, ... armani 90, 256, 372, 511, ... . . . . . zebra 602, 1189, 3209, ... indexing disks “inverted index” • parse & build lexicon & build index • index very large • I/O-efficient techniques needed

  16. Querying aardvark 3452, 11437, ….. . . . . . arm 4, 19, 29, 98, 143, ... armada 145, 457, 789, ... armadillo 678, 2134, 3970, ... armani 90, 256, 372, 511, ... . . . . . zebra 602, 1189, 3209, ... Boolean queries: (zebra AND armadillo) OR armani compute unions/intersections of lists Ranked queries: zebra, armadillo give scores to all docs in union or intersection of lists look up

  17. Ranked Querying: • return best pages first • term- vs. link vs. log-based approaches • multiple phases, many features • also add meaningful “snippets”

  18. Challenges for search engines: • coverage (need to cover large part of the web) • good ranking (in the case of broad and narrow queries) • freshness (need to update content) • user load (> 50000 queries/sec - Google) • manipulation (sites want to be listed first) need to crawl and store massive data sets smart information retrieval techniques frequent recrawling of content many queries on massive data naïve techniques will be exploited quickly

  19. Summary: • Four main tasks: • - crawling (data acquisition) • - data/web mining • - index building and maintenance • - query execution • highly optimized ranking functions and systems • tens to hundreds of thousands of machines • major engines need hundreds of engineers • and PhD-level experts in stats, ML, systems, etc. • many more jobs in related companies

  20. Historical Roots: Information Retrieval (IR) • “IR is concerned with the representation, storage, • organization of, and access to information items” • focus on automatic processing (indexing, clustering, • search) of unstructured data (text, images, audio, ...) • subfield of Computer Science, but with roots in • Library Science, Information Science, and Linguistics • main focus on text data, but also images, audio, video • applications: • - searching in a library catalog • - categorizing a collection of medical articles by area • - web search engines

  21. Historical Perspective • WW2 era computers built for number crunching: • ballistics computations, code breaking • since earliest days, also used to “organize information” • - Memex (Vannevar Bush, 1945) • today, this is the main application! • - store and organize data and documents • - model organizations and work processes • Computer Organizer (also: communications/media) • … however, no clean separation

  22. Structured vs. Unstructured Data • IR: lesser known cousin of field of Databases • Databases: focus on structured data • IR: unstructured data: “documents” • IR focused on providing info directly to human users • IR has false positives and negatives (is fuzzy) - scientific articles, novels, poems, jokes, web pages

  23. IR Before 1945: • Babylonians, Greeks, Romans, etc. • Indexing & creation of concordances (by hand) • - “algorithms for full-text indexing” ! • Lib. of Cong. and Dewey Library Classifications • Documentalism • Bibliometric and Informetric distributions: • - Bradford, Lotka, Zipf, Pareto, Yule (1920s-40s) • Citation Analysis and Social Network Analysis • Microfilm rapid selectors: (e.g., E. Goldberg 1931) • Memex (Vannevar Bush, 1939/45)

  24. Memex: Vannevar Bush (1890-1974) • “As We May Think”, Atlantic Monthly, 1945 (mostly written 1939)

  25. IR Techniques and Applications: • Querying documents by keywords • Classifying documents by topic • Hypertext and analyzing links between documents • Library catalogs and digital libraries • Searching a collection of news or medical articles • National security: • - analyzing communication streams, financial networks • Most widely used application: web search!

  26. 2. Technical Challenges and Opportunities: Graphs and Social Networks • What is a graph? • - nodes and edges • - edges directed or undirected • Graphs are used to model many scenarios • - social relationships: who knows whom, who is friends with whom? • - citations in literature: which physicist cites which other physicist? • - email, telephone: who communicates with whom? • - follow the money: who gives money to whom? • - the web: who links to whom?

  27. Example: Scientific Literature • nodes are researchers • connected by edge if one • cites the other • From: University of Cottbus

  28. Example: Enron Email Network • nodes are employees • connected by edge if they • exchanged more than 5 emails • From: Shetty/Adibi (USC)

  29. Social Networks: Why do we care? • search engines: use hyperlink graph to improve ranking • national security: who calls whom and what does it mean? • social sciences: understanding societies

  30. Link-Based Ranking Techniques • Basic idea: exploits judgments by millions of web pages • A page that is highly referenced should be better or • more important • Pagerank (Brin&Page at Google) • “significance of a page depends on • significance of those referencing it” • s(a) = s(b)/2 + s(c)/3 + s(d)/1 • System of equations • Unique solution under some assumptions

  31. 0.2 Pagerank 1/2 1/2 0.2 1 0.2 1/2 1/2 1/2 1 1/2 0.2 0.2 • initialize the rank value of each node to 1/n(0.2 for 5 nodes) • a node with k outgoing links transmits a 1/k fraction of • its current rank value over that edge to its neighbor • iterate this process many times until it converges • NOTE: this is a random walk on the link graph • Pagerank: stationary distribution of this random walk

  32. (1) (2) 0.2 0.2 1/2 1/2 1/2 1/2 0.2 1 0.2 0.3 1 0.1 1/2 1/2 1/2 1/2 1/2 1/2 1 1 1/2 1/2 0.2 0.2 0.2 0.2 (3) (n) 0.3 0.286 1/2 1/2 1/2 1/2 0.25 1 0.1 .. 0.286 1 0.143 1/2 1/2 1/2 1/2 1/2 1/2 1 1 1/2 0.2 1/2 0.15 0.143 0.143

  33. A Matrix notation 0.2 0 1/2 0 0 1/2 0 0 1/2 0 1/2 0 0 0 1 0 0 0 1/2 0 1/2 1 0 0 0 0 1/2 1/2 0.2 1 0.2 1/2 1/2 1/2 1 1/2 0.2 0.2 • stationary distribution: vectorxwith xA = x • A is primitive, and x Eigenvector of A • computed using Jacobi or Gauss Seidel iteration

  34. Now every engine is using link-based ranking • But the idea is much older! • Who is the most important physicist? • - citation analysis, 1960s • Who is the most important person in town? • - social network analysis, 1950s • Who is the most important person on Facebook? • - the most friends? • - or the most important friends? (recursive) • - or the best friends? • …. and on Twitter ?

  35. Computational Advertising: • New field dealing with mathematical techniques • for targeting ads in electronic media • a multi-billion $ business • economic foundation of search engines and web • many startups in New York City • ITV: TV is moving to the internet • how to match ads with the right consumers • scary privacy implications

  36. Comp. Adv. Basics • Search ads: matching ads to queries • - ads on top and right hand side of search results • - based on query and past behavior (cookies etc) • - pay-per-click model with bids by advertisers • - or pay-per-action: goal is immediate action by user • Display ads: large, shiny ads for brands • - banner ads on major sites for cars, movies, etc. • - per-par-display: e.g., $100 per million impressions • - sold by contract: e.g., ads for upcoming movies • AdSense (Google): text ads on 3rd-party pages • - placing ads on pages based on content and user • - Google sharing money with site owner • - danger of manipulation by site owner

  37. Comp. Adv. Marketplace • Very complicated zoo of companies and roles • - ad networks • - ad campaign coordination/optimization • - companies providing user data • - arbitrage & manipulation • - hundreds/thousands of companies • - real-time auctions in tens of millisecs • Emerging ads scenarios • - monetizing social networks (facebook, linkedIn etc.) • - mobile ads and ads in app-space (e.g., flurry) • - ITV and internet radio: ads not a broadcast (hulu) • - ads in games and virtual worlds • Privacy: looking bad at the moment • - (almost) everything is for sale …

  38. Adversarial Information Retrieval • search engine positions is $$$ • web pages are cheap • make lots of automatic junk … • use ads to make $ (or $$$)

  39. 3. Education and Opportunities: • Web search at center of the online world • interesting technical challenges • many professional opportunities • NYC is a center of this industry • Google, plus media, ads, e-commerce, mobile • what qualifications are needed?

  40. Perspective: AI machine learning natural language processing systems information retrieval library & information science web search algorithms databases

  41. In the curriculum • Search is somewhat Computer Science oriented • … it’s all software (mostly) • also, media/design exp. becoming important • courses/topics to learn: • - algorithms • - distributed systems • - databases • - web search & information retrieval • - machine learning, data mining, and statistics • - natural language processing

  42. Course Offering & Student Activities • CS6913: Web Search Engines (Spring Sem.) • Course Objectives • web and search engine architecture how does it all work? • working with massive data sets storing and analyzing terabytes • introduction to Information Retrievalunstructured data, text • system building skills building distributed systems • the Web as a social network adversarial behavior, spam, communities • Student Activities • course projects build your own (small-scale) search engine • independent research projects, theses, assistantships

More Related