Data Cloud

1. Data Cloud Yury Lifshits Yahoo! Research http://yury.name

2. My Beliefs The key challenge in web search is structured search Part 1: What is structured search? The key challenge in structured search is collecting data Part 2: Data distribution & idea of Data Cloud Part 3: Demo: numeric data distribution The key challenge in collecting data is incentive design Part 4: Economics of data distribution

3. Structured Search

11. Data = data of entities + data of content Data Structured data Entity unit: • Identifier • Metadata: • Explicit key-value pairs • Relational properties • Evaluation Semi-structured data Content unit: • Body: text, video, audio, or image • Metadata: • Explicit key-value pairs • Relational properties • Evaluation

12. Structured Search Factoid search “what's the value of property X of object Y“ Entity hubs • Domain hubs Structured object search "all concerts this weekend in SF under 20$ sorted by popularity" • Time focus • Ranking focus • Relations focus Structured content search "all videos with Tom Brady" “all comments and blog posts about Bing"

13. Yury’s Wishlist Business-generated data • Products, services, news, wishlists, contact data Reality stream, sensors • Where what have happened Expert knowledge • Glossary, issues, typical solutions, object databases, related objects graph Events • Sport, concerts, education, corporate, community, private Market graph & signals • Like, interested, use, following, want to buy; votes and ratings

14. Query analysis Post analysis App 3 App 1 Classic search App 2 Structured Data Web index Search as a Platform App 4

15. Data Cloud How to collect all structured data in one place?

16. Data Producers • People: forums, wiki, mail groups, blogs, social networks • Enterprizes: product profiles, corporate news, professional content • Sensors: GPS modules, web cameras, traffic sensors, RFID • Transactional data

17. Data distributor is any technical solution to accumulate, organize and provide access to structured and semi-structured data Data publisher: the original distributor of some data Data retailer: a consumer-facing distributor of some data Data Distributors

18. Data Consumers • Humans • Email • Aggregators: news, friend feeds, RSS readers • Search • Browsing / random walks • Intelligence projects • Recommendation systems • Trend mining

19. Data Cloud Data Cloud is a centralized fully-functional data distribution service Success metric for data cloud strategy = the total “value” of data on the cloud

20. To-Cloud Solutions • Extraction • DBpedia.org, “web tables” • Semantic markup, data APIs • Yahoo! SearchMonkey • Feeds • Yahoo! Shopping • Disqus.com, js-kit.com, Facebook Connect • Direct publishing

21. On-Cloud Solutions • Ontology maintenance • Freebase • Normalization, de-duplication, antispam • Named entity recognition, metadata inference, ranking • Data recycling (cross-references) • Amazon Public Data Sets • Viral license • Hosted search • Yahoo! BOSS

22. From-Cloud Solutions • Search, audience • Y! SearchMonkey, Google Base • Data API, dump access, update stream • Custom notifications • Gnip.com • Data cloud as a primary backend • Access control • Ad distribution. (AT&T and Yahoo! Local deal)

23. Demo: webNumbr.com Joint work with Paul Tarjan

25. webNumbr.com: Import • Crawl numbers from the web URL + XPath + regex • Create “numbr pages” • Update their values every hour • Keep the history Anyone can create a numbr http://webnumbr.com/create

26. webNumbr.com: Export • Embed code • Graphs • Search & browse • RSS

27. Economics of Data Distribution Joint work with Ravi Kumar and Andrew Tomkins

28. Two sided market = every product serves consumers of two types A and B Cross-side network effect: the more type-A users product X has, the more attractive it is for type-B consumers and vice versa Examples: operating systems, credit cards, e-commerce marketplaces Two-sided network effects: A theory of information product design G. Parker, M.W. Van Alstyne, N. Bulkley, M. Van Alstyne Network Effect in Two-Sided Markets

29. Basic model • Distributors D1, … Dk • Producer/consumer joins only one distributor • Initial shares (p1,c1) … (pk,ck) • New consumer selects a distributor with a probability proportional to pi • New producer selects a distributor with probability proportional to ci

30. Basic model a2 a4 a3 a1 a1 a3 a4 a2

31. Market Shares Dynamics Theorem 1 Market shares will stabilize Theorem 2 With super-liner preference rule one of distributors will tip Theorem 3 With sub-liner preference rule market shares will flatten

32. External Factor Preference rule with external factor: ei+ci/(c1+…+ck) • Theorem 4 • Market shares will stabilize on • e1 : e2 : … : ek

33. Coalition Data Cloud

34. Coalitions Theorem 5 If all market shares are below 1/sqrt(k) coalition (sharing data) is profitable for all distributors Corollary Coalitions are not monotone Example: 5 : 4 : 1 : 1

35. Model Variations • Same-side network effect • Different p-to-c and c-to-p rules • Multi-homing (overlapping audiences) • n^2 vs. nlog n revenue models • Mature market: newcomer rate = departing rate • Diverse market (many types of producers and consumers) • Newcoming and departing distributors • Directed coalitions

36. Challenges

37. Marketing • Data demand? • Data offerings? • Requirements for distribution technology?

38. Incentive design • Incentives for data sharing? • Centralized or distributed? • For profit or non-profit? • Data licensing and ownership? • Monetizing data cloud?

39. More Challenges Prototyping: • Data marketplace: open data & data demand • Search plugins: related objects, glossaries, object timelines • Publishing tools for structured data • Data client: structured news, bookmarking, notifications Tech design: • Access management • Namespace design User interface: • Structured search UI • Discovery UI

40. Thanks! Follow my research: http://twitter.com/yurylifshits http://yury.name/blog