Introduction and Analysis of Web 2.0 Technologies

1. Introduction and Analysis of Web 2.0 Technologies November 23, 2006 Jaesun Han (jshan0000@gmail.com) Research Fellow / Ph.D ANLAB, Dept. of EECS, KAIST Contact : http://www.web2hub.com

2. Contents • Introduction of Web 2.0 • Key Philosophy of Web 2.0 • Contents Production by User Participation • Decentralization of Contents Consumption • Contents Sharing by Openness • Web as Platform • Definitions • Case Studies • Enabling Technologies • Client technologies • Server technologies • Global Platform technologies

3. The Origin of Web 2.0 New Conference? Question: What are the characteristics of web companies surviving dot-com collapse? (Amazon, eBay, Yahoo, Google, etc)

4. Seven Principles of Web 2.0 1. The Web as Platform 2. Harnessing Collective Intelligence 3. Data is the Next Intel Inside 4. End of the Software Release Cycle 5. Lightweight Programming Models 6. Software Above the Level of a Single Device 7. Rich User Experiences -from Tim O’Reilly’s “What is Web 2.0?”

5. Tim O’Reilly’s Web 2.0 Meme Map

6. Web 2.0 Mind Map

7. Traditional Contents Distribution Contents Providers Centralized Information Production Information Producer Consume Information Consumer Users

8. Web as Platform Key Philosophy of Web 2.0 Providers Contents mashup (web services, RSS) tagging Openness Participation Users social network Decentralization

9. Participation

10. UCC(User-Created Contents) Contents Production by User Participation Text Video Photo Audio

11. UCC Services Text Photo Audio Video podcasting photo storing & sharing video storing & sharing blog & wiki

12. Expansion of User Paricipation User-Created Media User-Created Software P2P Network  User-Generated Network WiFi Community for free WiFi access  User-Generated Infrastructure

13. Collective Intelligence • Collective Intelligence • Crowd's "collective intelligence" will produce better outcomes than a small group of experts (Users add value)  Network effects from user contributions • Requirements for Collective Intelligence • diversity of opinion • independence of members from one another • decentralization • a good method for aggregating opinions • Example cases of aggregating methods • Google PageRank • Yahoo Collaborative Tag Suggestion • Amazon Recommendation

14. Google PageRank • The Philosophy of PageRank • PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page's value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. But, Google looks at more than the sheer volume of votes, or links a page receives; it also analyzes the page that casts the vote. Votes cast by pages that are themselves "important" weigh more heavily and help to make other pages "important." • PageRank Algorithm • PR(A) : PageRank of page A • L(A) : The number of links going out of page A • d : damping factor

15. Yahoo Collaborative Tag Suggestion • The Philosophy of Collaborative Tag Suggestion • Selecting good tags • High popularity  tag quality • High coverage of multiple facets  good recall • Least effort  reduce the cost involved in browsing • Eliminating tag spam Utilizing collective user tagging behavior and authorities • Collaborative Tag Suggestion Algorithm S(t’,o) = S(t’,o) + Ps(t’|ti;o) x S(ti,o) – Pa(t’|ti) x S(ti,o) • Ps(t’|ti;o) : correlation probability between t’ and ti for the object o • Pa(t’|ti) : overlap probability in terms of the concepts between t’ and ti • S(t’,o) : goodness measure (score) of the tag t’ to an object o • a(u) : authority score of a given user u

16. Yahoo Collaborative Tag Suggestion • Experiments based on Yahoo My Web 2.0 tag data • Suggested Tags for the URL http://wiki.osfoundation.org/bin/view/Projects/AjaxLibraries

17. Tagging Systems • What is Tagging? • keyword, description, classification, user-based, collaboration, easy, linear thinking, flickr, del.icio.us, hotissue, muststudy • Best examples: flickr, del.icio.us, gmail, technorati • Goals: organizing, sharing, navigating, filtering, searching, etc • Taxonomy vs. Folksonomy • Taxonomy: hierarchical & exclusive • Folksonomy(by tagging): non-hierarchical & inclusive • Automatic annotation vs. Tagging • Automatic annotation: content-based & good for only text • Tagging: context-based & good for multimedia data

18. Kinds of Tags • Identifying What it is About • ex) ajax, cat, mountain, etc • Identifying What it Is • ex) article, blog, book, etc • Identifying Who Owns It • ex) MichaelArrington, DionHinchcliffe, etc • Refining Categories • ex) 25, 100, etc • Identifying Qualities or Characteristics • ex) funny, stupid, interesting, inspiration • Self Reference • ex) mystuff, mycomments, etc • Task Organizing • ex) toread, jobsearch, musthave, etc

19. Unified User-Resource-Relation Model Analysis for Tagging Resources (URL, Blog Post, Photo, Video, etc) Users t1,t2,t3 t1,t4,t5 t6,t7 t8,t9 User-User Links t2,t7,t9 Resource-ResourceLinks t1,t3 t2,t6 t2,t6 Relation (Tagging)

20. Issues of Tagging http://www.web2hub.com/wiki/index.php/20061117_TagDay

21. Amazon Recommendations So many Recommendations!

22. Amazon Recommendations • Item-to-Item Collaborative Filtering • Matching user’s purchased and rated items to similar items  similar-items table • Identify similarities between different items • Items more static than users • Offline: Item similarity computation • Online: Prediction computation

23. Recommendation Systems • Recommendation • Solution for Information Overload • cf. Reputation System: Solution for Transacting with Strangers • Example applications • E-commerce : product recommendations • Corporate Intranets : Finding domain experts • Digital Libraries : Finding pages/books people will like • Medical Applications : Matching patients to doctors • Customer Relationship Management (CRM) : Matching customer problems to internal experts • Types of Recommendation Systems • Content-based Recommendations: The user will be recommended items similar to the ones the user preferred in the past. • Collaborative Recommendations: The user will be recommended items that are preferred by other people with similar tastes and preferences. • Hybrid approaches: These methods combine collaborative and content-based methods.

24. Case Study: Content-based • Pandora (www.pandora.com) • Created by the Music Genome Project • The most comprehensive analysis of music • Over the past 6 years, the songs of over 10,000 different artists are analyzed the musical qualities of each song one attribute at a time. • Musical Genome (Hundreds of musical attributes) • melody, harmony, rhythm, instrumentation, orchestration, arrangement, lyrics, singing and vocal harmony, etc

25. Content-based Recommendation • Content-based approach • Its roots in information retrieval and information filtering • Focus on recommending items containing textual information, such as documents, Web sites (URLs), and news messages etc • Improvement by the use of user profiles that contain information about users’ tastes, preferences, and needs • The process of Content-based recommendation • Construction of per-user content-based profile • TF : Term Frequency, IDF : Inverse Document Frequency • N : Number of the documents • ni : How many times keyword ki is seen in the document • fi,j : Number of times keyword ki is seen in the document dj

26. Content-based Recommendation • Similarity measurement • u(c,s) : the utility function • ContentBasedProfile(c) = (wc1, …, wck) : the profile of user c • cosine similarity measure • Limitations of content-based approach • Limited Content Analysis • automatic feature extraction is much harder to multimedia data • cannot distinguish between a well-written article and a badly written one • Overspecialization • no experience, no recommendation • New User Problem

27. Case Study: Collaborative Filtering • Last.fm (www.last.fm) • Automatic construction of personalized music profile • Scrobbling a song : sending the list of listened songs to Last.fm • Recommendation by collaborative analysis of music profiles • Recommended tracks • Recommended readings • Recommended users • Similar artists

28. Collaborative Filtering 4) Request Recommendations 1) Submit Ratings 6) Select Items & Predict Ratings C.F. Engine 5) Identify Neighbors 2) Store Ratings 3) Compute Correlations Ratings Correlations

29. Collaborative Filtering Algorithm User-Item Matrix Meg & David: similarity -0.59 Meg & Amy: similarity 0.67 Meg & Joe: similarity 0.47 Recommendations for Meg: Movies 7

30. Collaborative Filtering • Limitations • New User Problem • Can be addressed using hybrid recommendation • New Item Problem • Until the new item is rated by a substantial number of users, it is not recommended • Can be addressed using hybrid recommendation • Rating Sparsity • the number of ratings already obtained is usually very small compared to the number of ratings that need to be predicted • Can be addressed using demographic filtering

31. Links derived from similar attributes, similar content, explicit cross references Unified User-Resource-Relation Model Analysis for Recommendation Resources (Book, Music, Movie, Product, Article, Web Page) Users Links derived from similar attributes, explicit connections A 1 B 2 Resource -ResourceLinks User-User Links C 3 D 4 E 5 Observed preferences (Ratings, purchases, page views, wish lists, play lists)

32. Decentralization

33. Text Photo Audio Video Contents Production  Consumption Contents (User-Created Contents, Ready-Made Contents) Contents Consumption Recommendation Recommendation, Search, Tagging • Searching • Discovering (links, tags, directories) • Recommended Long Tail Personalization

34. Personalized News Personalization Personalized Search Personalized Homepage

35. The Long Tail http://www.wired.com/wired/archive/12.10/tail.html?pg=3

36. The Long Tail • The Long Tail • Coined by Chris Anderson • Infrequent events(the long tail) can cumulatively outweigh the initial portion of the graph, such that in aggregate they comprise the majority • Overcoming space-time limitation of offline market • Long Tail in Online Ads Market Web 1.0 : DoubleClick Web 2.0 : Google AdSense

37. Openness

38. Openness & Mashup • Openness • Data by RSS • Service by Open API • Mashup • Website or web application that seamlessly combines content from more than one source into an integrated experience • Examples • Google News : News aggregation • Newsmap : News visualization (Using Google News) • WingBus : Travel info service (Aggregating blog posts on traveling) • Housingmaps.com = Google Maps + Craigslist • Amazon Light = Amazon + Other Searches + Yahoo News + … • Chicago Crime Map = Google Maps + CPD's Citizen ICAM Web site • Aladdin TTB Review = Aladdin + Blog APIs

39. RSS • Really Simple Syndication • A family of XML based web-content distribution and republication (Web syndication) protocols • Primarily used by news sites and weblogs • Currently used by various types of contents like search results, bug reports, wiki updates, podcasting&videocasting, even fortune-telling • Two Software for RSS • RSS Feeder: web application by which RSS feeds are dynamically updated with the change of contents • RSS Reader(Aggregator): program that checks RSS-enabled feeds and displays any updated information that it finds (ex. HanRSS) • Standards • RSS 0.9x, RSS 1.x, RSS 2.x, Atom

40. Examples of Open APIs API Scorecard (http://programmableweb.com/scorecard)

41. Types of Open APIs • SOAP • Protocol for exchanging XML-based message, normally using HTTP • Much more robust way to make requests, but more robust than most APIs need • More complicated to use • REST • Software architectural style for distributed hypermedia systems like WWW • Quickly gained popularity through its simplicity • XML-RPC • RPC protocol with XML as a encoding and HTTP as a transport • More complex than REST but simpler than SOAP • JavaScript • The newest trend in APIs • Offering a free JavaScript library that is the only way to access data • Limited integration with other services • JSON-RPC • RPC protocol encoded in JSON instead of XML • Very simple protocol (and very similar to XML-RPC)

42. Example Scenario: Book Shopping source: http://www.razorsoft.net/slides/RESTfulSOAP.ppt

43. SOAP (Simple Object Access Protocol) • Simple, lightweight XML protocol for exchanging structured and typed information on the Web • Used for communication between applications • XML as the standard message format • Mostly HTTP as the transport method • Platform & Language independent • Simple and Extensible • Baseline standard for rich set of messaging features • Addressing, Routing • In-Message Security, Identity Federation • Reliable Messaging

44. SOAP Message Format (Request) <SOAP-ENV:Envelope xmlns:SOAP-ENV=“http://schemas.xmlsoap.org/soap/envelope/” SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/”> <SOAP-ENV:Header> <t:transId xmlns:t=“http://foo.com/trans”>1234</t:transId> </SOAP-ENV:Header> <SOAP-ENV:Body> <m:Add xmlns:m=“http://foo.com/Calculator”> <n1>1</n1> <n2>2</n2> </m:Add> </SOAP-ENV:Body> </SOAP-ENV:Envelope> SOAP namespace Encoding style of message Transaction ID Method Parameters

45. Example Scenario with SOAP 1.1

46. Example Scenario with SOAP 1.1

47. Mashup Example using SOAP • MapMash (Google Maps Mashup) • Google Maps + Geocoder.us • Example from JavaWorld article • Apache Tomcat 5.5 (Java Servlet), Apache Axis library, Direct Web Remoting(DWR) library • Demo • Code example • Google Maps (JavaScript) <script src="http://maps.google.com/maps?file=api&v=1&key=Your_Key" type="text/javascript"></script> • Geocoder.us (SOAP) GeoCoder.geocode(address, moveMapCallback);

48. MapMash Sequence Diagram

49. REST (Representational State Transfer) • REST • Proposed by Roy Fielding in a 2000 doctoral dissertation • Architectural Styles and the Design of Network-based Software Architectures • Design pattern for creating Web Services (Not a standard) • Three fundamental aspects of the REST Design Pattern Resources Every distinguishable entity is a resource Simple Operations Most web interactions are done using HTTP CRUD (PUT,GET,POST,DELETE) URLs Every resource is uniquely identified by a URL

50. Example Scenario with REST