Open Search

1. Open Search David Wolber

2. Overview • Proliferation of Digital Libraries • Metasearch and Fixed Lists of Sources • Open Search Architecture • PublishMe for P2P knowledge Sharing • Webtop Metasearch Clients

3. Contributors • Michael Kepe • Igor Ranitovic • Iman Sadreddin • Senior Team ’03 • Ken Chong • Rudd Stevens • Colin Bean • Tim Chan • Julian Chan • Pooja Garg

4. Information Source Explosion • Google, Amazon APIs • Internet Archive • Technorati– The World Live Web • Domain Specific: • ACM Digital Library for CS • Lexis-Nexis for law • MLA for literature

5. Nth Degree 2nd Degree 1st Degree PersonalWeb End-User Created Digital Libraries • Personal Web (shared Google desktop) • Personal Web Neighborhood • Topic-Specific Personal Crawlers • Ordinary people creating search engines as easily as web pages

6. Subsets of the Web

7. Motivation for Small, Independent Subsets of the Web • Avoid information being channeled through a single portal: Googleopoly • Google does no evil, but… • Censorship in China • Creeping level of commercialization • Unregulated manipulation of secret ranking algorithms (see PageKing case) • Other media is lost, this is the last frontier

8. Little support for using multiple search engines

9. Overview • Proliferation of Digital Libraries • Metasearch and Fixed Lists of Sources • Open Search Architecture • PublishMe for P2P knowledge Sharing • Webtop Metasearch Clients

10. Metasearch • Help users discover and use digital libraries • Send queries to multiple, selected search engines • filter, process, and unify results • A9.com – Amazon’s metasearch

11. Web Services Basis html server Web Page Model html server software xml server Web Service Model

12. How does metasearch evolve? New Digital library

13. How does metasearch evolve? New Digital library Metasearch clients discover it

14. How does metasearch evolve? New Digital library Metasearch clients discover it Metasearch Programmers write adaptor/scraper

15. How does metasearch evolve? New Digital library Metasearch clients discover Metasearch Programmers write adaptor/scraper User can access within metasearch SLOWLY…

16. Overview • Proliferation of Digital Libraries • Metasearch and Fixed Lists of Sources • Open Search Architecture • PublishMe for P2P knowledge Sharing • Webtop Metasearch Clients

17. Goal: Automate the Process • Metasearch engines should provide users with up-to-date lists of existing digital libraries • Digital libraries should be able to register and be made immediately available to all Metasearch clients. • Metasearch and Library development is independent.

18. What is Necessary? • Standard Search API • So Metasearch clients can use polymorphism to access sources. for each source s in sourceList { searchEngine.endPointUrl = s.endPointUrl; resultList += searchEngine.keywordSearch(keywords)} • Search API Registry • Metasearch clients can get dynamic list

19. Web Service Standards • WSDL – Web Service Description Language • SOAP – Simple Object Access Protocol • UDDI – Universal Description, Discovery, and Integration

20. Standards on top of Web Services • WSDL, SOAP, UDDI basis for standards in many domains. • e.g., MS initiated for securities information providers • Businesses agree on a standard, then client applications can use polymorphism and new businesses can register services. • In this case, we want cross-domain standard.

21. Open Search Architecture • Open Search Protocol (OSP) • Cross-Domain: Search-related services • Not just keyword search, but citations, authorOf, etc. • Open Search Registry • Based on UDDI • Can add customization, e.g., parsing to find out which search operations are implemented. • Web and web service access

22. Open Search Architecture OSP metasearch clients source list OS Registry Register service OSP-Conforming Libraries

23. User Can Choose Sources

24. Open Search Protocol • Keyword search • Citations (inward links, outward links) • AuthorOf and other associative operations… • Metadata object results based on Dublin Core • Restriction object for “advanced search” stuff

25. Publishing a Library • Access OSP WSDL Specification from webtop.cs.usfca.edu • Generate code in language of choice • Implement the search operations for the digital library • Deploy the service • Register with Open Search registry

26. Deploying an Open Search Lib. Library server 4. deployed service Open Search information Registry 1. OS wsdl programmer 2.wsdl 5. registration info 3. skeleton code wsdl2java

27. Wrapping a Library Custom search API, e.g., Google API 2. Custom query 3. Custom Result Open Search Wrapper Located on 3rd party server 1. OSP Query 4. OSP Result Metasearch Client

28. Wrappers Developed at USF • Google • Amazon (sort of) • Internet Archive • Technorati • Feedster

29. Overview • Proliferation of Digital Libraries • Metasearch and Fixed Lists of Sources • Open Search Architecture • PublishMe for P2P knowledge Sharing • Webtop Metasearch Clients

30. PublishMe • Like Google Desktop, but shared. • Periodically updates inverse index and linkbase on PC • Deploys Web Service on User’s PC • Auto-Registers with Open Search Registry

31. Metasearch with P2P Knowledge Sharing WEBTOP

32. Integrating Global and Personal Libraries

33. Motivation for Sharing Personal Webs People create knowledge everyday when they bookmark, annotate, link, organize, and synthesize. Communication is a separate step which often doesn’t happen

34. Motivation for Sharing Personal Webs Collaborative Work Experts

35. Computers are designed using our brains for a model • Knowledge creation and dissemination separate • Explicit effort required to communicate • Just as we model our word processors on paper.

36. Additions to OSP for P2P • GetFile • OnLine(ip) • Handles user starting up • Dynamic IPs • OffLine

37. But What About PRIVACY? The Big Question: How much of the information hidden within your personal web is hidden due to privacy concerns?

38. I Want you to be a Search Engine!

39. Overview • Proliferation of Digital Libraries • Metasearch and Fixed Lists of Sources • Open Search Architecture • PublishMe for P2P knowledge Sharing • Metasearch Clients

40. Goal: Implement Vannevar Bush’s Association Trails View a document/thing in context History of an idea

41. Thinkmap-like Interface

42. Association Types • Outward links • Inward links • Similar-Content links • People Links • author, people referenced in paper • Domain-Specific links • law citations • movie-actor • Associations specified by Annotators

43. Webtop Tree View webtop.cs.usfca.edu

44. Expanding a Tree • Bird’s Eye View • Local/Web files integrated • Follow different Associative Trails • Ins of Outs of Ins, etc. • Siblings • Weird though, as ins and outs both expand right

45. Webtop Side Panel View

46. Project Status Too many bugs, Dad

47. Future Work • Open Search Protocol • In-depth study of existing search APIs • Provide Rest alternative to SOAP • Metasearch development • Complete and refine existing clients • Dream up new ones • Thinkmap Graph • Automated Source Selection and Reputation System • Page Ranking • Initiate grass-roots involvement

48. Future Work: Documents and Things resourceassociationsannotations document person creative work html word pdf film book

49. Stop talking about Webtop daddy! webtop.cs.usfca.edu