SmartSearch: A Voice Sensing Personalized Mobile Web Search Application

1. SmartSearch: A Voice Sensing Personalized Mobile Web Search Application Team: Abilash Bhanoori, Amit Chaube, Sumit Shrivastava Date-11/24/2011 Mobile Computing Fall 2010

2. Introduction • SmartSearch is a convenient and efficient way to search the web using mobile phones. • Supports two input modes, i.e. voice and keypad input. • Maintains User Profile. • It’s a meta search engine which uses results from Google, Yahoo, and Bing. • Personalizes the results using the clickthrough history, and user’s preference profile. • Concept Video http://www.youtube.com/watch?v=Hwx0JO6ESvg Mobile Computing Fall 2010

3. Challenges • How to efficiently convert Speech to Text. • How to classify users? • How to create and maintain user profile? • How to get the meaning of what the user is trying to search? Mobile Computing Fall 2010

4. Significance to Mobile Computing • Mobile phones based Web Search Application. • Usage convenience provided by voice enabled input. • Context Awareness provided by personalization of results. • Extracting and Maintaining Session becomes easy as mobile devices are personal to the users. Mobile Computing Fall 2010

5. Personalized Web Search with Location Preferences Authors Kenneth Wai-Ting Leung, Dik Lun Lee, Wang-Chien Lee Presented By Amit Kumar Chaube IEEE International Conference On Data Engineering (ICDE 2010)

6. Overview • The Problem • Proposed Solution in brief • Ontologies • Concept and Entropy • Query and User Clustering • Personalized Ranking Functions • Important Formulae • Evaluation and Analysis • Conclusions • Criticism Personalized Web Search with Location Preferences (ICDE 2010)

7. The Problem • As the amount of Web information grows rapidly, search engines must be able to retrieve information according to the user's preference. • The interaction between users and mobile devices are constrained by the small form factors of the mobile devices. • Different classes of users and queries having different emphases on content and location information. Personalized Web Search with Location Preferences (ICDE 2010)

8. Proposed Solution Personalized Web Search with Location Preferences (ICDE 2010)

9. Proposed Solution (Cont…) • Reranking: • When a user submits a query, the search results are obtained from the backend search engines (e.g.,Google, MSNSearch, and Yahoo). The search results are combined and reranked according to the user's profile trained from the user's previous search activities. • Profile Updating: • After the search results are obtained from the backend search engines, the content and location concepts (i.e. important terms and phrases) and their relationships are mined online from the search results and stored, respectively, as content ontology and location ontology. • When the user clicks on a search result, the clicked result together with its associated content and location concepts are stored in the user's clickthrough data. The content and location ontologies, along with the clickthrough data, are then employed in RSVM training to obtain a content weight vector and a location weight vector for reranking the search results for the user. Personalized Web Search with Location Preferences (ICDE 2010)

10. Ontologies • Authors propose an ontology-based, multi-facet (OMF) framework, in which concepts can be classified into different types, such as content concepts, location concepts, name entities, dates etc. • A content concept, like a keyword or key-phrase in a Web page, defines the content of the page, whereas a location concept refers to a physical location related to the page. Personalized Web Search with Location Preferences (ICDE 2010)

11. Concept and Entropy • For a given query issued by a particular user, if the personalization based on content concepts is more effective than based on location concepts, more weight should be put on content-based preference; and vice versa. • Content and location entropies are used for measuring the diversity of content and location information from the search results of a query. • The click content and location entropies are used to determine how much a user is interested in the content and location information associated with a query. • A query result set with high content/location entropy indicates that it has a high degree of ambiguity. • If the click content/location entropies are low, the personalization effectiveness would be high because the user has a focus on certain precise topic in the search results. Personalized Web Search with Location Preferences (ICDE 2010)

12. Concept and Entropy (Cont…) Personalized Web Search with Location Preferences (ICDE 2010)

13. Query and User Clustering • By using K-Means clustering the test queries are classified into four classes: • Explicit Queries: Queries with low degree of ambiguity, i.e., they have small total content and location entropies. • Content Queries: Location specific Queries. • Location Queries: Content centric Queries. • Ambiguous Queries: Queries with high degree of ambiguity, i.e., they have large total content and location entropies. • Users are divided into four user classes as follows: • Very Focused: Users with low content and location entropies, i.e. they have very clear topic focuses in the search results, and only click on a few topics. • Focused: Users with higher content and location entropies and hence less focused than the Very Focused class. • Diversified: Users with even higher content and location entropies and hence more diversified topical interests than the first two user classes. • Very Diversified: Users with high content and location entropies; they click on many topics. These users can be considered novice search engine users. Personalized Web Search with Location Preferences (ICDE 2010)

14. User Preferences Extraction • Joachims Method: • A user would scan the search result list from top to bottom. • If a user skips a document dj at rank j but clicks on document di at rank i where j < i, he/she must have read dj 's web snippet and decided to skip it. Thus the user prefers di to document dj • SpyNB Method: • Users would only click on documents that are of interest to them. Thus, it is reasonable to treat the clicked documents as positive samples. • Predict from the unlabeled set reliable negative documents which are irrelevant to the user. To do this, the ‘spy’ technique incorporates a novel voting procedure into Naive Bayes classier. Personalized Web Search with Location Preferences (ICDE 2010)

15. Personalized Ranking Functions • A set of content concepts and a set of location concepts are extracted from the search result as the document features. • Since each document can be represented by a feature vector, it can be treated as a point in the feature space. • The feature vectors are extracted by taking into account the concepts existing in a documents and other related concepts in the ontology of the query. • The similarity and parent-child relationships of the concepts in the extracted concept ontologies are also incorporated in the training based on the following four different types of relationships: • (1) Similarity, (2) Ancestor, (3) Descendant, and (4) Sibling Personalized Web Search with Location Preferences (ICDE 2010)

16. Personalized Ranking Functions (Cont…) Personalized Web Search with Location Preferences (ICDE 2010)

17. Important Formulae • To measure the interestingness of a particular keyword/phrase ci with respect to the query q: Where sf(ci) is the snippet frequency of the keyword/phrase ci (i.e. the number of web-snippets containing ci), n is the number of web-snippets returned and |ci| is the number of terms in the keyword/phrase ci. • To compute the content and location entropies of a query q (i.e. HC(q) and HL(q)) • Equations to estimate the personalization effectiveness using the extracted content and location concepts with respect to the user u.

18. Evaluation • Using a metasearch engine which comprises Google, MSNSearch and Yahoo as the backend search engines to ensure a broad topical coverage of the search results. • Joachims(Content) method performs the best on content queries. It boosts the top 1, 10, and 20 precisions of content queries from 0.4583, 0.3563, and 0.3125 to 0.7519, 0.5874, and 0.4176 (64%, 65%, and 34% in percentage gain), comparing to the baseline method. • Joachims(Location) method performs the best on location queries, boosting the top 1, 10, and 20 precisions of location queries from 0.5208, 0.4063, and 0.3563 to 0.6989, 0.4269, and 0.3583 (34%, 5%, and 0.5% in percentage gain). • Using SpyNB for preference extraction performs better than using Joachims' method in all classes of queries, because SpyNB generates more accurate preferences comparing to Joachims' method. Personalized Web Search with Location Preferences (ICDE 2010)

19. Conclusions • In this paper, an Ontology-Based, Multi-Facet (OMF) personalization framework has been proposed for automatically extracting and learning a user's content and location preferences based on the user's clickthrough. • The notion of content and location entropies are introduced to measure the diversity of content and location information associated with a query and and click content and location entropies to capture the breadth of the user's interests in these two types of information. • Experimental results confirmed that OMF can provide more accurate personalized results comparing to the existing methods. Personalized Web Search with Location Preferences (ICDE 2010)

20. Criticism • Memory and space issues are completely ignored. • Testing of the solution is required by large number of random users (ignorant to the implementation) to validate the efficiency of the algorithm. Personalized Web Search with Location Preferences (ICDE 2010)

21. Survey of The Speech Recognition Techniques for Mobile Devices Author Dmitry Zaykovskiy Presented By Abilash Bhanoori SPECOM’2006

22. Overview Introduction Basic Methodology of ASR Systems Mobile ASR Dilemma ASR architectures Conclusion Criticism SPECOM’2006

23. Introduction This paper presents an overview of different approaches for providing automatic speech recognition technology to mobile users. Why do we need Automatic Speech Recognition (ASR) in mobile devices? i) The basic problem of handheld gadgets is their miniature size. ii) Typing on such tiny keyboards or pointing with the stylus is very uncomfortable & error prone. E.g. PDA are often used when a person is really “on the move”. Operating in such conditions is very tedious. SPECOM’2006

24. Basics of Automatic Speech Recognition Goal: Finding most probable sequence of words W =(w1,w2,w3…..) belonging to a fixed vocabulary given some set of acoustic observations O= (o1,o2…..ot). Calculating Best estimation for the word sequence (Using Baye’s Theorem): In order to generate an output the speech recognizer has to basically perform the following operations: i) Extracting acoustic observations (features) out of the spoken utterance. ii) Estimating P(W)- probability of individual word sequence to happen, regardless of acoustic observations. iii) Estimating P(O/W) –the likelihood that the particular set of features originates from certain sequence of words. iv) Find word sequence that delivers maximum of above equation. SPECOM’2006

25. The Mobile ASR Dilemma The implementation of effective & efficient mobile ASR systems is challenged by many border conditions . In contrast to the generic ASR, the mobile recognition system has to encounter the following aspects: i) Limited available storage volume. ii) Tiny cache of 8-32KB and small & slow memory from 1MB up to 32 MB. iii) Low processor clock frequency. iv) Cheap Microphones. v) Highly challenging acoustic environments. SPECOM’2006

26. System Configurations for Mobile Speech Recognition ASR systems can be structurally decomposed into two parts namely: i) Acoustic front-end where the feature extraction takes place. ii) Acoustic back-end where Viterbi search is implemented based on the acoustic & language models. Based upon the location of the front-end & back-end mobile ASR systems can be classified into three principal system structures: i) Client Based architecture or embedded ASR, where both front-end & back–end are implemented on the terminal. ii) Server Based or network speech recognition (NSR), where speech is transmitted over communication channel and the recognition is performed on the remote server. iii) Client-Server ASR or DSR , where the features are calculated on the terminal, while the classification is done on the server side. SPECOM’2006

27. Embedded Speech Recognition Entire process of speech recognition is performed on the terminal device. Figure 1 :Embedded Speech Recognition Architecture Embedded ASR is often the architecture of the choice for PDAs for the following reasons : i) PDAs are driven well under well established operating systems. ii) PDAs have well known processor architectures, e.g. Intel XScale. iii) PDAs don’t always have a wireless communication link available ,hence remote connection is rather unwelcome for PDAs. SPECOM’2006

28. Cont… Advantage: No communication between the server and the client is needed. Hence ASR system is always ready for use & doesn’t depend on quality of data transmission. Disadvantage: Embedded ASR systems has very limited system resources on the mobile devices. For effective implementation of embedded ASR two important characteristics have to be considered: i) Memory Usage. ii) Execution Speed. SPECOM’2006

29. Network Speech Recognition (NSR) Figure 2: ASR features extracted from the transcoded speech (Initial Version) Advantages: i) One basic advantage over Embedded ASR is that in NSR all complications caused by the resource limitations of the mobile devices can be avoided shifting both ASR front-end and back-end from the terminal to the remote server. SPECOM’2006

30. Cont… ii) Unlike the embedded ASR, the NSR architecture can not only augment not only PDAs but also thin terminals ,e.g. cellular phones , with a very large vocabulary ASR. iii) NSR can provide access to the recognizers based on the different grammars or even different languages. Disadvantage: Performance degradation of the recognizer caused by using low bit rate codecs, which become more severe in presence of data transmission errors and background noise. SPECOM’2006

31. Improved Version of NSR Architecture Distortion introduced from source coding in previous version can be diluted to certain extent if the recognizer is trained on the respectively corrupted speech. However, grouping of different source coding schemes in addition to the different channel noise levels spans too large number of possible acoustic models. In the improved version recognition is performed based on the features derived from the parametric representation of the encoded speech with out the actual speech reconstruction as shown in the figure below. Figure 3: ASR features derived from speech codec parameters SPECOM’2006

32. Distributed Speech Recognition (DSR) DSR represents the client-server architecture, where one part of the ASR system , primary features ,resides on the client, while the computation of temporal derivatives and the ASR search are performed on the remote server as shown in figure below. Figure 4: Client-Server based ASR system SPECOM’2006

33. Advantages of DSR over NSR Even though both DSR & NSR make use of the server based back-end, there are substantial differences in these two schemes favoring DSR. Following are advantages of DSR over NSR: i) Speech codecs unlike the feature extraction algorithms are optimized to deliver the best perceptual quality and not for providing WER. ii) DSR doesn’t need high quality speech, but rather some set of characteristic parameters. Thus it requires lower data rate. iii) In DSR unlike NSR we are not constrained to the error mitigation algorithm of the speech codecs, hence better handling methods in terms of WER can be developed. SPECOM’2006

34. Conclusions Medium Recognition tasks having 1000-2000 words, which represent good coverage of the certain application domain will be successfully running on the terminal devices like PDAs or in car embedded systems. DSR can be used to implement for high data bit rate networks. Because of its superior performance in presence of the transmission errors and surrounding noise it is expected that NSR will be totally supplanted by the DSR architecture in the near future. SPECOM’2006

35. Criticism This paper gives brief introduction of different architectures but fails to analyze them in detail. Only Trivial advantages or very basic advantages have been mentioned of ASR systems. SPECOM’2006

36. Context-Aware Ranking in Web Search Authors BIAO XIANG, DAXIN JIANG, JIAN PEI Presented By Sumit Shrivastava ACM SIGIR conference on Research and development in information retrieval 2010

37. ISSUES IN RE-RANKING WHY NEED RE-RANKING • How can we take advantage of different types of contexts in ranking? • How can we integrate context information into a ranking model? • Current search engines don’t consider context in ranking. • Mobile phone’s screen is very small and cannot accommodate much record. • What if the records re-ranked so that desired records get higher ranking.

38. EXAMPLE CONTEXT ORIENTED RE-RANKING • Maintains log of previous search queries, returned result and clicked results. • Assumes inspection of returned results sequentially from top to bottom. • User is looking for car. He searches BMW. • His second query is jaguar. • Jaguar could be a bird or could be a car model.

39. CONTEXT AWARE RE-RANKING PRINCIPLES • Responsible for promoting or demoting documents according to the context of the current query. • Studies say a search result is likely to be viewed by a user if it is a) among the top two search results. b) ranked above the lowest clicked result or c) ranked one position below the lowest clicked result.

40. CONTEXT AWARE RE-RANKING PRINCIPLES • Unrelated query. • Reformulating. • Specializing. • Generalizing. • Associated with previous query in session. There can not be any kind of prediction for Unrelated queries.

41. REFORMULATION • A user may reformulate her previous query into a new one because the search results for the previous one do not or only partially fulfill her information need. Example 1) first query house on rent. 2) next query home on rent.

42. SPECIALIZATION • When a user issues a specializing query, she likely wants to see results that are more specific about her interests. Example a) The user first asked “time life music” and clicked on the homepage of the store. b) The user further asked “time life Christian CDs” and clicked on the 4th and 5th results.

43. GENERAL ASSOCIATION • When a query (especially an ambiguous one) is generally associated with its context, the context may help to narrow down the user’s search intent. Example a. First query “Xbox 360”. b. Second query “Fifa 2010”. If both queries are related. Second query is meant for fifa 2010 game on Xbox.

44. GENERALIZATION • A user may ask a query more general than the previous one • In such a situation, the user may like to see some information not covered by the first query. Example a) A user first asked query “free online Tetris game” and clicked on the 1st and 2nd search results. b) The user then asked query “Tetris game” and clicked on the 3rd and 4th results.

45. EFFECTIVENESS OF PRINCIPLES • Data were collected from major search engines. • Session boundary was 30 minute’s idle time. • Experiment was done on query pairs qn and qn-1. • Relations were labeled manually.

46. CHALLENGES IN CURRENT MODEL • Learning to rank approach to build model. rankSVM model is used. • In training it takes ordered pair of document with respect to query under context. • A feature in this model is a function of query, document, and context. • We considered only consecutive pair of queries. • More than one principals can be applied at a time. • There could be some more useful contexts like positions of documents returned by queries or shared terms between current and previous query. SUGGESTED METHODOLGY

47. EXPERIMENTAL RESULT WITH rankSVM • MCP is mean click position. • Less MCP better performance. • Compared with baseline method.

48. CONCLUSION • we studied the problem of using context information in ranking documents in Web search. • Conducted an empirical study on real search logs and developed four principles for context-aware ranking. • adopted a learning-to-rank approach and incorporated our principles to ranking models. • The experimental results verified the effectiveness of our approach.

49. Questions?

50. Thank You!