WMES3103

1. WMES3103 INFORMATION RETRIEVAL WEEK 1 AND 2

2. WHAT IS INFORMATION RETRIEVAL? • Information Retrieval – IR • Information • Retrieval • Lancaster (1968) : An information retrieval system does not inform (I.e change the knowledge) of the user on the subject of his inquiry. It merely inform on the existence (or non-existence ) and whereabouts of documents relating to his request

3. IR – process of getting/retrieving information • Now : a lot of information – print and electronic • Requirement : obtain information quickly and accurately • IR – aims to provide fast , effective and efficient methods of representing, managing , searching, retrieving and presenting such information • IR = the representation , storage, organization of and access to information items

4. Computer science perspective • Design and build a large scale system that will store, manipulate, retrieve and display electronic information of any kind • Text, audio, image and graphics that are stored in such a way that they are available for interaction with human or machine • Library and information perspectives • Search features – au, ti, su, keywords • Relevance of retrieve items

5. Examples of IRS

6. Examples of IRS

7. 3 challenges for IR researchers and practitioners • Technical challenge : what tools should IR systems provide to allow effective and efficient manipulation of information within such diverse media as text, image, video and audio? • Interaction challenge : what features should IR systems provide in order to support a wide variety of users in their search for relevant information. • Evaluation challenge : how can we evaluate which tools and features are effective and usable, given the increasing diversity of end-users and information seeking situations?

8. 3 basic areas of research • Content analysis – describing the contents of the documents in a form suitable for computer processing • Information structures – exploiting relationships between documents to improve the efficiency and effectiveness of retrieval strategies • Evaluation – measurement of effectiveness of retrieval

9. Information Retrieval System • Information Retrieval System = IRS • Before :index document and retrieve • Eg. OPAC of library – cataloguing • Now: modelling, document classification and categorization, system architecture, user interface, data visualization, filtering languages • Eg. WWW

10. Basic Information Retrieval Process Question OR Full description of user information needs Translate into query OR keywords which summarizes the description of user information needs Query processed by a search engine or IRS IRS retrieves information which is useful/relevant to the user

11. Basic Concepts in Information Retrieval • User Task • Logical View of documents

12. User Task • A user has to translate his information needs into query in the language provided by the system • Specify a set of words • English Language Statement : I want a book by J. K Rowling titled The Chamber of Secrets

13. Query entered in a computer system • Au = Rowling • Ti = Chamber of Secrets • “Chamber of Secret” • Rowling AND Stone • Au rowling ti chamber of secrets ti stone

14. 2 User Task • 2 user task – browsing and retrieval • Browsing – the process of retrieving info. Whereby the main objective is not clearly defined from the beginning and whose purpose might change during the interaction with the system. • Eg. User search the internet for info about marine organism  look for info. About Australian aborigines  user is said to be browsing in the collection and not searching • Eg. Searching for a book in the library shelves

15. Retrieval – process of retrieving info whereby the main obj. is clearly defined from the onset of searching process – eg. Eg. Searching for a book in the library shelves

16. 2 actions when user interacts with an IRS 2 actions can be identified when a user interacts with an IRSYS – pulling and pushing actions. Pulling action user request for info in interactive way eg browsing and retrieval Pushing action  push info towards the user periodically through the use of a specified or specially designed s/ware  also known as filtering eg. Yahoo Msgr Service  alert user each time new message arrive Online Stock Exchange

17. Interaction of the user with IRSYS through distinct task IR DB Browsing USER

18. Logical View of Documents Documents in a collection are represented by a set on index terms or keywords Keywords Abstract Full text

19. Logical View of Documents • Documents in a collection are represented by a set of index term/keywords Documents Indexing Process Extracted from text of document Assigned by humans Keywords/subject headings = Logical view of document

20. LISANET – search by abstract

21. MJLIS - EJournal

22. If full text : • Each word in the text is a keyword • Most complex form • Expensive • If full text is too large, there are mechanisms built into the IRS to reduce the number of keyword :

23. Logical view of documents - continue • Stop words (eg articles and connectives – a, the , an, and, of, etc) • Stemming (reduce distinct words to their common grammatical root) eg diary** will find diary or diaries • Truncation – eg catalog* will retrieve catalog, catalogs, catalogue, catalogues • Noun words (eliminates adjectives, adverbs, verbs) eg run will represent runs, running • compression Conversion Process

24. Logical view of documents - continue • This conversion process is known as text operation or transformation • It reduce the complexity of the document representation and allow the logical view from that of a full text to a set of index terms • On the other hand, the human assigned keywords provides the most concise logical view of a document but might lead to retrieval of poor quality – different interpretations, limited keywords if using thesaurus

25. 2 modes of retrieval • Ad-Hoc – the documents in the IRS remains static but new queries are submitted to the system – eg. CD-ROM Database • Filtering – the queries remain relatively static but new documents come into the IRS eg. Stock market

26. Filtering • Construct a user profile that reflects the user’s preferences and profile is matched against incoming documents to find a match or a hit • Retrieve only documents of interest to the user and as specified in the user profile • User select relevant documents from the list. • Filtered documents can also be ranked to further assist the user as to relevance • Construction of a user profile - user provide necessary keywords or collect info about preferences from the user and use this to construct a user profile dynamically

27. INFORMATION RETRIEVAL PROCESS • DEFINE TEXT DATABASE • The text database has to be defined before the retrieval process begins • Done by database manager – documents to be used, operations to be performed on the text, text model • Original documents is transformed into a logical view of the documents via the various text operations • The database manager will then build up the index of the text – manually / computer generated • The retrieval system is tested

28. B. RETRIEVAL PROCESS • The IRS can be used once the document database has been indexed • User puts or present his question/ user need to the IRS • Question is change to a logical view of the document via the text operation • The query operation will present this to the system in a form understandable by the system • Query is processed to obtain the retrieved documents.

29. Continue… • The retrieved document are ranked according to relevance • Retrieved document are sent to the user • User looks through at the ranked documents and can modify question/user need/ query via the user feedback cycle • Same process repeated

30. DEVELOPMENT • For the past 4000 years , man has always been organizing information for retrieval and usage. • It started out with a table of contents for a book. Then, the amount of information extended over a number of books • A specialized data structure is needed to ensure faster access to the stored info. • The oldest and the most popular data form of data structure for fast IR is a collections of words or concept with which are associated pointers to the related info = INDEX • Previously – Manual

31. Development…continue • Now, with the advent of computers, large indexes can be generated automatically. This automatic indexes provide the logical view of the document as perceived by the system and not the user • 2 different views of the IR problems: • Computer-centered building efficient indexes , processing user queries with high performance, develop ranking algorithm which will improve the quality of the answer set • Human-Centered  studying the behavior of the user , understand his main needs, and of determining how such understanding affects the organization and the operation the the IRSYS.

32. IR in the Library • Libraries are the first users of IRSYS to retrieve information • Usually develop by academic institution and later by commercial vendors • 1st generation – automation of the card catalog and allowed searches based on author and title • 2nd generation – increased search functionality - searching by subject headings, keywords, complex queries -OPAC • 3rd generation – graphical interfaces, electronic forms, hypertext features, open system architecture – Digital Libraries

33. The Web and Digital Libraries • Search engine on the web are still using indexes which are similar to the ones used by libraries years ago. • So, what has change? • Advances in computer technology has led to: • Cheaper access to various sources of information • Greater access to network due to advances in all kind of digital communication • Freedom to post information on the web

34. Problems • People still find it difficult to retrieve info relevant to their information needs from the web • Issues to address: • Dynamic world on the web • Demand for access and quick response • Quality of retrieval task is affected by user interaction with the system

35. THANK YOU