Search strategy & tactics

Search strategy& tactics Governed by effectiveness & feedback tefkos@rutgers.edu; http://comminfo.rutgers.edu/~tefko/ Tefko Saracevic

Central ideas As a searcher you are striving toward doing effective searches Knowing searching = also knowing tactics to reach toward desired effectiveness • Searches are all about being effective • finding what is needed, desired • To measure effectiveness common measures of precision and recall are used • Various tactics are used to affect effectiveness • Application of tactics for a desired results is a second nature of professional searchers Tefko Saracevic

ToC Major concepts Measuring effectiveness Search tactics Feedback types Tefko Saracevic

can it be done? 1. Major concepts Effectiveness, relevance Tefko Saracevic

Some definitions • Search statement (query): • set of search terms with logical connectors and attributes - file and system dependent • Search strategy (big picture): • overall approach to searching of a question • selection of systems, files, search statements & tactics, sequence, output formats; cost, time aspects • Search tactics (action choices): • choices & variations in search statements • terms, connectors, attributes Tefko Saracevic

Some definitions (cont.) • Move : • modifications of search strategies or tactics that are aimed at improving the results • e.g. from searching for digital AND libraries to digital(w)libraries (Dialog) or “digital libraries” (Scopus) • Cycle (particularly applicable to systems such as Dialog): • set of commands from start (begin) to viewing (type) results, or from a viewing to a viewing command Tefko Saracevic

Search strategy – big pictureall kinds of actions from start to end • The entire approach to a search – selection of • files and sources to use • approaches in proceeding to search & combining • search terms • operators to use • fields to search • formats for viewing results • alternative actions if search yields • too much • too little • problem-solving heuristics Tefko Saracevic

Search tactics – specific actionsconnected with a given search as it progresses • A query - command line entered into a system in order to retrieve relevant information • terms, operators & attributes as allowed by a given system • vocabulary & syntax used in conjunction with connectors &/or limiters to search a system • Again: depends on a system how it is done • for example, a search statement might be: • in Dialog: b 47; ss (garbanzo? or chickpeas) and (hum?us or humus) • in Scopus you can enter: (garbanzo? or chickpeas) and (hum?us or humus) • how would you do that in ? Tefko Saracevic

Search performance is expressed in terms of: • Effectiveness : • performance as to objectives • to what degree did a search accomplish what desired? • how well done in terms of relevance? • Efficiency : • performance as to costs • at what cost and/or effort, time? • But here we concentrate on effectiveness Both are KEY concepts & criteria for selection of strategy, tactics & evaluation Tefko Saracevic

Effectiveness criteria • Search tactics chosen & changed following some criteria of accomplishment, such as: • none - no thought given – hard to imagine but happens sometimes • relevance (most often) – is it relevant to a need, task, problem? • magnitude (also very often)- is it to much retrieved to start with? • output attributes- is it trustworthy? Authorities? Understandable? … • topic – is it on the topic of the question? • Tactics are chosen or altered interactively to match given criteria - feedback plays an important role Knowing what choice of tactics may produce what results is key to professional searcher Tefko Saracevic

Relevance:key concept in IR & key criterion for assessing effectiveness “Relevant: having significant and demonstrable bearing on the matter at hand.” “Relevance: the ability (as of an information retrieval system) to retrieve material that satisfies the needs of the user.” Merriam Webster (2005) • Attribute/criterion reflecting effectiveness of exchange of inf. between people (users) & IR systems in communication contacts, based on valuation by people Tefko Saracevic

Variation in names • Relevance or relevant can be expressed in different terms • utility, pertinence, appropriate, significant, useful, germane, applicable, valuable … • But the concept still remains relevance That which we call relevance by any other word would still be relevance Relevance is relevance is relevance is relevance thank you Shakespeare and Gertrude Stein Tefko Saracevic

Some attributes of relevance • in IR - user dependent – users are the final judges at the end • multidimensional or faceted – users asses relevance on a number of factors, not only topic, but also authority, novelty, source … • dynamic – users may change relevance assessments & criteria as search progresses or they learn or the problem is modified • not only binary – users assess object not only as relevant/not relevant, but also more on a continuum (scale) - partially relevant included • intuitively well understood – nobody has to explain to users what it is Tefko Saracevic

ConsistencyIndividual differences • consistency of assessment varies over time & between people to a (sometimes significant) degree – but (in)consistency is similar as in other processes e.g. indexing, classifying • Subject expertise affects consistency of relevance judgments. Higher expertise results in higher consistency and stringency. Lower expertise results in lower consistency and more inclusion. • relevance is not fixed; individual differences may be large • It is most helpful to discuss with a user what kind of documents they may asses as relevant, what are their criteria and then try to incorporate that in searching & own assessments Tefko Saracevic

Two major types of relevance • leading to a dichotomy of systems- & users-view of relevance: • Systems or algorithmic relevance • the way a system or algorithm assesses relevance • relation between a query & objects in the file of a system as retrieved or failed to be retrieved by a given algorithm • User relevance • the way a user or user surrogate (searcher, specialist …) assesses relevance • relation between information need or problem at hand of a user and objects retrieved or out there in general • But Topical relevance can be considered by both system & user • the degree to which topics or subjects of a query & topics or subjects of objects (documents) in a file or retrieved match • relation between topic in the query & topic covered by the retrieved objects, or objects in the file(s) of the system, or even in existence Tefko Saracevic

User relevance can be further distinguished as to: • Cognitive relevance or pertinence: • possible changes in user’s cognitive state due to objects retrieved • relation between state of knowledge & cognitive information need of a user and the objects provided or in the file(s) • Motivational or affective relevance • matching & satisfying user’s intentions, purposes, rationales, emotions • relation between intents, goals & motivations of a user & objects retrieved by a system or in the file, or even in existence. Satisfaction • Situational relevance or utility: • value of given objects or information for user’s situation or changes in situation • relation between the task or problem-at-hand & the objects retrieved (or in the files). Relates to usefulness in decision-making, reduction of uncertainty ... Tefko Saracevic

2. Measuring effectiveness Precision and recall Tefko Saracevic

How? The basic way by which effectiveness is established in IR searching is to compare Tefko Saracevic

Effectiveness measures Two measures are used universally: Precision: • of the stuff retrieved & given to user how much (what %) was relevant? • more formally: probability that given that an object is retrieved it is relevant, or the ratio of relevant items retrieved to all items retrieved Recall: • of the stuff that is relevant in the file how much (what %) was actually retrieved? • more formally: probability that given that an object is relevant it is retrieved, or the ratio of relevant items retrieved to all relevant items in a file Tefko Saracevic

Calculation Precision = High precision = maximize a, minimize b a a + b a High recall = maximize a, minimize c Recall = a + c Tefko Saracevic

Examples of calculation • If a system retrieved 16 documents and only 4 were assessed as relevant by a user then precision is 25% • If a system had 40 documents in the file that were relevant but managed to retrieve only 12 of them then recall is 30% • Precision is easy to establish, recall is not • union of retrievals is used as a “trick” to establish relative recall • you do a number of searches or use a number of tactics or algorithms then you take together all that was retrieved (union) and have that assessed; then you can calculate a relative recall of each search, tactic or algorithm in respect to the union & see which one provides better or worse relative recall Tefko Saracevic

Interpretation: PRECISION • Precision= percent of relevant stuff you have in your set of answers retrieved • or conversely percent of junk (false drops) in the answer set • high precision = most stuff relevant • low precision = a lot of junk • Some users demand high precision • do not want to wade through much stuff • but it comes at a price: relevant stuff may be missed • Tradeoff almost always: high precision = low recall • we will get to tradeoff a bit later, but it is VERY important to consider Tefko Saracevic

Interpretation: RECALL • A file may have a lot of relevant stuff • Recall = percent of that relevant stuff in the file that you retrieved in your answer set • conversely percent of stuff you missed • high recall = you missed little • low recall = you missed a lot • Some users demand high recall (e.g. PhD students doing a dissertation; patent lawyers; researchers writing a proposal or article) • want to make sure that important stuff is not missed • but will have to pay a price of wading through a lot of junk • Tradeoff almost always: high recall = low precision Tefko Saracevic

3. Search tactics Tradeoff between precision and recallUsing it for different tactics Tefko Saracevic

Aim of search tactics Thus many search tactics are geared toward achieving certain level of precision or recall • Since there is no such thing as a perfect search: • the aim is to search in a way that will insure a given, chosen or desired, level of effectiveness • That means that we have to agree on • what do we mean by effectiveness in searching? • general agreement is that retrieval of relevant answers is the major criterion for effectiveness of searching • what measures do we use to express achievement of effectiveness in terms of relevance • general agreement is that we use measures of precision and recall (or derivatives) Tefko Saracevic

Precision-recall trade-off 100 % Ideal Precision Improvements Usual 0 Recall 100 % • USUALLY: precision & recall are inversely related • higher recall usually means lower precision & vice versa Tefko Saracevic

Precision-recall trade-off • It is like in life, usually: • you get some, lose some; you can't have your cake and eat it too • Usually, but not always (keep in mind these are probabilities) • when you have high precision most stuff you got is relevant or on the target but you also missed other stuff (could be a lot) that is relevant – it was left behind • when you have high recall you did not miss much but you also got junk (could be a lot) - you have to wade through it • There is price to pay either way • but then a lot of users are perfectly satisfied with and are aiming at high precision • give me a few good things, that is all what I need Tefko Saracevic

Effectiveness (Cleverdon’s) “laws” High precision in retrieval is usually associated with low recall. High precision = low recall High recall in retrieval is usually associated with low precision. High recall = low precision You – with your user – have to decide if you are aiming toward high recall or high precision & you have to be aware of & explain tradeoffs Originally formulated by Cyril Cleverdon, a UK librarian, in 1960s, in first IR tests done in Cranfield, UK, thus “Cranfield tests” You use different tactics for high recall from those for high precision Tefko Saracevic

Search tactics • several ‘things’ or variables in a query can be selected & changed to affect effectiveness Tefko Saracevic

Actions and consequences • BUT: each variation has consequence in output • if you do X then Y will happen Tefko Saracevic

Tactics: what to do? • These “laws” lead to precision & recall devices • tactics to increase/decrease precision or recall Tefko Saracevic

Precision, recall devices With experience use of these devices will become second nature Tefko Saracevic

Other tactics (but that gets us in the next unit on advanced searching) • Citation growing: • find a relevant document • look for documents cited in • look for documents citing it • repeat on newly found relevant documents • Building blocks • find documents with term A • review – add term B & so on • Using different feedbacks • a most important tool Tefko Saracevic

4. Feedback Types. Berry picking Tefko Saracevic

Feedback in searching • Extremely important! • And constantly used, consciously or unconsciously • Formally: • the process in which part of the output of a system is returned to its input in order to regulate its further output • Simply put: • you search, find something that may be relevant, then look at it, then on the basis of AHA! change your next query or tactic to get better or more stuff Tefko Saracevic

Feedback in searching … • Any feedback implies loops • a completion of a process provides information for modification, if any, for the next process • information from output is used to change previous or create new input • In searching: • some information taken from output of a search is used to do something with next query (search statement) • examine what you got to decide what to do next in searching • a basic tactic in searching • Several feedback types used in searching • each used for different decisions Tefko Saracevic

Feedback types • Content relevance feedback • judge relevance of items retrieved • make decision what to do next • switch files, change exhaustivity … • Term relevance feedback • find relevant documents • examine what other terms used in those documents • search using additional terms • also called query modification & in some systems done automatically • Magnitude feedback • on the basis of size of output make tactical decisions • often the size so big that documents are not examined but next search done to limit size Tefko Saracevic

Feedback types (cont.) • Tactical review feedback • after a number of queries (search statements) in the same search review tactics as to getting desired outputs • review terms, logic, limits … • change tactics accordingly • Strategic review feedback • after a while (or after consultation with user) review the “big” picture on what searched and how • sources, terms, relevant documents, need satisfaction, changes in question, query … • do next searches accordingly • used in reiterative searching Tefko Saracevic

But then There are other ways to look at searching where many of these things are combined Here is one way that looks at the whole search as a process of complex wandering Shifting exploration & feedback are implied Tefko Saracevic

Bates’ Berry-picking model of searching “…moving through many actions towards a general goal of satisfactory completion of research related to information need.” • query is shifting (continually) • as search progresses queries are changing • different tactics are used • searcher (user) may move through a variety of sources • new files, resources may be used • think of your last serious search • isn’t this what you were doing? Elaborated by Marcia Bates UCLA Tefko Saracevic

Berry-picking … • closer to the real behavior of information searchers • new information may provide new ideas, new directions • feedback is used in various ways • question is not satisfied by a single set of answers, but by a series of selections & bits of information found along the way • results may vary & may have to be provided in appropriate ways & means • you go as if berry-picking in a field Tefko Saracevic

A berry-picking evolving search(from the article) “1) The nature of the query is an evolving one, rather than single and unchanging, and 2) the nature of the search process is such that it follows a berrypicking pattern, instead of leading to a single best retrieved set.” Tefko Saracevic

Conclusionssearch tactics have to be mastered Users are not concerned about searching but about finding Effective searching is a prerequisite for finding the right stuff Search tactics are critical for effective searching They need to be understood and followed if you do X you can expect Y in order to get to Y you have to know what X to do Tefko Saracevic

Thank you M. C. Escher Tefko Saracevic

Search strategy & tactics