Chapter 7 Similarity Based Retrieval

1. Chapter 7Similarity Based Retrieval Stand 20.12.00

2. Recommended References (1) • The retrieval algorithms are presented in this chapter in some detail. Further information can be found in the following original literature: • M. Lenz: Case Retrieval Nets as a Model for Building Flexible Information Systems. Dissertation Humboldt Universität Berlin 1999. • J. Schaaf: Über die Suche nach situationsgerechten Fällen im fallbasierten Schließen. Dissertation Kaiserslautern 1998, Reihe DISKI 179, infix Verlag. • S.Wess: Fallbasiertes Problemlösen in wissensbasierten Systemen zur Entscheidungsunterstützung und Diagnostik. Dissertation Kaiserslautern 1995, Reihe DISKI 126, infix Verlag • Schumacher, J. & Bergmann, R. (2000). An effective approach for similarity-based retrieval on top of relational databases. 5th European Workshop on Case-Based Reasoning, Springer Verlag.

3. General Remarks • We distinguish to kinds of retrieval: • retrieval algorithms: They operate mainly on a fixed data base • agent based retrieval: This is mainly for search processes in many, not necessarily fixed data bases. This search is often distributed. • In principle, both techniques can be applied in any situation which is, however, not recommended. • This chapter deals with retrieval algorithms. • Search (with agents) will be discussed in chapter 14 where we consider several suppliers.

4. Motivation • In database retrieval the access to the desired data is done by presenting a certain key. • In information retrieval systems also a key word is presented. The problem with this is that one gets either no answer (silence) or very many answers (noise). • There are several situations in e-c where no exact queries can be raised, e.g.: • Which is the best available product with respect to customer demands ? • Which is the best profile covering a given customer ? • What is the most likely reason for the customer‘s complaint ?

5. Efficient Retrieval • Efficiency and accuracy are the two important issues for retrieval. • The efficiency of the different retrieval methods depends very much on the following: • The representation of the objects • The base structure • The similarity measure • The accuracy of the intended answer • These characteristics depend on the other hand on the domain and the specific properties of the application.

6. The Task • Central task of the retrieval: • given: • A base CB = {F1,...,Fn} ( a case base, a product base, ...) and • a similarity measure sim and • a Query: Q (new problem, demanded product etc.) • wanted either: 1. The most similar object Fi OR 2. the m most similar objects {F1,...,Fm} (ordered or unordered) OR 3. All objects {F1,...,Fm} which have to Q at least a similarity simmin • Problem: How to organize a case base for efficient retrieval?

7. Retrieval Methods Retriction w.r.t.Similarity Appropriate for ... Typ Method small case bases simple similarity brute force sequential search no small number of attributes large case bases reflexivity, monotony no class similarity kd-tre (Wess et al., 1993) index based complex similarity small case bases reflexivity, nonotony triangle inequality Fish & Shrink (Schaaf, 1996) Retrieval Nets (Burkhard & Lenz, 1996) few numeric attributes large case bases monotony, no class similarity dyn. databasequeries monotony no class similarity linear aggregation functions simple similarity large case bases dynamic case bases SQL Approximation (Schumacher & Bergmann, 2000)

8. Sequential Retrieval Data structures Types: SimObject = RECORD object: object; similarity: [0..1] END; SimOjectQueue = ARRAY[1..m] OF SimObject; Variables: scq: SimObjectQueue (* m most similar objects *) cb: ARRAY [1..n] OF object (* Object base *) scq[1..m].similarity := 0 FOR i:=1 to n DO IF sim(Q,cb[i]) > scq[m].similarity THEN insert cb[i] in scq RETURN scq Retrieval algorithm

9. Properties of Sequential Retrieval • Complexity of sequential retrieval: O(n) • Disadvantages: • Problems if base very large • Retrieval effort is independent of the query • Retrieval effort is independent of m • Avantages: • Simple implementation • No additional index structures needed • Arbitrary similarity measures can be used

10. base index structure retrieval generates similarity Index Oriented Retrieval • Preprocessing: Generates an index structure • Retrieval: Makes use of the index structure for efficient retrieval

11. Retrieval with kd-Trees(S. Wess) • k-dimensional binary search tree (Bentley, 1975). • Idea: Decompose data (i.e. case base) iteratively in smaller parts • Use a tree structure • Retrieval: • Searching in the tree top-down with backtracking

12. Definition: kd-Tree • Given: • k ordered domains T1,...,Tk of the attributes A1,...,Ak, • a base CB Í T1x...xTk. and • some parameter b (bucket size). A kd-tree T(CB) for the base CB is a binary tree recursively defined as: • if |CB| £ b: T(CB) is a leave node (called bucket) which is labelled with CB • if |CB| > b: T(CB) then T(CB) is a tree with the properties: • the root is labelled with an attribute Ai and a value viÎTi • the root has two kd-treesT£ (CB£) and T>(CB>) as successors • where CB£ := {(x1,...,xk) Î CB | xi£ vi} and CB> := {(x1,...,xk) Î CB | xi> vi}

13. Properties of a kd-Tree • Ein kd-tree partitions a base: • the root represents the whole base • a leave node (bucket) represents a subset of the base which is not further partitioned • at all inner nodes the base is further partitioned s.t. base is divided according to the value of some attribute.

14. A 1 A2 <35 >35 F 50 G C A 2 A 2 40 <30 >30 <35 >35 30 E H B 20 C(20, 40) H(70, 35) F(50, 50) A 1 D E(35, 35) I(65, 10) G(60, 45) <15 >15 10 I A A(10, 10) D(30, 20) B(15, 30) 10 20 30 40 50 60 70 A1 Example for a kd-Tree CB={A,B,C,D,E,F,G,H,I}

15. Generating kd-Trees (1) Algorithm: PROCEDURE CreateTree(CB) : kd-tree IF |CB| £b THEN RETURN leave node with base CB ELSE Ai := choose_attribute(CB); vi := choose_value(CB,Ai) RETURN Tree with root labelled with Ai and vi and which has the two subtrees CreateTree({(x1,...,xk) Î CB | xi£ vi}) and CreateTree({(x1,...,xk) Î CB | xi> vi})

16. 25% 25% 25% 25% Ti q1 q2 q3 Selection of Attributes • Many techniques possible, e.g. the use of entropy • typical: Interquartil distance Number of occurrences Interquartil distance iqr = d(q1,q2) Selection of attribute with largest interquartil distance Iqr

17. largest gap Selection of the Values • Two methods: • Median splitting: choose the median d2 as partition point • Maximum splitting:

18. Idea for an algorithm: 1. Search the tree top-down to a leave 2. compute similarity to objects found 3. Determine termination by BWB-test 4. Determine additional candidates using a BOB-test 5. If overlapping buckets exist then search in alternative branches (back to step 2) 6. Stop if no overlapping buckets Retrieval with kd-Trees BWB test Ball-within-Boands Most similar object in the bucket ? BOB test: Ball-overlap-Boands overlap Most similar object in the bucket ?

19. Retrieval with kd-Trees Algorithm PROCEDURE retrieve(K: kd-tree) IF K is leave node THEN FOR each object F of K DO IF sim(Q,F) > scq[m]. THEN insert F in scq ELSE (* inner node *) If Ai is the attribute and vi the value that label K IF Q[Ai] £ vi THEN retrieve(K£) IF BOB-Test is satisfied THEN retrieve(K>) ELSE retrieve(K>) IF BOB-Test is satisfied THEN retrieve(K£) IF BWB-Test is satisfied THEN terminate retrieval with scq

20. BOB-Test:Ball-Overlap-Bounds Are there more similar objects then the m-most similar object found in the neighbor subtree ? A2 n-dimensional hyper ball overlap ? A1 m-most similar object in scq boundaries of the actual node

21. A2 n-dimensional hyper balll ? m-most-similar object in scq A1 boundaries of the actual node BWB-Test:Ball-Within-Bounds Is there no object in the neighboring subtreethat is more similar than the m-most-object?

22. Restrictions to the Applicable Similarity Measures • retrieval with a kd-tree guarantees the m-most similar object if the similarity measure satisfies the following restrictions: • Compatibility with the ordering and monotonicity • " y1,..., yn, x1,..., xn, xi’ if yi <i xi <i xi’ or yi>i xi>i xi’ then sim( (y1,...,yn) , (x1,...,xi,...,xn) ) ³ sim( (y1,...,yn) , (x1,...,xi’,...,xn) )

23. Properties of Retrieval with kd-Trees • Disadvantages: • Higher effort to built up the index structure • Restrictions for kd-trees: • only ordered domains • problems with unknown values • Only monotonic similarity measures compatible with the ordering • Advantages: • Efficient retrieval • Effort depends on the number m of objects to find • Incremental extension possible if new objects arise • Storage of the objects in a data base possible • There are improvements of kd-trees (INRECA)

24. Case Retrieval Nets (Lenz & Burkhard) • We formulate the techniques not only for cases but for more general situations. • Partitioning of object information in information units(e.g. attribute-value-pairs) • Each information unit is a node in the net • Each object is a node in the net • Information units which have a similarity > 0 are connected with strenght = similarity. • For the retrieval information units of the query are activated. • The activity is propagated through the net until object nodes are reached. • The activity at the object nodes reflects the similarity to the query.

25. Concepts (1) • An information entity (IE) is an atomic knowledge unit, e.g. an attribute-value-pair; it represents the smallest knowledge unit for objects, queries or cases. • An object (e.g. a case) is a set of information entities.A retrieval Net (Basic Case Retrieval Net, BCRN) is a 5-tupel N=(E, C, s, r, P) with: • E is a finite set of information entities • C is a finite set of object nodes • s is a similarity measure: s: E x E -> IRs (e,e’) describes the local similarity between two IUs e, e’ • r is a relevance function: r: E x C -> IRr(e,c) describes the relevance (weight) of the IU for the object c • P is a set of propagation functions pn: IRE -> IR for each node n Î E È C.

26. Price 1099.- r 1/3 case1 1/3 Beach close 1/3 0.5 Region: Mediterranean 0.5 1/3 s case2 1/3 Price: 1519.- 0.4 1/3 Beach distance medium 0.9 1/3 case3 Price: 1599.- 1/3 Region: Caribic 1/3 Example Objectnode • IEs

27. Concepts (2) • An activation of a BCRN is a function a: E È C -> IR. • The activation a(e) of an IE e describes the relevance of this IE for the actual problem. The influence of an IE on the retrieval depends on that value and its relevances r(e,c) for the objects c. • The activation at time t is a function at: E È C -> IR defined by: • IEs: • objects:

28. Retrieval • Presented: Query Q, consisting of some set of IEs • Retrieval: 1. Determination of the activation a0 by: 2. Similarity propagation: for all e Î E, which have a connection to some activated IU. 3. Relevance propagation: for all object nodes c Î C, which have a connection to some activated IU.

29. Example cont´ed Objectnode • IEs Price 1099.- r 1/3 case1 1/3 Beach close 1/3 0.5 Region: Mediterranean 0.5 1/3 s case2 1/3 Price: 1519.- 0.4 1/3 Beach distance medium 0.9 1/3 case3 Price: 1599.- 1/3 1/3 Region: Caribic 1/3

30. Example cont´ed Objectnode • IEs 1 Price 1099.- r 1/3 case1 1 1/3 1/3 Beach close 1/3 Price: 1550.- 0.5 Region: Mediterranean 0.5 1/3 1 s case2 1 1/3 1/3 Price: 1519.- 0.4 1/3 Beach distance medium 0.9 0.9 1/3 case3 Price: 1599.- 1/3 0.63 0.9 Region: Caribic 1/3

31. Properties of Retrieval Nets • Disadvantages: • High costs to construct the net. • New query nodes may be necessary for numerical attributes. • Many propagation steps necessary if the degree of connectivity is high (i.e. many similar IEs) • Advantages: • Efficient retrieval • Effort depends on the number of activated IEs of the query • Incremental extension possible if new objects arise • There are improvements of this approach (e.g.. Lazy Spreading Activation)

32. Retrieval with “Fish and Shrink “Schaaf (1996) • Object representation is partinioned in different aspects. • An aspect is a complex property Examples for aspects in medicine: EKG, X-ray image. • For each aspect there are individual similarity measures defined. • Assumption: similarity computation for aspects is expensive. • Total similarity is computed by weighted combinations (aggregation) of aspect similarites. • Weights are not static but given by the query. • Approach for retrieval: • look ahead computation of the aspect similarities between certain objects • Net construction • Test for similarity between the query and the test object • Conclusion for similarity of other objects possible without computation

33. Concepts (1) • An object F consists of several aspectsai(F) • for each aspect there is an aspect distance functiondi(ai(Fx), ai(Fy)) Î [0..1] (short: di(Fx, Fy)). • a view is a weight vector W=(w1,...,wn) mit w1+...+wn=1Observe: The view is part of the query and presented by the user. • A view distance for two objects and a view is a function VD(Fx,Fy,W) = w1d1(Fx,Fy) +...+ wndn(Fx,Fy) • A case Fx is view neighbor of the object Fy w.r.t. a view distance VD, if VD(Fx,Fy,W) < c holds (e.g. c=1).

34. Assumptions for Fish & Shrink • If an object F is not similar to query A it is an indication that other object s F’ which are similar to F are also not similar to query A. • More precise: Assumption of the triangle inequality is made: F 1. A T 2.

35. Algorithm (Idea) • Given: • A base with precomputed aspect distances between certain objects (not necessarily between all objects) • query A and view W (weight vector) • Idea of the Algorithm • Determine for each object of the base a distance interval (initially [0..1]) • Choose a test object T and determine the view distance between A and T (expensive). • determine for most objects of the base the new smaller distance interval by using inequalities 1) and 2). • Iterate these steps until the intervals are small enough

36. T3 T1 T2 Example 0 distance interval of object F (not shown) to the query 1

37. Algorithm • Given: base CB, query A, view W • Algorithm: FOR EACH F ÎCB DO F.mindis:= 0 F.maxdis := 1 END WHILE NOTOK OR Interrupted DO determine precision line PL Choose (fish) an object T with T.mindis = PL testdis, T.mindis, T.maxdis := VD(A,T,W) (*view distance*) FOR EACH F ÎCB AND F view neighbor of T AND F.mindis ¹ F.maxdis DO basedis := VD(T,F,W) F.mindis := max(testdis-basedis, F.mindis) (*Shrink*) F.maxdis := min(testdis+basedis, F.maxdis) END END

38. Predicate OK and Precision Line (1) • By a suitable choices of OK and precision line PL different retrieval tasks can be satisfied: M|M|=k PL S PL N a) All objects better than a threshold S b) The best k objects unordered

39. M|M|=k PL N c) The best k objects ordered, but no exact distances Predicate OK and Precision Line (2) M|M|=k PL N d) The best k objects ordered with exact distances

40. Predicate OK and Precision Line (3) • Formal definition for OK and PL for variation a) (all cases better than a threshold)

41. Properties of Fish & Shrink • Disadvantages : • aspect distances between objects have to be precomputed • the distance function has to satisfy the triangle inequality • Advantages: • Flexible distance computation to the query (views) • different retrieval tasks can be performed • Efficient because many distance computations can be saved • Can be used as anytime-algorithm • Suitable If very expensive similarity computations occur (e.g. for graph representations)

42. Retrieval by SQL Approximation: Application Scenario(Schumacher & Bergmann, 2000) • Product database exists and is used for many services in the business processes • Very huge number of products • Product database changes continuously and rapidly • Product representation is very simple: usually a list of attribute values

43. Possible Approaches • Retrieval inside the database • Solution depends on database system • Retrieval on top of the database • Bulk-loading all products: duplication of data, inefficient, consistency problems • Building an index: consistency problem remains • MAC/FAC Approach: Approximating similarity-based retrieval with SQL queries: no duplication of data, no consistency problem

44. Two Level Retrieval • Idea: Two levels, one for reducing the search space and one for the retrieval itself Mac/Fac (Many are called; Few are chosen) • 1. Preselection of possible candidates MQÍ CB MQ = {F Î CB | SIM(Q,F) } • 2. Ordering of the (few) candidates according to sim Application of sequential retrieval • Problem: To define the predicate SIM

45. Examples for Preselection Predicates • Partial equality: • SIM(Q,F) if Q and F coincide in at least one attribute value • Local similarity: SIM(Q,F) if Q and F are sufficiently similar with respect to each local measure • Partial local similarity: SIM(Q,F) if Q and F are sufficiently similar with respect to one local measure

46. Properties of Two Level Retrieval • Advantages: • more efficient if only few objects are preselected • Disadvantages: • retrieval errors possible, i.e. • a-errors: Sufficiently similar objects may not be preselected ® retrieval may be incomplete • The definition of the predicate SIM is usually difficult

47. k most similar cases 1 query n SQL queries cases Approximating Similarity-Based Retrieval Query Results XML Similarity Measure DB queryconstruction Similarity-based Retrieval sort by similarity SQL modified by other applications Database

48. Assumptions • Representation: Attribute-Value • Similarity: • Local similarity measures for attributes • Global similarity: weighted average • Efficiency Assumption: Cases are distributed equally across n-dimensional space

49. Q Similarity-based Retrieval • The most similar cases lie within a hyper-rhombus centered on the query • Less similar cases are all outside the rhombus k most similar case Ck

50. SELECT a1, a2 FROM table WHERE (a1 >= min1 AND a1 <= max1) AND (a2 >= min2 AND a2 <= max2) Selected cases lie within a hyper-rectangle All cases within the rectangle have a minimum similarity More similar cases can lie outside the rectangle Ck Q SQL Query