ACTION RULES

1. ACTION RULES Slides by A A Tzacheva.

2. Knowledge Discovery • The nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data. (Fayyad, et al 1996) Data mining is the process of discovering meaningful new correlations, patterns and trends by sifting through large amounts of data stored in repositories. Gartner Group “The Saying that Knowledge Is Power Is Not Quite True… Used Knowledge Is Power” Edward E. Free

3. Knowledge Discovery • Knowledge Discovery of Databases (KDD) is a new area of research that combines many algorithms and techniques used in artificial intelligence, statistics, databases, machine learning, etc. • KDD is the process of extracting previously unknown, not obvious, new, and interesting information from huge amount of data • Past research on data mining has mostly been focused on techniques for generating rules from datasets

4. Knowledge Discovery Selection Transformation Cleaning Integration Evaluation Visualization Data Mining Data Warehouse Prepared data Patterns Knowledge Knowledge Base Data

5. Knowledge Discovery [Pohle, 2003] Many data mining systems are great in deriving useful statistics and patterns from huge amounts of data, but they are not very smart in interpreting these results, which is crucial for turning them into interesting, understandable and actionable knowledge. Lack of sophisticated tool support for incorporating human domain knowledge into the mining process. This domain knowledge should be updated with the mining results. Mining Process (Fayyad):[[Business Understanding]  [Domain Knowledge]]  [Data Understanding] [Data Preparation]  [Modeling/Mining]  [Evaluation]  [Deployment]

6. Interestingness Function E = [Cond1 => Cond2] Presumptive Objective Associations: two conditions occur together, with some confidence Data Mining Task: For a given dataset D, language of facts L, interestingness function ID, L and threshold c, find association E such that ID,L(E) > c efficiently. Knowledge Engineer definesc

7. Are All the “Discovered” Patterns Interesting? • The rules discovered by data mining algorithm are large and we want a subset of rules, which are interesting, because these algorithms discover accurate rules rather than interesting rules. • Association is interesting if it is easily understood by humans, valid on new or test data with some degree of certainty, potentially useful, novel, or validates some hypothesis that a user seeks to confirm . • There are two aspects of rules’ interestingness that have been studied in data mining literature, objective and subjective measures • Objective measures are data-driven and domain-independent. Generally, these measures evaluate the rules based on the quality as well as the similarity between them, rather than considering the user belief about the domain.

8. Objective Measure Examples Basic Measures for : Domain: card[] Support or Strength: card[  ] Confidence or Certainty Factor: card[]/card[] Coverage Factor: card[]/card[] Leverage: card[]/n – [card[]/n]*[card[]/n] Lift: n  card[]/[card[]*card[]]

9. Problem: Subjective Interestingness • Rule is: • unexpected, if it contradicts the user belief about the domain and therefore surprises the user • novel, if to some extent contributes to new knowledge • actionable, if the user can take an action to his/her advantage based on this rule • Silberschatz & Tuzhilin quantify the actionability in terms of unexpectedness. They claim that: - the most of actionable knowledge is unexpected. - the most of unexpected knowledge is actionable.

10. Actionability - Subjective Actionability [Piatesky-Shapiro & Matheus, 1994] Actionability is seen as an elusive concept because it is difficult to know the space of all rules and the actions to be attached to them.  The task of finding actionable rules is not trivial. Most papers emphasize the importance of finding actionable rules but without well-defined strategy of discovering them.  Data mining is viewed as the process of turning data into information, information into action, and action into value or profit [Silberschatz & Tuzhilin].

11. Argument: Objective Unexpectedness • Unexpectedness [Suzuki, 1997] • /does not depend on domain knowledge/ • If r = [AB1] has a high confidence and r1 = [A*CB2] • has a high confidence, then r1 is unexpected. • [Silberschatz & Tuzhilin] • Unexpectedness is inherently subjective and prior beliefs of • the user form its important component. • [Padmanabhan & Tuzhilin] • A  B is unexpected with respect to the belief • on the dataset D if the following conditions hold: • B   = False [ B and  logically contradict each other] • A   holds on a large subset of D  A*  B holds which means A*  

12. Actionability • The actionability measure is based on the rules’ benefit to the user, that is, the user can do something to his/her interest with the rule. • This measure is very important for the rules to be interesting in the sense that the users always are looking for patterns to improve their performance and establishing better work. • The practical implication of getting information is to improve the business, that is, the information must ensure the success of business for decision-making. Actions can be performed to make the business succeed. • Previous studies quantify the actionability in terms of unexpectedness. They define unexpectedness as a subjective measure of interestingness. They show that the most of actionable knowledge is unexpected and most of the unexpected knowledge is actionable.

13. Actionable Rules and Action Rules • Methods have been proposed to define actionability as an approximation of unexpectedness. • In order to produce unexpected and/or actionable rules, the system must know what the user expects, i.e., his/her existing knowledge or concepts about the domain. • Machine learning research also typically assumes that the domain knowledge is correct or at least partially correct. • Actionable rule mining deals with profit-driven actions required by business decision making • Rules are unexpected if they "surprise" the user, and rules are actionable if the user can do something with them to his/her advantage • For example, a user may be able to change the nondesirable/non-profitable patterns to desirable/profitable patterns • Previous studies show that the measures of unexpectedness and actionability are not mutually exclusive.

14. Actionable Rules Actionable Rules • Limited work has been done on incorporating human concepts, which may be correct, partially correct or completely wrong, in a learning system to generate subjectively interesting rules, e.g., unexpected rules. • In data mining, an approach exists for dealing with the subjective interestingness problem, which employs a postanalysis module at the back-end of the knowledge discovery system. • Such module uses the user's input knowledge about the domain to help him/her identify interesting rules.

15. Actionable Rules Actionable Rules • Although the discovery of interesting and previously unknown patterns are important for data mining applications, it is more important to discover actionable rules. • Data mining is viewed as the process of turning the data into information, the information into action, and the action into value or profit. The later part of the process attracts much research work. • Although both unexpectedness and actionability are important, actionability is the key concept in most applications because actionable rules allow the user to do his/her job better by taking some specific actions in response to the discovered knowledge. • Actionability has also been seen as an elusive concept because it is difficult to know the space of all rules and the actions to be attached to them.

16. Action Rules Action Rules • After reviewing the previous work, we saw that the task of finding actionable rules has been studied for a while, and is not trivial. Most papers emphasize the importance of finding actionable rules, however only mention them in a fuzzy way, without a well-defined strategy of discovering such rules. • In this work, we focus on a special type of rules, called action rules, which are actionable rules. A well-defined algorithm for discovering such rules has been proposed. • These rules can be constructed from classification rules to suggest a way to re-classify objects (for instance customers) to a desired state. • In e-commerce applications, this re-classification may mean that a consumer not interested in a certain product, now may buy it, and therefore may fall into a group of more profitable customers.

17. Action Rules Action Rules • These groups are described by values of classification attributes in a decision table schema.By a decision table we mean any information system where the set of attributes is partitioned into conditions and decisions. • To discover action rules it is required that the set of conditions is partitioned into stable conditions and flexible conditions/attributes. For simplicity reason, we also assume that there is only one decision attribute. • For example, date of birth is a stable attribute, and interest rate on any customer account is a flexible attribute (dependable on bank). • By a decision table we mean any information system of the form S = (X, Ast  AFl {d}), where d  ASt AFlis a distinguished attribute called the decision. • The assumption that attribute d is flexible is quite essential.

18. Action Rules • Action rules: suggest a way to re-classify objects (for instance customers) to a desired state. • Action rules can be constructed from classification rules. • To discover action rules it is required that the set of conditions (attributes) is partitioned into stable and flexible. • For example, date of birth is a stable attribute, and interest rate on any customer account is a flexible attribute (dependable on bank). The notion of action rules was proposed by [Ras & Wieczorkowska, PKDD’00]. Slowinski at al [JETAI, 2004] introduced similar notion called intervention.

19. Action Rules Decision table Any information system of the form S = (U, AFl ASt {d}), where • d  AFl ASt is a distinguished attribute called decision. • The elements of ASt are called stable conditions • the elements of AFl {d} are called flexible conditions Example of action rule: [ (b1, v1 w1)  (b2, v2 w2)  …  (bp, vp wp)](x)  [(d, k1 k2)](x) Assumption: (i)[(1 i  p)  (biAFl)]

20. Action Rules X  a b c d x1 0 S 0 L x2 0 R 1 L x3 0 S 1 L x4 0 R 1 L x5 2 P 2 L x6 2 P 2 L x7 2 S 2 H {a, c} - stable attributes, {b,d} - flexible attributes, d - decision attribute. Decision Table (r1, r2)- action rule: [(b, P S)](x)[(d, L H)](x) Rules discovered: r1 =[ (b, P)  (d, L)] r2 =[(a, 2) ^(b, S)  (d, H)] Notation: (r2)={a,b}, (r2)=d.

21. [Object-Based] Support of Action Rules Action rule r:[ (b1, v1 w1)  (b2, v2 w2)  …  (bp, vp wp)](x)  [(d, k1 k2)] (x) Object x certainly supports rule r in S = (X, A) if:1) (i  p)[ bi(x)= vi] and d(x) = k12) (y  X)(i  p)[ bi(y) = wi ] and d(y) = k23) (b  A – [{bi : 1  i  p}  {d}])[ b(x) = b(y) ] CSupS(r) = card{x: x certainly supports r in S}

22. Action rule r: [(b1, v1 w1)  (b2, v2 w2)  …  (bp, vp wp)](x)  [(d, k1 k2)] (x) Object x possibly supports rule r in S = (X, A) if:1) (i  p)[ bi(x) = vi ] and d(x) = k12) (y  X)(i  p)[ bi(y) = wi] and d(y) =k2 3) (c  ASt)[c(x) = c(y)] [Object-Based] Support of Action Rules PSupS(r) = card{x: x possibly supports r in S}

23. [Rule-Based] Support of Action Rules Action rule r: [(b1, v1 w1)  (b2, v2 w2)  …  (bp, vp wp)](x)  [(d, k1 k2)] (x) Object xX supports rule r in S = (X, A), if there are two rules r1, r2 extracted from S and there exists object y  X satisfying two conditions:  (i  p)[[ bi (r1)]  [bi(x) = vi]] (r1)=dd(x) = k1  (i  p)[[ bi (r2)]  [bi(y) = wi]] (r2)=dd(y) = k2  [[b  ASt]  [b(x) = b(y)]] (r2) = {a1,a2,b1,b2,c1,c2} Confidence: ConfS(r) = RSupS(r)/SupS(r1) (r1) RSupS(r) = card{x: x supports r in S} (r2)

24. Cost of Action Rule Assumption: S= (X, A, V) is information system, Y  X. Attribute b  A is flexible in S and b1, b2 Vb. By S(Y, b1, b2) we mean a number from (0, +] which describes the average predicted cost of approved action associated with a possible re-classification of qualifying objects in Y from class b1 to b2. Object x  Y qualifies for re-classification from b1 to b2, if b(x) = b1. S(Y, b1, b2) = +, if there is no action approved which is required for a possible re-classification of qualifying objects in Y from class b1 to b2 If Y is uniquely defined, we often write S(b1, b2) instead of S(Y, b1, b2).

25. Cost of Action Rule Action rule r: [(b1, v1→ w1)  (b2, v2→ w2)  … ( bp, vp→ wp)](x)  (d, k1→ k2)(x) The cost of r in S: costS(r) = {S(vi , wi) : 1  i  p} Action rule r is feasible in S, if costS(r) < S(k1 , k2). For any feasible action rule r, the cost of the conditional part of r is lower than the cost of its decision part.

26. Extension: Cost of Action Rule RS[(d, k1 → k2)] denotes set of feasible action rules in S having term (d, k1 → k2) as their decision part. Assumption: Among action rules in RS[(d, k1 → k2)] the user identifies rule r of minimal cost value. But that cost value may still be too high to get his approval for implementation of r. The cost of r might be high because of the high cost of one of its sub-terms (bj, vj → wj). In such case, we may look for an action rule in RS[(bj, vj → wj)] of minimal cost value needed to re-classify qualifying objects from vj to wj. Rules short on left side. It was observed such rules were not interesting – active mining.

27. Cost of Action Rule Example: r = [(b1, v1 → w1) … (bj, vj → wj) …  ( bp, vp → wp)](x)  (d, k1 → k2)(x) In RS[(bj, vj → wj)] we find r1= [(bj1, vj1 → wj1)  (bj2, vj2 → wj2)  … ( bjq, vjq → wjq)](x) (bj, vj → wj)(x) Then, we can compose r with r1 and the same replace term (bj, vj → wj) by term from the left hand side of r1: [(b1, v1 → w1)  … [(bj1, vj1 → wj1)  (bj2, vj2 → wj2)  …  ( bjq, vjq → wjq)] … ( bp, vp → wp)](x)  (d, k1 → k2)(x)

28. Search Graph • In order to construct action rules of the lowest cost, we build Search Graph GS, which is a directed graph, that is dynamically built by applying action rules discovered from S to its nodes. • The initial node n0of the graph GS contains information coming from the user, associated with the system S, about what objects he/she would like to reclassify(ex. from the class described by value k1 of the attribute d to the class k2) and what is the current cost, S(k1, k2), of thereclassification k1 → k2 . • Any other node n in GS shows an alternative wayto achieve the same reclassification with a cost that is lowerthan the cost assigned to all nodes which are preceding n in GS.

29. Search Graph Assume that N is the set of nodes in graph GS and n0 is its initial node. For any node nN, by f(n) = (Yn, {[ vn,j → wn,j , S(vn,j, wn,j)]} j In) we mean its domain(set of objects in S), set of actions needed to reclassify objects from Yn, and their cost, where Yn X. We say that action rule r, discovered from S, is applicable to node n if: Yn  RSupS(r) ≠ Ø (k  In)[r RS[ vn,kj → wn,k]]

30. Information System SRS [(d, k1 →, k2)] r1 r2 r3 rn n0 = {[ k1 → k2 , S (k1, k2)]} r = [(b1, v1 → w1) ^ (b2, v2 → w2)^ … ^( bp, vp → wp)](x) => (d, k1 → k2)(x) n1 = {[ v1 → w1 , S (v1, w2)], [ v2 → w2 , S (v2, w2)], …, [ vp → wp , S (vp, wp)]} r1 n2 n3 r4 rn rj nn Figure 4. Lowest Cost Reclassification Search Graph in a standalone system S Minimal Cost Reclassification Search Graph for S.

31. Search GraphProperties Property 1. Let f(n0) = (Y, {[k1→k2, S(k1,k2)]}), f(n) = (Yn, {[ vn,,k → wn,,k , S (vn,,k, wn,,k)]}k  In). The cost assigned to the node n for reclassifying x  Yn from k1to k2 is equal to: Costk1→k2(n, x) = {S(vn,,k, wn,,k): k  In} Property 2. If node n2 is a successor of the node n1, then Confk1→k2(n2, x)Confk1→k2(n1, x) Property 3. If node n2 is a successor of the node n1, then Costk1→k2(n2, x)Costk1→k2(n1, x)

32. Search for Action Rules [Tzacheva & Ras] • We propose A* type algorithm for speeding up the construction ofthe shortest path from the root to the goal nodein graph GS. • A* is probably one of the most popular search algorithms in AI. It is an informed, optimal search algorithm, which uses a heuristic estimate of remaining distance to the goal by means of a heuristic function h(N) . • We assume that user provides three threshold values: 1 - threshold for minimum confidence of action rules. 2 - threshold for maximum cost of action rules. 3 - threshold for minimum feasibility of action rules.

33. Heuristic Method - A* • We assume that: h(ni) = [cost(ni,Yi) - 2]/3 • Heuristic value h(ni) is associated with any node ni in G. It shows the maximal number of steps that might be needed to reach the goal. • Also, we assume that: g(ni) is the number of edges to the current node • Then, we associate an estimated path length to the goal for each node as follows: f(ni) = h(ni) + g(ni)

34. Proposed Algorithm - A* Initialize Q with search node [([conf(no),h(no)],[no])] as the only entry; Initialize domain of no (given by user) as Yo. If Q is empty, fail. Else, pick search nodes from Q with a least value of f. If two search nodes in Q have the same least value of f assigned to them, if an ontology is available, pick search node s from Q with the highest value of Ont(s). If state(s) is a goal and conf(s)1, return s (we have reached the goal). Otherwise remove s from Q. Find all children of state(s) and create all the one-step extensions of s to each descendant. If state(s1) is a child of state(s) and r is the action rule applied to s in order to move from s to s1, then initialize Ystate(s1) as Ystate(s) DomS(r) and if an ontology is available, Ont(s1) as Ont(r) 6.Add all the extended paths to Q; 7. Go to step 2.

35. Example n1$800, 85% n378% n4$500, 85% n578% n677% n2$700, 88% Cheapest rule: cost $1000, confidence 95% All action rules Action rules extracted from classification rules Below $200 User request: find action rule reclassifying patients from one diagnostic class (k1) into another (k2) where the cost less than $200, the confidence is 80% and the minimum gain is $50. h(n2) = [cost(ni,Yi) - 2]/3 = (700 – 200)/50 =10 f(n2) = h(ni) + g(ni) = 10 + 1 = 11 f(n4) = h(ni) + g(ni) = (500-200)/50 + 2 = 8

36. Implementation and Testing • The heuristic strategy for lowest cost reclassification – LowestCostReclassifier software,is implemented in C++ using the Microsoft Visual Studio 7.0 IDE and compiler. • The user is asked to enter the attribute in which he/she is interested in reclassifying, and the desired values. Also the users chooses the following thresholds: 1 - minimum confidence of action rules 2 - maximum cost of action rules 3 - minimum feasibility of action rules. And the currently known to the user cost of reclassification • The action rules above are of the following form: (attribute, valueFrom - > valueTo | cost ) => (attribute, valueFrom -> valueTo | cost) confidence • The LowestCostReclassifier software was tested and applied to three different databases. Two in medical domain, and one in financial domain.

37. Binding to Thrombin Database • The first database, is the Binding to Thrombin database, which is used for drug design, and provided in the KDD Cup 2001 Competition. • Drugs are typically small organic molecules that achieve their desired activity by binding to a target site on a receptor. The first step in the discovery of a new drug is usually to identify and isolate the receptor to which it should bind, followed by testing many small molecules for their ability to bind to the target site. • This leaves researchers with the task of determining what separates the active (binding) compounds from the inactive (non-binding) ones. Such a determination can then be used in the design of new compounds that not only bind, but also have all the other properties required for a drug (solubility, oral absorption, lack of side effects, appropriate duration of action, toxicity, etc.).

38. Binding to Thrombin Database • The data set consists of 1909 compounds tested for their ability to bind to a target site on thrombin, a key receptor in blood clotting. • Each compound is described by binary features, which describe three-dimensional properties of the molecule.Biological activity in general, and receptor binding affinity in particular, correlate with various structural and physical properties of small organic molecules. The task with KDD Cup 2001 was to determine which of these properties are critical in this case and to learn to accurately predict the class value: Active or Inactive. • In this testing we use the class attribute, which has value A for active and I for inactive, as the re-classification attribute for the actionRules. In this way, we provide suggestions to the user to what molecular properties can be changed in order to reclassify the chemical compound from inactive to active class, in order to bind to thrombin.

39. Binding to Thrombin Database • We found the following results with LowestCostReclassifier : decisionAttribute = activityvalueFrom = 0valueTo = 1 dataFile= thrombinnumberOfObjects: 1908 minConfidenceL1 = 0.65minFeasibilityL3 = 0.0001knownCost = 0.3maxCostL2 = 0.01 ---- Goal Node:(f304, 1->0 | 0.12327) => (activity, 0->1 | 0.3) 1(f172, 0->1 | 0.00472118) => (f304, 1->0 | 0.12327) 0.998424 ---- Action Rule of Min Cost Found: ----(f172, 0->1 | 0.00472118) => (activity, 0->1 | 0.3) 0.998424

40. Insurance Company Benchmark database • We also tested the LowestCostReclassifier with a database in the financial domain, the Insurance Company Benchmark (COIL 2000) database used with the CoIL 2000 Challenge. • The data contains 5,822 tuples. The features includes product usage data and socio-demographic data derived from zip area codes. • The data was supplied by the Dutch data mining company Sentient Machine Research and is based on a real world business problem. • In our testing the user would like to reclassify the attribute Contribution car policies from a value of 5 to 6.

41. Insurance Company Benchmark database • We found the following results with LowestCostReclassifier : decisionAttribute = Contribution car policiesvalueFrom = 5valueTo = 6 dataFile= InsurancenumberOfObjects: 5822 minConfidenceL1 = 0.72minFeasibilityL3 = 0.0001knownCost = 0.4maxCostL2 = 0.02 ---- Goal Node:(Private health insurance, 3->4 | 0.02423844) => (Contribution car policies, 5->6 | 0.4) 0.833333(High level education, 2->3 | 0.00176027) ^ (Social class B1, 2->4 | 0.0146667) => (Private health insurance, 3->4 | 0.3) 0.714286 ---- Action Rule of Min Cost Found: ---- (High level education, 2->3 | 0.00176027) ^ (Social class B1, 2->4 | 0.0146667) => (Contribution car policies, 5->6 | 0.4) 0.714286

42. Breast Cancer Database Time • The next database used is a breast cancer database. It was obtained at the University of Wisconsin Hospitals, Madison from Dr. William H. Wolberg. • It contains a class attribute which classifies the tumor as benign or malignant. The rest of the attributes contain descriptions of common factors radiologists, and pathologists examine in order to place the diagnosis, such as Clump Thickness, Uniformity of Cell Shape, Bare Nuclei, etc. • The database has 700 instances, Benign: 458 (65.5%) and Malignant: 241 (34.5%). • We used the class attribute as the re-classification attribute with LowestCostReclassifier. This way, we will provide suggestions/actions to be undertaken in order to change the class from malignant to benign.

43. Breast Cancer Database • We found the following results with LowestCostReclassifier : decisionAttribute = ClassvalueFrom = 4valueTo = 2 dataFile= brcancernumberOfObjects: 699 minConfidenceL1 = 0.7minFeasibilityL3 = 0.0001knownCost = 0.3maxCostL2 = 0.00059.. . !!!!!!!List of sifted action rules is emptyi.e. no feasible action rules were found, sothis state/child has no descendents:(Marginal Adhesion, 1->3 | 0.00128449) @@@@@@@@ Traversing next nodeThe list Q of nodes traversed was empty.

44. Breast Cancer Database ---- Best Node ----(Uniformity of Cell Size, 8->1 | 0.00179706) ^ (Normal Nucleoli, 5->3 | 0.00721622) => (Class, 4->2 | 0.3) 1(Marginal Adhesion, 1->3 | 0.00128449) => (Uniformity of Cell Size, 8->1 | 0.00179706) 0.85 ---- Action Rule of Min Cost Found: ----(Marginal Adhesion, 1->3 | 0.00128449) => (Class, 4->2 | 0.3) 0.85 • In this case, the goal could not be reached, possibly because the desired maximum cost specified, 0.00059was too low. However, still the best node found thus far was returned, which cost 0.00128449 is still lowerthat the currently known cost to the user 0.3.

45. Conclusions • Initially we built action rules from the shortest classification rules. If the cheapest action rule does not satisfy the user thresholds, then we follow a heuristic approach using A* algorithm for building a search graph G. It allowed us to identify an action rule of the lowest cost considering three thresholds the user provides: min confidence, max cost, and min feasibility. • Further, we observed that even the maximum cost threshold is not reachable, we will still return the best node found thus far, which cost would still be lower than the currently known cost to the user. • In that sense, the leaves in our graph G and the nodes close to them would represent the most actionable knowledge and the same the mostly unexpected/interesting knowledge related to a desired reclassification of objects.

46. Final Claims Subjective measure: user-driven, domain-dependent. Include unexpectedness [Silberschatz and Tuzhilin, 1995], novelty, actionability [Piatesky-Shapiro & Matheus, 1994]. Claim1[Suzuki, Padmanabhan & Tuzhilin] Unexpectednessis partially an objective concept. A  B is unexpected with respect to the belief  on the dataset D if the following conditions hold: B   = False [ B and  logically contradict each other] A   holds on a large subset of D A*  B holds which means A*   Our Claim: Actionability is partially an objective concept. Actionability measure = Cost of an action rule

47. Final Claims Questions? • Our Claim: the most cheap rules are most of actionable • Claim 2 [Silberschatz & Tuzhilin] • the most of actionable rules are unexpected • Our Claim: The most cheap rules are unexpected • References: • Z. Ras, A. Tzacheva, L.-S. Tsay, “Action Rules”, in Encyclopedia of Data Warehousing and Mining, • (Ed. J. Wang), Idea Group Inc., 2005, will appear • A. Tzacheva, Z. Ras, "Action rules mining", in the Special Issue on Knowledge Discovery, • International Journal of Intelligent Systems, Wiley, 2005, will appear • A. Tzacheva, Z. Ras, “Discovering non-standard semantics of semi-stable attributes”, • Proceedings of Flairs-2003, St. Augustine, Florida, AAAI Press, 2003, 330-334 Thank You