GENERIC ENTITY RESOLUTION WITH NEGATIVE RULES

1. GENERIC ENTITY RESOLUTION WITH NEGATIVE RULES Steven Euijong Whang · Omar Benjelloun · Hector Garcia-Molina Compiled by – DarshanaPathak

2. CONTENTS • Introduction • Example • ER-N model • GNR Algorithm • ENR Algorithm

3. CONTENTS • How to choose negative rules? • Conclusion • References

4. Introduction • What is entity resolution? A two step process: • Identifying records that refer to same real-world entity • Merge them together • Also known as record linkage, merge-purge, deduplication etc. • Application-specific, complex and error-prone process.

5. Introduction • Why is this so complex? Most of the times because data is – • Ambiguous • Missing or incomplete • Incorrect These things are difficult to capture no matter what logic is used to decide whether records match or how they should be merged!

6. Introduction • What are negative rules? • Integrity constraints – Rules that tell us what data is invalid. • Sanity check in order to remove inconsistencies. e.g. • One person with 2 genders, • Same address location with two different street names.

7. Example • General ER-Process: • Match Function M(r1, r2) = true if same, false if different. Denoted by (r1 ≈ r2) or (r1 ≠ r2) • Merge Function µ(r1, r2) = ‹r1, r2›

8. Example Match r1 and r2, if (r1 ≈ r2), merge them r12 | Pat, Patricia | 999-04-1234 | F Match r12 with r3, if (r12 ≈ r3), merge them r123 | Pat, Patricia | 999-04-1234 | F, M Problem: Violation of negative rule

9. Example • Why was this constraint not enforced during ER process? • Constraints are much more complex (may be a big computer program considering many factors) • Patches added to program over time by different people • Condition acceptable during ER process and fixable.

10. Example • Resolving the inconsistency Because, r123 is not acceptable as a final merge, • Unmerge r123 into {r12, r3}. • No two final records can have same SSN. • R12 and r3 can not be in final record set. • Problem occurred because r1 was initially merged with r2 instead of r3. In practice, there will be no obvious ordering of merging!

11. ER-N Model • Basic properties of Match and Merge functions: • Idempotence: Any record matches itself and merging a record with itself yields the same record. Idempotence: ∀r, r ≈ r and r, r = r . • Commutativity: If r1 matches r2, then r2 matched r1. Commutativity: ∀r1, r2, r1 ≈ r2 iff r2 ≈ r1, and if r1 ≈ r2, then <r1, r2> = <r2, r1>.

12. ER-N Model • Basic properties of Match and Merge functions: • Domination: r1 ≤ r2 Record r1is dominated by r2if both records refer to the same entity, but r2’s information “includes” that of r1. Thus r1 is redundant information. We can have r1 ≤ r2, whenever r2 = <r1, r’> for some r’.

13. ER-N Model – Merge Closure • Merge closure A merge closure ī contains all the possible records that can be generated from I using M and μ, where I = {r1, ……, rn}. • Definition: The merge closure ī ofI satisfies the following conditions: 1. I ⊆ ī 2. ∀r1, r2 ∈ īs.t. r1≈ r2, <r1, r2> ∈ī. 3. No strict subset of īsatisfies conditions 1,2.

14. Algorithm overview: • Steps to find closure from a set I of records: • Start with empty ī . • Loop until I is empty. • r = record from I. Remove r from I. • For all r’ from ī, follow steps 1 to 2. 1. If r’ ≈ r then merged = <r,r’> 2. If merged not in I U ī U {r} then I = I U {merged} • ī = ī U {r} • Return ī. * This basic algorithm does not consider negative rules.

15. Time to apply negative rules! Classified according to number of arguments: • Unary negative rule: Checks if a record r is valid by itself. • Binary negative rule: Checks if two different records r1 and r2 can coexist • Two inconsistent records cannot coexist in ER solution. • Match & merge rules and negative rules cannot be combined together.

16. Properties of negative rules • A set of records is inconsistent if there exists a single • Record violating a unary negative rule and/or • A pair of records violating binary negative rule. • Commutativity for negative rules: For all r1 and r2, if r1 is not consistent with r2, then r2 is not consistent with r1.

17. ER-N Model • According to definition of ER-N Model – Given an instance of I and the merge closure ī, an ER-N of I is a consistent set of records J such that – • J is a maximal consistent subset of ī. • For all records r (ī - J), • There exists r’ J s. t. r r’ or • J U {r} is inconsistent.

18. Back to our example • ī = {r1, r2, r3, r12, r13, r23, r123} • The instance {r13, r2} is a valid ER-N solution. where r13 = Pat | 999-04-1234 | M, r2 = Patricia | … | F • This satisfies all conditions of ER-N model.

19. Resolving Inconsistencies • Late approach: • Using match & merge rules, ER solution is generated • Solution is checked for inconsistencies. • Appropriate fixes are applied to remove inconsistencies with the guidance of domain expert – solver. • Early approach: • With the help of solver, start identifying records that we want in the final answer J. • Start fixing problems between the selected records in J and other records not yet selected. * Early approach is preferred over late approach.

20. Resolving Inconsistencies • Ways inconsistencies can be fixed: • Discard data: Solver may decide to drop the record. • Forced merge: Solver decides that two inconsistent records should have been merged. • Override negative rule: Solver decides that flagged record(s) are indeed consistent i. e. negative rule was wrong flagging that record as inconsistent. e.g. Comfort Inn vs Comfort Inn Milton

21. The GNR Algorithm General algorithm for negative rules: • Solver plays key role in making decisions. • If no solver is available, algorithm makes choice at random (!) or based on some heuristic. e.g. A record with more fields available is preferable to the one with fewer fields. • The solution generated without solver may not be the “most desirable solution”.

22. The GNR Algorithm • Algorithm overview: • Generate closure ī using ER algorithm & Set S = ī. • Select set of non-dominated records (ndS) from ī. • Select record r from ndS with the help of solver. • S = S \ {r} means remove r from S. • If r is self-inconsistent, discard r. Continue from step II. • Else J = J U {r} • Remove all records from S that are either inconsistent with r or dominated by r. • Continue from step II till S is empty. • Return J.

23. Back to our example • ī = {r1, r2, r3, r12, r13, r23, r123} • Select r123, but it is inconsistent, so discard it. • Select one from {r12, r13, r23}  r13 • Remove all records dominated by or inconsistent with r13 • So, S = {r2, r23} • We discard r23 because its internally inconsistent. • Final Solution {r13, r2}

24. Things to ponder • An important matric for GNR algorithm is the “Human effort” of the solver. • How to calculate human effort? • Entity resolution is inherently expensive operation. • If we apply negative rules, it becomes more expensive!

25. Techniques to reduce cost • Semantic partitioning: Data is divided into independent blocks using semantic knowledge. e.g. Category: Book, camera, snacks, … The technique is commonly known as blocking. • Exploiting properties Exploit properties of match and merge rules to make it possible to find correct solution with less effort.

26. Exploiting properties • Desirable properties for M and • Associativity: Merge order is irrelevant. • Representativity: Merged record represents its base records and matches with all records that match with the base records. May represent simple identity or a range.

27. ENR algorithm • Enhanced algorithm for negative rules - Makes things simpler and more efficient. • Rather than looking at entire merge closure of I, partition I and look at merge closure of each partition. • This partitioning is different that blocking, as these do not assume any semantic knowledge. • This is similar to our first algorithm, except once records r and r’ are merged, they are removed from further consideration. • This works because any future records that match r or r’ match the merged record <r, r’>.

28. Some more negative rules • Borderline cases: In practice, many times rules are written such that all borderline cases are flagged, so that solver can check them out. e. g. The case in which name, DOB, address, gender all match except one digit of SSN. • NameAddr negative rules: Special string comparison rules for name and address checks. e. g. Are two street with similar names are “too far apart” to be in the same record?

29. How to choose negative rules? • Important: Design negative rules that do not generate too many unnecessary checks. • Always remember, more the flagged records, more are the human efforts required. • Good understanding of the application and match and merge rules is necessary. • Knowing “common errors” with match and merge rules helps a lot! (knowing the weak-points).

30. Conclusion • In ER process, negative rules capture “sanity checks”. • ER process often requires human guidance to handle real-world data and unexpected situations. • GNR algorithm represents generic way to solve ER-N. • ENR algorithm makes GNR algorithm less costly. • Choice of negative rules is very important.

31. References • Paper: Generic entity resolution with negative rules: by Steven Euijong Whang, Omar Benjelloun, Hector Garcia-Molina (The VLDB Journal (2009) 18:1261–1277 DOI 10.1007/s00778-009-0136-3) • http://infoblog.stanford.edu/2008/10/generic-entity-resolution-with-negative.html • http://infolab.stanford.edu/serf/

32. Thank you …