An Empirical Study of the Performance of Preprocessing and Look-ahead Techniques

1. July 30, 2003 An Empirical Study of the Performance of Preprocessing and Look-ahead Techniques for Solving Finite Constraint Satisfaction Problems Zheying Jane Yang Constraint Systems Laboratory Department of Computer Science and Engineering University of Nebraska-Lincoln Under the supervision of Prof. B.Y. Choueiry

2. Outline • Motivation & results • Background • Experimental design and empirical study • Results and analysis • Conclusions & relation to previous work • Summary of contributions • Future work

3. Search w/ look-ahead CSP Preprocessing Solution(s) SmallerCSP 1. Motivation • CSP used to model NP-complete decision problems • Search (exponential) is necessary, improved with • Preprocessing algorithms: remove inconsistent combinations prior to search • Look-ahead algorithms: remove inconsistencies during search

4. Algorithms studied & goal • Several competing algorithms • Preprocessing: removes inconsistencies prior to search • Arc-consistency: AC3, AC2001 • Neighborhood inverse consistency (NIC), requires search • Look-ahead: filters search space during search • Forward checking (FC) • Maintaining arc-consistency (MAC) • Controversies about their relative performance exist Our goal is to characterize empirically the relative performance of combinations of preprocessing and look-ahead schemes as a function of the CSP’s constraint probability & tightness

5. Issue Claims AC2001 is better Bessière & Régin 2001 AC3 vs AC2001 Preprocessing AC vs NIC NIC is better Freuder & Elfe 1996 Haralick & Elliott 1980 FC is better Nadel 1989 Sabin & Freuder 1994 MAC is much better MAC vs FC Look-ahead Bessière & Régin 1996 Grant 1997 Gent & Prosser 2000 No winner performs extremely well on all types of CSPs MAC is the winner in large spase CSPs FC is the winner in dense CSPs Our results:We specify when the above claims hold or not Current beliefs & results

6. 2. Background

7. X {a,b} {(b,c), (b,d)} {(b,e), (a,f)} Z Y {c,d} {e,f} {(c,e), (d,f)} What is a CSP? • A CSP problem is defined as a triple P=(V,D,C) • The goal is to find One solution All solutions

8. Solving CSPs with systematic BT search • Backtrack search is sound and complete, but exponential • expands a partial solution • explores every combination systematically • Improve performance of search • Preprocessing (constraint filtering/propagation) • Look-ahead (constraint filtering during search) • Backtracking: chronological/intelligent • Variable & value ordering: static/dynamic, heuristic • etc.

9. Solving a CSP Look-ahead Preprocessing • We study the performance of • Preprocessing • Hybrid search = preprocessing x look-ahead algorithms Following an assignment, removes values from the domains of the future variables that are inconsistent with the current assignment. Enforces different levels of local consistency by deleting inconsistent values from variable domains Algorithms: AC3, AC2001, NIC Algorithms: FC & MAC

10. {B} V1 V2 V3 {R} {G} V1 V1  {R, G,B} {R, G,B}  V2 V3 V2 V3 {R, G} {G} {R, G} {G} V1 {R, B} V2 V3 {R, G} {G} V1 {R, B}  V2  V3 {R} {G} Preprocessing a graph coloring problem

11. Start V1 V2 V3 V1 {R} V2 V3 {R,G} {G} Solving CSP using forward checking (FC)

12. Start V1 V2 V3 V1 {R} V2 V3 Domain wiped-out backtrack {G} {G} Solving CSP using forward checking (FC)

13. Start V1 V2 V3 {G} V1 V2 V3 Domain wiped-out backtrack {R, G} {G} Solving CSP using forward checking (FC)

14. Start V1 V2 V3 V1 {B} V2 V3 {R, G} {G} Solving CSP using forward checking (FC)

15. Solving CSP using forward checking (FC) Start V1 V2 V3 V1 {B} V2 V3 {G} {R}

16. Start V1 V2 V3 Solution! V1 {G} V2 V3 {R} {G} Solving CSP using forward checking (FC)

17. 3. Experimental design & empirical study

18. Algorithms tested Preprocessing  Look-ahead AC3 AC2001 FC MAC-AC3 MAC-AC2001 FC MAC-AC3 MAC-AC2001   NIC • Preprocessing: 5 algorithms • Look-ahead: 3 algorithms • Hybrid search: 7 combinations

19. Working assumptions • Considered only binary constraints • Restricted to finding first solution • Restricted to chronological backtracking • Use least domain (LD) as variable ordering heuristic • Variable ordering done dynamically • No value ordering heuristic, too costly in general

20. Problems tested • Random CSPs • We need many instances with similar characteristics in order to analyze the performance of search • Real-world problems cannot be controlled by explicit parameters to enable statistical analysis of the performance • Generated instances • connected graphs • instances guaranteed solvable

21. C Cmax Control parameters < n, d, C, t > • Number of variables n • We choose n = 50 • Domain size d (uniform) • We choose d = 10. Thus, problem size is 1050, relatively large • Constraint tightness t = (uniform) • We vary t = [0.1, 0.2, …, 0.9] and t = [0.05, 0.1, …, 0.95] • Number of constraints C determines constraint probability p= , Cmax= n(n-1)/2 • We vary C = [20, 490] corresponding to p = [0.024, 0.4] • We report C = 30, 50, 80, 120, 130, 245, 340, 490 Disallowedtuples Allpossibletuples

22. Comparisons • Evaluation criteria • Filtering effectiveness measures reduction of CSP size • Number of constraint checks #CC measures filtering effort • Number of nodes visited #NV measures backtracking effort • CPU time [ms] measures overall performance Since constraints are defined in extension, CPU time reflects #CC • Preprocessing: Filtering effectiveness, #CC, CPU time • Hybrid search: #CC, #NV, CPU time

23. Sampling and code characteristics • Evaluated performance by running each algorithm • on 30 CSP instances of equal characteristics, • calculating average & standard deviation values for each evaluation criterion • Generated approximately 6,000 CSP instances • Implemented in JAVA: • 21 JAVA classes • 4,000 lines of code • Experiments carried out on PrairieFire.unl.edu

24. 4. Results and analysis Part I: Preprocessing Part II: Hybrids

25. Preprocessing: filtering effectiveness NIC reduces search space at a large amount

26. Preprocessing results » Better than,  comparable

27. Preprocessing: #CC, sparse CSPs CC (AC3) > CC (AC2001) CC (NICx) >> CC (AC3, AC2001)

28. Preprocessing: #CC, denser CSPs When p=0.2, NICx are all costly NIC should never be combined with MAC

29. Preprocessing results » Better than,  comparable

30. Preprocessing: CPU time When p > 0.2, NIC-x is too costly

31. Preprocessing results » Better than,  comparable

32. Preprocessing: summary • AC2001 is not significantly better than AC3, and is not worth the extra data structures In general, we disagree with Bessière and Régin • NIC-x: powerful filtering, but too costly. Use it on large problems when checking constraints is cheap

33. Hybrids: #CC, p = 5%, 11% When p < 0.05: ACx is not effective,ACx-FC are costly, NIC-FC-FC is OK, NIC-MAC-x is best When p > 0.10: avoid MAC, stick with FC

34. Hybrids results » Better than,  comparable #CC & CPU time

35. Hybrids: #CC, p= 15%, 28% As p increases: MAC deteriorates, NIC becomes expensive, use ACx-FC

36. Hybrids results » Better than,  comparable #CC & CPU time

37. Hybrids: #NV At phase transition, NIC and MAC • do powerful filtering • but influence of MAC is stronger

38. Hybrids: summary » Better than,  comparable #CC & CPU time

39. 5. Conclusions • Preprocessing • AC3 vs. AC2001: AC2001 is not significantly better than AC3, and is not worth the extra data structures In general, we disagree with Bessière and Régin • NIC-x: powerful filtering, but too costly Use it on large problems when checking constraints is cheap • Look-ahead • MAC vs. FC: performance depends on constraint probability and tightness. MAC only wins in low p and high t In general we disagree with results of Sabin, Freuder, Bessière & Régin

40. Relation to previous work (I) • NIC is better than ACFreuder & Elfe 1996 The instances tested were all with low probability (p < 0.05). In this region, AC is ineffective. • MAC is better than FCSabin & Freuder 1994 They tested CSPs with low probability (p = 0.018-0.09), and relatively high constraint-tightness (t = 0.15 – 0.675). In our study, MAC is effective in this region, but not outside it. • MAC is better than FCBessière & Régin 1996 The instances tested were also in the region of low probability (p = 0.017, 0.024, 0.074, 0.08, 0.1, 0.12, 0.15), except instance#1 and instance#2, with relative high probability (p = 0.3, and 0.84). But here they test only 2 instances

41. Relation to previous work (II) • Gent & Prosser 2000 questioned validity of previous results on MAC. They concluded that: • in large, sparse CSPs with tight constraints, MAC is winner • in dense CSPs with loose constraints, FC is winner. • Grant 1997 showed that FC is winner on small CSPs with all range of probabilities • All concluded that: “A champion algorithm which perform extremely well on all types of problems does not exist.” • Our characterizations are more thorough and precise.

42. 6. Summary of contributions • Random generator that guarantees solvability • Empirical evaluation of the performance of 7 combinations of preprocessing and look-ahead • Uncovered (restricted) validity conditions of previously reported results • Summarized best working conditions for preprocessing and look-ahead algorithms • Developed a Java library with 7 hybrid algorithms

43. 7. Directions for future work • Compare to other, less common filtering algorithms, e.g. SRPC, PC, SAC, Max-RPC Debryune & Bessière 2001 • Combining these preprocessing algorithms with intelligent backtrack search algorithms • Validate results on larger CSPs, real-world applications, non-binary • Test and analyze the effect of the topology of constraint networks on the performance of search

44. Acknowledgments Dr. Berthe Y. Choueiry (advisor) Dr. Sebastian Elbaum Dr. Peter Revesz Ms. Catherine L. Anderson Mr. Daniel Buettner Ms. Deborah Derrick (proof reading) Mr. Eric Moss Mr. Lin Xu Ms. Yaling Zheng Mr. Hui Zou

45. THANK YOU!