AAAI00 Austin, Texas

1. Generating Satisfiable Problem InstancesDimitris AchlioptasMicrosoftCarla P. Gomes Cornell University Henry KautzUniversity of WashingtonBart SelmanCornell University • AAAI00 • Austin, Texas

2. Introduction • An important factor in the development of search methods is the availability of good benchmarks. • Sources for benchmarks: • Real world instances • hard to find • too specific • Random generators • easier to control (size/hardness)

3. Random Generators of Instances • Understanding threshhold phenomena lets us tune the hardness of problem instances: • At low ratios of constraints - • most satisfiable, easy to find assignments; • At high ratios of constraints - • most unsatisfiableeasy to show inconsistency; • At the phase transition between these two regions • roughly half of the instances are satisfiable and we find a concentration of computationally hard instances.

4. Limitation of Random Generators • PROBLEM: evaluating incompletelocal search algorithms • Filtering out Unsat Instances - use a complete method and throw away unsat instances. Problem: want to test on instances too large for any complete method! • “Forced” Formulas Problem: the resulting instances are easy – have many satisfying assignments

5. Outline • I Generation of only satisfiable instances • II New phase transition in the space of satisfiable instances • III Connection between hardness of satisfiable instances and new phase transition • IV Conclusions

6. Generation of only satisfiable instances

7. Quasigroup or Latin Squares Given an N X N matrix, and given N colors, color the matrix in such a way that: -all cells are colored; - each color occurs exactly once in each row; - each color occurs exactly once in each column; Quasigroup or Latin Square

8. Quasigroup Completion Problem (QCP) Given a partial assignment of colors (10 colors in this case), can the partial quasigroup (latin square) be completed so we obtain a full quasigroup? Example: 32% preassignment

9. QCP: A Framework for Studying Search • NP-Complete. • Random instances have structure not found in random k-SAT Closer to “real world” problems! • Can control hardness via % preassignment • BUT problem of creating large, guaranteed satisfiable instances remains… (Anderson 85, Colbourn 83, 84, Denes & Keedwell 94, Fujita et al. 93, Gent et al. 99, Gomes & Selman 97, Gomes et al. 98, Shaw et al. 98, Walsh 99 )

10. 32% holes Quasigroup with Holes(QWH) • Given a full quasigroup, “punch” holes into it Difficulty: how to generate the full quasigroup, uniformly. Question: does this give challenging instances?

11. Markov Chain Monte Carlo (MCMM) • We use a Markov chain Monte Carlo method (MCMM) whose stationary (egodic) distribution is uniform over the space of NxN quasigroups (Jacobson and Matthews 96). • Start with arbitrary Latin Square • Random walk on a sequence of Squares obtained via local modifications

12. Generation of Quasigroup with Holes (QWH) • Use MCMM to generate solved Latin Square • Punch holes - i.e.,uncolor a fraction of the entries • The resulting instances are guaranteed satisfiable • QWH is NP-Hard Is there % holes where instances truly hard on average?

13. Complete (Satz) Search Order 30, 33, 36 Easy-Hard-Easy Pattern in Backtracking Search QWH peaks near 32% (QCP peaks near 42%) Computational Cost % holes

14. Local (Walksat) Search Order 30, 33, 36 Easy-Hard-Easy Pattern in Local Search Computational Cost % holes First solid statistics for overconstrainted area!

15. Phase Transition in QWH? • QWH - all instances are satisfiable - does it still make sense to talk about a phase transition? • The standard phase transition corresponds to the area with 50% SAT/UNSAT instances • Here all instances SAT Does some other property of the wffs show an abrupt change around “hard” region?

16. Backbone Preassigned cells Backbone Backbone size = 2 Backbone is the shared structure of all solutions to a given instance (not counting preassigned cells) Number sols = 4

17. Phase Transition in the Backbone • We have observed a transition in the size of backbone • Many holes – backbone close to 0% • Fewer holes – backbone close to 100% • Abrupt transition – coincides with hardest instances!

18. Sudden phase Transition in Backbone and it coincides with the hardest area New Phase Transition in Backbone % Backbone % of Backbone Computational cost % holes

19. Why correlation between backbone and problem hardness? • Intuitions: Local Search • Near 0% Backbone = many solutions = easy to find by chance • Near 100% Backbone = solutions tightly clustered = all the constraints “vote” in same direction • 50% Backbone = solutions in different clusters = different clauses push search toward different clusters (Current work – verify intuitions!)

20. Why correlation between backbone and problem hardness? • Intuitions:Backtracking search • Bad assignments to backbone variables near root of search tree cause the algorithm to deteriorate • For the algorithm to have a significant chance of making bad choices, a non-negligible fraction of variables must appear in the backbone

21. Reparameterization of Backbone Backbone for different orders (30 - 57) % of Backbone

22. ReparameterizationComputational Cost Computational Cost different orders (30, 33, 36) % of Backbone Local Search (normalized) Local Search (normalized & reparameterized)

23. Summary • QWH is a problem generator for satisfiable instances (only): • Easy to tune hardness • Exhibits more realistic structure • Well-suited for the study of incomplete search methods (as well as complete) • Confirmation of easy-hard-easy pattern in computational cost for local search • New kind of phase transition in backbone • Reparameterization • GOAL – new insights into practical complexity of problem solving

24. QWH generator, demos, available soon (< one month):www.cs.cornell.edu/gomeswww.cs.washington.edu/home/kautzSATLIBCSPLIB