Muhammed J. Al-Muhammed David W. Embley Brigham Young University

1. Resolving Underconstrained and Overconstrained Systems of Conjunctive Constraints for Service Requests Muhammed J. Al-Muhammed David W. Embley Brigham Young University Sponsored in part by NSF (#0083127 and #0414644)

2. The Problem: Underconstrained “I want a dodge a 2000 or newer. The Mileage should be less than 80,000 and the price should not exceed $15,000” www.cars.com, November 2005

3. The Problem: Overconstrained “I want a dodge a 2000 or newer. The Mileage should be less than 80,000 and the price should not exceed $4,000.” www.cars.com, November 2005

4. Key Observations • Some (near) solutions are better than others • People specify constraints on some concepts in a domain more often than other concepts

5. Fundamental Concepts: reward, penalty, and expectation • A reward is a positive or zero real number given to a solution for satisfying a constraint • A penalty is a negative real number given to a near solution for violating a constraint • An expectation for a conceptis the probability that people will specify constraint for the concept

6. Fundamental Concepts:Pareto Optimality • Based on dominance relations • The reward for S1 is as high as the reward for S2 • For at least one reward S1 has a higher reward • Dominated solutions are not Pareto optimal

7. Too Many Solutions:Reward-Based Ordering • Calculate rewards and combine them • Order solutions, highest combined reward first • Select the top-m Pareto optimal solutions • Discard non-Pareto optimal solutions from the reward ordering • Return the top-m for consideration

8. S1 better The same S1 better Example “I want a dodgea 2000 or newer. The mileage should be less than 80,000 and the price should not be more than $15,000.” Solution Make Model Price Year Mileage --------------------------------------------------------------- S1 Dodge Stratus 13,999 2005 15,775 S2 Dodge Stratus 11,998 2004 23,404 S3 Dodge Stratus 14,200 2005 16,008 S4 Dodge Stratus 14,557 2005 16,954 S5 Dodge Stratus 10,590 2003 38,608

9. Too Many Solutions: Expectation-Based Constraint Elicitation • Associate expectations with domain concepts • Order the concepts in a domain based on their expectations • Most expected first • Example: Make > Price > Model > … • Elicit additional constraints over unconstrained concepts • Most expected first • If no preferred make provided, ask for Make; if no price, ask for Price; …

10. No Solution: Penalty-Based Ordering • Calculate penalties and combine them • Order close solutions, lowest combined penalty first • Select the top-m Pareto optimal near solutions • Discard dominated near solutions from penalty ordering • Return the top-m near solutions for consideration

11. No Solution:Expectation-Based Constraint Relaxation • Select the near solutions violating fewer constraints than a threshold • Compute the relaxation cost: rsi = kekCk(si). • Suggest constraints of the near solution with the least rsi for relaxation penalty expectation

12. Can this constraint “1:00 PM or after” be relaxed to “12:40 PM” Can this constraint “the 20th” be relaxed to “the 19th” Example “I want to see a dermatologist on the 20th, 1:00 PM or after. The dermatologist should be within 5 miles from my home and must accept myIHCinsurance.” 12:40 PM the 19th

13. Performance Analysis • Tested on appointment and car purchase domains • 16 human subjects • The best-5 near solutions from 19 appointments • The best-5 solutions from 32 cars • Compare human selection with system selection with respect to the best-5

14. Performance Analysis Human selection versus system selection: appointment

15. Performance Analysis Human selection versus system selection: car purchase

16. Performance Analysis • Inter-observer agreement test • Results • kappa 0.74 (appointment) • kappa 0.67 (car purchase) • “Substantial” agreement based on kappa values