An Interactive System for Hiring & Managing Graduate Teaching Assistants

1. An Interactive System for Hiring & Managing Graduate Teaching Assistants Ryan Lim Venkata Praveen Guddeti Berthe Y. Choueiry Constraint Systems Laboratory University of Nebraska-Lincoln

2. Outline • Task & Motivation • System Architecture & Interfaces • Scientific aspects • Problem Modeling • Problem Solving • Comparing & Characterizing Solvers • Motivation revisited & Conclusions

3. Task • Hiring & managing GTAs as instructors + graders • Given • A set of courses • A set of graduate teaching assistants • A set of constraints that specify allowable assignments • Find a consistent & satisfactory assignment • Consistent: assignment breaks no (hard) constraints • Satisfactory: assignment maximizes • number of courses covered • happiness of the GTAs • Often, number of hired GTAs is insufficient

4. Motivation • Context • “Most difficult duty of a department chair” [Reichenbach, 2000] • Assignments done manually, countless reviews, persistent inconsistencies • Unhappy instructors, unhappy GTAs, unhappy students • Observation • Computers are good at maintaining consistency • Humans are good at balancing tradeoffs • Our solution • An online, constraint-based system • With interactive & automated search mechanisms

5. Outline • Task & Motivation • System Architecture & Interfaces • Scientific aspects • Problem Modeling • Problem Solving • Comparing & Characterizing Solvers • Motivation revisited & Conclusions

6. Password Protected Access for GTAs http://cse.unl.edu/~gta Password Protected Access for Manager http://cse.unl.edu/~gta • Web-interface for applicants • Web-interface for manager • View / edit GTA records • Setup classes • Specify constraints • Enforce pre-assignments Visualization widgets Local DB Other structured, semi-structured, or unstructured DBs Interactive Search Automated Search Heuristic BT Stochastic LS Multi-agent Search Randomized BT • A local relational database • Graphical selective queries Cooperative, hybrid Search Strategies • Drivers for • Interactive assignments • Automated search algorithms In progress System Architecture

7. GTA interface: Preference Specification

8. Manager interface: TA Hiring & Load

9. Outline • Task & Motivation • System Architecture & Interfaces • Scientific aspects • Problem Modeling • Problem Solving • Comparing & Characterizing Solvers • Motivation revisited & Conclusions

10. Constraint-based Model • Variables • Grading, conducting lectures, labs & recitations • Values • Hired GTAs (+ preference for each value in domain) • Constraints • Unary: ITA certification, enrollment, time conflict, non-zero preferences, etc. • Binary (Mutex): overlapping courses • Non-binary: same-TA, capacity, confinement • Objective • longest partial and consistent solution (primary criterion) • while maximizing GTAs’ preferences (secondary criterion)

11. Outline • Task & Motivation • System Architecture & Interfaces • Scientific aspects • Problem Modeling • Problem Solving • Comparing & Characterizing Solvers • Motivation revisited & Conclusions

12. Problem Solving • Interactive decision making • Seamlessly switching between perspectives • Propagates decisions (MAC) • Automated search algorithms • Heuristic backtrack search (BT) • Stochastic local search (LS) • Multi-agent search (ERA) • Randomized backtrack search (RDGR) • Future: Auction-based, GA, MIP, LD-search, etc. • On-going: Cooperative/hybrid strategies

13. Manager interface: Interactive Selection

14. Dual perspective Task-centered view Resource-centered view

15. Shallowest level reached by BT after … Number of variables: 69 24 hr: 51 (26%) 1 min: 55 (20%) Max depth: 57 Depth of the tree: 69 Heuristic BT Search • Since we don’t know, a priori, whether instance is solvable, tight, or over-constrained • Modified basic backtrack mechanism to deal with this situation • We designed & tested various ordering heuristics: • Dynamic LD was consistently best • Branching factor relatively huge (30) • Causes thrashing, backtrack never reaches early variables

16. Stochastic Local Search • Hill-climbing with min-conflict heuristic • Constraint propagation: • To handle non-binary constraints (e.g., high-arity capacity constraints) • Greedy: • Consistent assignments are not undone • Random walk to avoid local maxima • Random restarts to recover from local maxima

17. Multi-Agent Search (ERA)[Liu et al. 02] • “Extremely” decentralized local search • Agents (variables) seek to occupy best positions (values) • Environment records constraint violation in each position of an agent given positions of other agents • Agents move, egoistically, between positions according to reactive Rules • Decisions are local • An agent can always kick other agents from a favorite position even when value of ‘global objective function’ is not improved • ERA appears immune to local optima • Lack of centralized control • Agents continue to kick each other • Deadlock appears in over-constrained problems

18. Randomized BT Search • Random variable/value selection allows BT to visit a wider area of the search space [Gomes et al. 98] • Restarts to overcome thrashing • Walsh proposed RGR [Walsh 99] • Our strategy, RDGR, improves RGR with dynamic choice of cutoff values for the restart strategy [Guddeti & Choueiry 04]

19. Optimizing solutions • Primary criterion: solution length • BT, LS, ERA, RGR, RDGR • Secondary criterion: preference values • BT, LS, RGR, RDGR • Criterion: • Average preference • Geometric mean • Maximum minimal preference

20. More Solvers… • Interactive decision making • Automated search algorithms • BT, LS, ERA, RGR, RDGR. • Future: Auction-based, GA, MIP, LD-search, etc. • On-going: Cooperative / hybrid strategies

21. Outline • Task & Motivation • System Architecture & Interfaces • Scientific aspects • Problem Modeling • Problem Solving • Comparing & Characterizing Solvers • Motivation revisited & Conclusions

22. Comparing Solvers • Using the same CSP encoding, students implements solvers separately and competed for best results • Experience lead to the identification of behavioral criteria and regimes that characterize the performance of the various solvers in the context of GTAP

23. Characterizing Solvers • General criteria • Stability, solution length, vulnerability to local optima, deadlock, thrashing, etc. • Tight but solvable instances • ERA  RDGR  RGR  BT  LS • Over-constrained instances • RDGR  RGR  BT  ERA  LS

24. Outline • Task & Motivation • System Architecture & Interfaces • Scientific aspects • Problem Modeling • Problem Solving • Comparing & Characterization Solvers • Motivation revisited & Conclusions

25. Motivation (revisited) • “Most difficult duty of a department chair” • Keeps the manager in the decision loop while removing the need for tedious and error-prone manual assignments • Helps producing quick (3 weeks down to 2 days) and satisfactory (stable) assignments • Initially, assignments were manually done on paper • Now, on-line data acquisition process • Enabled department to streamline & standardize GTA selection, hiring, and assignment • Overworked staff, unhappy GTAs • Overjoyed staff (relieved from handling application forms and massive paperwork) • Enthusiastic anonymous online reviews from applicants

26. History & Evaluation • System entirely built by students • Modeling started in January 2001 • Prototype system used since August 2001 • Features improved and added as needs arised • No formal longitudinal study • Since August 2003: 109 GTA users, 23 feedback responses • Since April 2004, CSE implemented on-line GTA evaluation by faculty on top of GTAAP

27. GTA Online Feedback Navigation Data entry 23 responses

28. Conclusions • Integrated interactive & automated problem-solving strategies • Reduced the burden of the manager • Lead to quick development of ‘stable’ solutions • Our efforts • Helped the department • Trained students in CP techniques • Paved new avenues for research • Cooperative, hybrid search • Visualization of solution space

29. <<< end of presentation I welcome your questions Please contact me for a live demo

31. Manager interface: Course Load Specification

32. Manager interface: Preassignment

33. Manager interface: Constraint Specification