Analysis of usability of scheduling algorithms: a case study in the Netherlands Railways

1. Analysis of usability of scheduling algorithms: a case study in the Netherlands Railways R.J. Jorna, W. van Wezel & J. BosFaculty of Economics and Business, University of Groningen (Groningen, The Netherlands) Contact: r.j.j.m.jorna@rug.nl Third Rail Human Factors Conference; Lille, France, March 3-5, 2009

2. Structure • Introduction • Netherlands Railways and their planning • Background & Problem statement: effects of planning support • Research: • What does “inadequate use” of algorithms mean? • Results of experiments with 8 planners • Comparison planners and OR-specialist • Algorithms and the function-allocation theory • Conclusions, statements and further research

3. The Netherlands Railways and Planning • Netherlands Railways (NS): 2700 railroad carriages; per day 5000 train services; 384 stations. • NS overall: about 350 planners; • 4 major kinds of planning: • A: timetables; • B: planning rolling stock and rolling staff; • C: local and central planning (stock and staff); • D: year plan (long term) & day plan (short term) • Sequence: timetable, rolling material, rolling staff and then shifts for individual persons.

4. Background and problem statement • In staff planning shift schedules need to be changed frequently (see Figure 1). • Here, study of 8 planners of Day Planning (Central) (re)schedule shifts of train drivers/ticket collectors. • Planners use DSS CREWS (see Figure 2), with a GUI to create plans by dragging and dropping. • Recently, an algorithm to automatically create plans is included. Planners can influence algorithm by setting parameters. • Problem: Use of algorithm did not result in expected increase in performance. What is going on?

5. Figure 1 Example of change within shifts

6. Screen shot of CREWS Figure 2

7. Theoretical assumptions: general • Task performance with DSS is a function of • Planning problem complexity, • Quality of automation (software support level), • Mental load of planners, • Many other aspects (trust, cost, cooperation, etc.) • Use of algorithmic support can be studied in quasi-experiments with planners

8. Factors influencing the adequate use of planning support Cognitive workloadSituation awarenessComplacency Skill degradation Quality of automation Risk of automation failure Trust Cost of incorrect decisions How can we apply this to planning / scheduling?

9. Theoretical assumptions: model • Relation between independent variables and performance Two conditions: manual and algorithm support Two conditions: simple and complex problem

10. Experiments with planners (1) • Procedure • Planner reads assignments, solves problem with system; • For Mental Load: planner uses NASA TLX questionnaire; • During experiment, number of activities of planner are counted: a) use of searching functionality in system; b) # backtrackings (withdrawn reasoning steps); # decisions resulting in constraint violations

11. Experiments with planners: results simple problem (2) +: algorithm better For simple problem: algorithm better than manual; for complex the reverse Question: WHY?

12. Experiments with planners: results mental load and complex problem (3) • No difference mental load for simple problem (N=8) nor for complex problem (N=2) • Experiment with complex problem took more than 8 hours (only two planners were analyzed)

13. Experiments with planners: conclusions (4) Some conclusions: • For algorithms, planners are concerned with usability: • does planner understand outcome of algorithm? • can schedule be explained to the staff that will execute it? • can planner change schedule easily, etc.) (all of the above are difficult to formalize) • If algorithm is not aligned to planner, no increase of performance • In using algorithms, significant individual planner differences. • Individual planner differences concerning: mental load, time to solve problem, quality of outcome. • Planners spend much time on configuring parameters and interpreting the outcome (from protocols).

14. Experiment comparing planners, algorithm and developer (5) Planners can not plan as good as OR specialists?

15. What is wrong with planners? Planners do not use their algorithms correctly and should be trained or Algorithms are not developed to be usable by planners (The algorithm was designed by winners of 2008 Informs “Franz Edelman Award for Achievement in OR and the Management Sciences”. Should they, the developers, become planners?) Is planning/scheduling in railways a matter of control of dynamic situations and should support be adjusted to this kind of task?

16. 12-02-2008 | 16 FIX ERRORS CALCULATE DURING THE NIGHT

17. Basic assumptions concerning planners in algorithmic support • Interaction with algorithms implies knowledge of the model being used, e.g.: • Specify parameters; specify weights on goal functions • Steer the backtracking process of the algorithm • Fix the solution manually • How does interaction between parameters lead to constraint violations and goal realization? What goal? • Planners do not have this knowledge !! • Planners will never have this knowledge !! • We propose that Interaction should be determined a priori, instead of a posteriori

18. The perspective of the control of dynamic situations “Dynamic situations are situations which are only partly controlled by the human. He/she then has to keep the situation between acceptable limits while managing high risk and/or time pressure” Examples: Air traffic control, nuclear power plant, aircraft operation, flexible manufacturing system In the field of Cognitive Ergonomics, Function Allocation is successfully used to design automation

19. Conclusions / statements The fundamental problems concerning planning support did not change in 30 years Practice is ahead of academia We don’t fully understand ‘old’ planning, yet businesses started ‘new’ planning Organization theory (various organizational forms) related to planning hinders adequate support Algorithms do not pay attention to human problem solving

20. Conclusions / further research Balance requirements of tasks, algorithmic possibilities, and human limitations/creativity Include the user in the development process from the beginning, but: Not necessarily take the limitations of current humans for granted! Training or even partly replacing human planners is an option as well. However, many goals of planning/scheduling are often confusing / difficult to formalize!

22. Interactive Scheduling Systems;gap between theory and practice Combinatorial problems are hard to tackle Problems in practice change Interactive systems are rare Operations managers lack understanding of scheduling systems Cost/benefit ratio hard to prove Human planners don’t have a good grasp of the implications of their decisions Schedulers are buffered from outside pressure

23. The planner and his tasks • Macro task division • Organizational setting • Who does what? Order acceptance, stock planning, scheduling, rescheduling, etc. • Micro task division • Collecting information • Counting • Checking errors • Evaluating • Problem solving