1 / 49

Ubiquity Generator (UG) Framework: Introduction

Ubiquity Generator (UG) Framework: Introduction. Yuji Shinano Zuse Institute Berlin. 14.01.2019. Outline. What is Ubiquity Generator (UG) Framework Brief history of the development C omputational behavior of state-of-the-art Branch-and-bound based solvers

red
Download Presentation

Ubiquity Generator (UG) Framework: Introduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ubiquity Generator (UG) Framework: Introduction Yuji Shinano Zuse Institute Berlin Firste intenational UG workshop 2019, Berlin, Germany 14.01.2019

  2. Outline • What is Ubiquity Generator (UG) Framework • Brief history of the development • Computational behavior of state-of-the-art Branch-and-bound based solvers • Example: Mixed Integer Linear Program (MILP) solver • Anomalies of Parallel Branch-and-Bound • UG and how UG does parallel tree search • Ramp-up mechanism • Dynamic load balancing • Checkpoint and restart mechanism • Deterministic mode implementation • What is ug[SCIP, *] libraries • UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

  3. Outline • What is Ubiquity Generator (UG) Framework • Brief history of the development • Computational behavior of state-of-the-art Branch-and-bound based solvers • Example: Mixed Integer Linear Program (MILP) solver • Anomalies of Parallel Branch-and-Bound • UG and how UG does parallel tree search • Ramp-up mechanism • Dynamic load balancing • Checkpoint and restart mechanism • Deterministic mode implementation • What is ug[SCIP, *] libraries • UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

  4. What is Ubiquity Generator Framework • A generic framework to parallelize branch-and-bound based solvers • Exploits powerful performance of state-of-the-art base solvers • The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software by Herb Sutter, 2005 • State-of-the-art (LP based) branch-and-bound solvers (base solvers) • Mixed Integer Program solvers: SCIP, Xpress, Gurobi, CPLEX • Steiner Tree Problem solver: SCIP-Jack • Traveling Salesman Problem solver: Concorde: Algorithmic improvements >> Parallelization Purpose • Development of massively parallel branch-and-bound based solvers • can solve instances that cannot be solved by state-of-the-art solvers • keeps catching up performance improvements state-of-the-art solvers Parallel solving with mathematically supercharged algorithms Firste intenational UG workshop 2019, Berlin, Germany

  5. Brief history of the UG framework development • ~mid 90s • A parallel branch-and-bound was realized for a specific problem and on a specific parallel computer • mid 90s~ • Standardization of the message passing interface • PVM(Parallel Virtual Machine), MPI(Message Passing Interface) • Parallel branch-and-bound software frameworks • PPBB-Lib, BOB, ZRAM, PUBB, …, COIN-OR projects(open source) • B&B tree data structure is managed by the framework • B&B solvers need to develop with the frameworkfrom scratch Firste intenational UG workshop 2019, Berlin, Germany

  6. Brief history of the UG framework development • 2003~: CPLEX solver parallelization on distributed memory computing environments with PUBB2 • Performance was good in terms of solving hard instances • Too tricky design for the parallelization • 2007: ParaLEX: a specialized implementation for CPLEX Firste intenational UG workshop 2019, Berlin, Germany

  7. Brief history of the UG framework development • 2008: Redesigned ParaLEX • 2009: Generalize ParaLEXto a software framework and use it to develop ParaSCIP • CPLEX parallelization was planning • Framework name: PUBB-ZIB • PUBB: Parallelization Utility for Branch-and-Bound Need to corporate with state-of-the-art solver developers! Rename Ubiquity Generator (UG) Framework Firste intenational UG workshop 2019, Berlin, Germany

  8. Outline • What is Ubiquity Generator (UG) Framework • Brief history of the development • Computational behavior of state-of-the-art Branch-and-bound based solvers • Example: Mixed Integer Linear Program (MILP) solver • Anomalies of Parallel Branch-and-Bound • UG and how UG does parallel tree search • Ramp-up mechanism • Dynamic load balancing • Checkpoint and restart mechanism • Deterministic mode implementation • What is ug[SCIP, *] libraries • UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

  9. Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example Mixed Integer Linear Programs (MILP) • Preprocessing (presolving) • Reformulate the original problem instance Original instance Presolved instance A’: Presolved A: Original Firste intenational UG workshop 2019, Berlin, Germany

  10. Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example • Root node computation takes much longer time • Root node: all procedures before the first branch • preprocessing, adding cutting planes, … root comp. time = 5.4 (sec), (comp. time)/node=28.88/770≒0.375(sec) Firste intenational UG workshop 2019, Berlin, Germany

  11. Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example • B&B tree can be highly unbalanced Firste intenational UG workshop 2019, Berlin, Germany

  12. Computational behavior of state-of-the-art Branch-and-bound based solvers: An MILP solver as an example • Performance variability • Changes in performance measures for the same problem that are caused by seemingly performance-neutral changes in the environment or the input format The best ten permutations need less than 3 min, but nearly 20% hit the time limit of 10 h. Thorsten Koch, Tobias Achterberg, Erling Andersen, Oliver Bastert, Timo Berthold, Robert E. Bixby, Emilie Danna, Gerald Gamrath, Ambros M. Gleixner, Stefan Heinz, Andrea Lodi, Hans Mittelmann, Ted Ralphs, Domenico Salvagnin, Daniel E. Steffy, Kati Wolter, MIPLIB2010, Mathematical Programming Computation, 3:2 (2011) 103-163 Firste intenational UG workshop 2019, Berlin, Germany

  13. Solving techniques involved in SCIP Firste intenational UG workshop 2019, Berlin, Germany

  14. SCIP (solving constraint integer programs) http://scip.zib.de - plugin structure - over 800,000 lines of C code Firste intenational UG workshop 2019, Berlin, Germany

  15. Outline • What is Ubiquity Generator (UG) Framework • Brief history of the development • Computational behavior of state-of-the-art Branch-and-bound based solvers • Example: Mixed Integer Linear Program (MILP) solver • Anomalies of Parallel Branch-and-Bound • UG and how UG does parallel tree search • Ramp-up mechanism • Dynamic load balancing • Checkpoint and restart mechanism • Deterministic mode implementation • What is ug[SCIP, *] libraries • UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

  16. 5 5 5 6 6 9 9 9 9 9 9 9 13 9 9 11 9 9 generated Pruned Incumbent processing Anomalies in parallel branch-and-bound • Speedup anomaly Sequential 15 9 9 10 Objective function value 5 Parallel Sequential 11 12 12 10 10 13 1 2 3 4 5 6 7 8 9 10 Time To simplify, assume that all nodes are solved by a unit of time Firste intenational UG workshop 2019, Berlin, Germany

  17. 5 5 5 6 9 9 9 9 generated Pruned Incumbent processing Anomalies in parallel branch-and-bound • Speedup anomaly faster more than 2 times Sequential Parallel: solve 2 nodes at the same time 15 6 9 10 Objective function value 9 9 5 Parallel Sequential 1 2 3 4 5 6 7 8 9 10 Time To simplify, assume that all nodes are solved by a unit of time Firste intenational UG workshop 2019, Berlin, Germany

  18. 5 5 8 8 9 9 10 generated Pruned Incumbent processing Anomalies in parallel branch-and-bound • Detrimental anomaly Sequential 15 9 8 10 Objective function value 9 9 10 5 Parallel Sequential 1 2 3 4 5 6 7 8 9 10 Time To simplify, assume that all nodes are solved by a unit of time Firste intenational UG workshop 2019, Berlin, Germany

  19. 5 5 6 6 6 9 9 9 9 9 9 10 10 10 10 9 10 10 11 11 14 14 14 15 15 15 12 12 12 12 13 13 13 13 generated Pruned Incumbent processing Anomalies in parallel branch-and-bound • Detrimental anomaly Sequential Parallel: solve 2 nodes at the same time Slow down in parallel 15 10 Objective function value 5 Parallel Sequential 1 2 3 4 5 6 7 8 9 10 Time To simplify, assume that all nodes are solved by a unit of time Firste intenational UG workshop 2019, Berlin, Germany

  20. Outline • What is Ubiquity Generator (UG) Framework • Brief history of the development • Computational behavior of state-of-the-art Branch-and-bound based solvers • Example: Mixed Integer Linear Program (MILP) solver • Anomalies of Parallel Branch-and-Bound • UG and how UG does parallel tree search • Ramp-up mechanism • Dynamic load balancing • Checkpoint and restart mechanism • Deterministic mode implementation • What is ug[SCIP, *] libraries • UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

  21. UG UbiquityGeneratorFramework - 1 • Existing (sequential) solvers can be used • Subproblem data need to be extracted from the solver by API • branching variables, branching constraints, local cuts, variable statistics, branching history, .. • Software Framework: A set of C++ base classes • Goal: To identifythe necessary APIs that are needed in base solvers to be parallelized in a waythatUGprovides • Naming of instantiated solver by UGug[base solver name, parallelization library name] Design policy: Simple, small and intuitive interfaces with enough flexibility Firste intenational UG workshop 2019, Berlin, Germany

  22. UG UbiquityGeneratorFramework - 2 • Abstractions of base solvers and parallelization libraries For Xpress For SCIP Subproblem Data scipInitiator scipSolver scipInstance scipDiffSubproblem scipSolution Parallelization framework for B&B (A load balancing mechanism, ramp-ups, checkpoint & restart) Solver Instance DiffSubproblem Solution Initiator ParaInitiator ParaSolver ParaInstance ParaSolution Communicator For MPI World For MPI Intercomm, Pthreads, GRID, P2P, Cloud computing etc. CommMpiWorld Firste intenational UG workshop 2019, Berlin, Germany

  23. Using API to control solving algorithms Using API to control solving algorithms Using API to control solving algorithms • Using MPI or C++11for communications • Using MPI or C++11 for communications Using MPI or C++11 for communications UG UbiquityGeneratorFramework - 3 UG framework LoadCorrdinator Loads are coordinated by a special process or thread Base solver I/O , presolve : Initiator Base solver Base solver Base solver (UG) Solver Parallel Solver Instantiation distributed memory ug[SCIP, MPI]: ParaSCIPug[Xpress, MPI]: ParaXpress shared memory ug[SCIP, C++11]: FiberSCIP ug[Xpress, C++11]: FiberXpress External parallelization Run on PC clusters and supercomputers Run on PC Firste intenational UG workshop 2019, Berlin, Germany

  24. HowUG initializes all solvers [Initialization] Original waiting: running: Base solver I/O , presolve LoadCoordinator A’: Presolved (sub-)problem A: Original (sub-)problem P presolved instance presolved instance presolved instance presolved instance Presolved instance is distributed Solver n Solver 3 Solver 4 Solver 2 Solver 1 AllSolvers keep the presolved instance Firste intenational UG workshop 2019, Berlin, Germany

  25. HowUGdoes ramp-up(start with a single tree search) [Normal Ramp-up] Original waiting: running: Base solver I/O , presolve LoadCoordinator presolved instance presolved instance presolved instance presolved instance Presolve again with added bound changes Solver n Solver 1 Solver 2 Solver 3 Solver 4 This procedure last until all solvers become busy Firste intenational UG workshop 2019, Berlin, Germany

  26. HowUGdoes ramp-up (start with a single tree search) [Normal Ramp-up] Original waiting: running: Base solver I/O , presolve LoadCoordinator presolved instance presolved instance presolved instance presolved instance Presolve again with added bound changes Solver n Solver 1 Solver 2 Solver 3 Solver 4 All transfer data need to be converted back for the presolved instance All feasible solutions need to be converted back for the original instance Firste intenational UG workshop 2019, Berlin, Germany

  27. HowUGdoes ramp-up (start with multiple trees search) [Ramp-up(Racing)] Original waiting: running: Base solver I/O , presolve LoadCoordinator Winner Solver n Solver 1 Solver 2 Solver 3 Solver 4 AllSolversstartsolvingimmediately,trying to generate different search trees ug[SCIP,*]: work with distributed domain propagation Firste intenational UG workshop 2019, Berlin, Germany

  28. HowUGdoes ramp-up (start with multiple trees search) [Ramp-up(Racing)] Original waiting: running: Base solver I/O , presolve LoadCoordinator Winner All nodes are not always collected ot LC Solver n Solver 1 Solver 2 Solver 3 Solver 4 When the racing terminated without enough open nodes, automatically the ramp-up is continued with normal ramp-up Firste intenational UG workshop 2019, Berlin, Germany

  29. Parameters to control racing termination • General parameters (red : default value) • NEvaluationSolversToStopRacing • -1 stops at all of the solvers, 0 stop at half of the solvers, n > 0 • RacingRampUpTerminationCriteria • 0 - stop at the number of nodes left reached • 1 - stop at time limit, • 2: - stop at the Solver with the best dual bound value has a certain number  of nodes • 3 - adaptive(node first) • 4 - adaptive (time first) • StopRacingNumberOfNodesLeft (300) • StopRacingTimeLimit (720 sec.) • NMaxRacingBaseParameters (in case RacingParamsDirPath is specified) • ug[SCIP,*] oriented parameters • CommunicateTighterBoundsInRacing (true): default racing parameters Combination of meta-parameters + random seeds Firste intenational UG workshop 2019, Berlin, Germany

  30. How UG does dynamic load balancing during primal single tree search in parallel p LoadCoordinator Open nodes: Try to keep p open nodes in LoadCoordinator Notification message: best lower bound, # nodes remain, # nodes solved - Send periodically and asynchronously - Interval is specified by a parameter Solver 1 Solver 2 Solver 3 Solver 4 Solver 5 Solver n LoadCoordinator makes selected Solversin collecting mode Expected to have heavy nodes: large subtree underneath Firste intenational UG workshop 2019, Berlin, Germany

  31. How UG does dynamic load balancing during primal single tree search in parallel p*mp p p LoadCoordinator Open nodes: Collecting mode Solver • Changes search strategy tobest dual bound first • Sends requested number of nodes Solver 1 Solver 2 Solver 3 Solver 4 Solver 5 Solver n Solver which has best lower bound node Firste intenational UG workshop 2019, Berlin, Germany

  32. How UG does dynamic load balancing during primal single tree search in parallel p LoadCoordinator Open nodes: Collecting mode Solver The # of solvers at atime is restricted Solver 2 Solver 1 Solver n Solver 5 Solver 4 Solver 3 • Starts from 1 • Dynamically switching • The number is increased by at most 250 even if run with 80,000 Solvers Solver which has best lower bound node Firste intenational UG workshop 2019, Berlin, Germany

  33. Parameters to control dynamic load balancing p p`*mp LoadCoordinator • NotificationInterval (1.0 sec) • NChangeIntoCollectingMode(1) • MultiplierForCollectingMode(2.0) • MinNumberOfCollectingModeSolvers (1) • MaxNumberOfCollectingModeSolvers(-1: nulimited) • TimeToIncreaseCMS(10.0 sec.) CMS: Collecting Mode solvers • If node pool is empty more than this time, increase # of CMS • CollectingModeInteraval(10.0 sec.) • Interval time between collecting modes less than this time, increase the number (+1) : NOTE: • Nodes are collected only from solvers which have heavy nodes • Solver dose not always send nodes (in case the root takes too long time) Open nodes: p mp p Firste intenational UG workshop 2019, Berlin, Germany

  34. What does the dynamic load balancing do Searches hardest part of sub-trees • Assign more computing resources to the sub-trees • Apply more presolving and cutting planes to sub-MIPs in the sub-tree. Firste intenational UG workshop 2019, Berlin, Germany

  35. Layered presolving Original A’’: Presolved (sub-)problem A’: Presolved (sub-)problem A: Original (sub-)problem A’: Original (sub-)problem UG causes algorithmic changes to the base solver Firste intenational UG workshop 2019, Berlin, Germany

  36. Check pointing Only the essential nodes are saved depending on run-time situation • Parameters for checkpoint • Checkpoint (FALSE) • CheckpointInterval(3600 sec.) Usually, 1800 sec. is used Firste intenational UG workshop 2019, Berlin, Germany

  37. Restarting Only the essential nodes are saved depending on run-time situation Huge trees might be thrown away, but the saved nodes’ lower bound values are calculated more precisely. Firste intenational UG workshop 2019, Berlin, Germany

  38. Implementation of deterministic mode LoadCoordinator Base solver I/O , presolve presolved instance presolved instance presolved instance presolved instance • All communications are done between LoadCoordinator and Solvers • => The messages between them are serialized • => Ordering the messages so that they are always performed the same order Solver 1 Solver 2 Solver 3 Solver 4 Firste intenational UG workshop 2019, Berlin, Germany

  39. Implementation of deterministic mode LoadCoordinator Base solver I/O , presolve presolved instance presolved instance presolved instance presolved instance Token • Tools: Deterministic timer (a counter) and a token • Circulate the token to all Solvers via LoadCoordinator • When a Solver reached to a communication interval time in the deterministic time, only the Solver that has token can starts communication, the others wait for the token Solver 1 Solver 2 Solver 3 Solver 4 Firste intenational UG workshop 2019, Berlin, Germany

  40. Instantiated parallel solvers by UG • Single thread base solver • ParaSCIP: ug[SCIP, MPI], FiberSCIP: ug[SCIP, Pthreads/C++11] • Multi-threaded base solver • ParaXpress: ug[Xpress, MPI], FiberXpress: ug[Xpress, Pthreads/C++11] • 80,000 (MPI processes) x 24 (threads) = 1,920,000 cores • ParaNUOPT: ug[NUOPT, MPI], FiberNUOPT: ug[NUOPT, Pthreads/C++11] • Only tested with single thread NUOPT • Distributed base solver • ug[PIPS-SBB, MPI] • PIPS-SBB: a distributed memory solver for Stochastic MIP • 80,000 (MPI processes) x 100 (MPI processes) = 8,000,000 cores • Concorde: TSP solver • ug[Concorde, MPI], ug[Concorde, Pthreds/C++11] • A new subproblem transfer mechanism via file • A new check-pointing and restarting mechanism • With Concorde inside parallelization Firste intenational UG workshop 2019, Berlin, Germany

  41. What is ug[SCIP,*] libraries • SCIP (Solving Constraint Integer Programs) • A software framework for LP based B&B algorithms • plugin based design • ug[SCIP,*] libraries allow SCIP users to parallelize their own customized SCIP solvers with minimal effort. • ug[SCIP, C++11/pthreads](=FiberSCIP) library: A software library to make a customized SCIP solver run on a shared memory environment in parallel • libugscip-0.8.6.linux.x86 64.gnu.opt.cpp11.a • libugscip-0.8.6.linux.x86 64.gnu.opt.pth.a • ug[SCIP, MPI](=ParaSCIP) library: A software library to make a customized SCIP solver run on a distributed memory environment in parallel • libugscip-0.8.6.linux.x86 64.gnu.opt.mpi.a User plugins are installed appropriately to FiberSCIP or ParaSCIPautomatically. • ug[SCIP,*] libraries are general purpose parallel B&B libraries. Firste intenational UG workshop 2019, Berlin, Germany

  42. Outline • What is Ubiquity Generator (UG) Framework • Brief history of the development • Computational behavior of state-of-the-art Branch-and-bound based solvers • Example: Mixed Integer Linear Program (MILP) solver • Anomalies of Parallel Branch-and-Bound • UG and how UG does parallel tree search • Ramp-up mechanism • Dynamic load balancing • Checkpoint and restart mechanism • Deterministic mode implementation • What is ug[SCIP, *] libraries • UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

  43. User needs to add glue code LoadCoordinator SCIP solver I/O , presolve FiberSCIP and ParaSCIP as libraries - Use “ScipUserPlugin class” class StpUserPlugins: public ScipUserPlugins { void operator()(SCIP *scip) {       /* include stppricer */       // SCIP_CALL_ABORT( SCIPincludePricerStp(scip) );       /* include steiner tree reader */       SCIP_CALL_ABORT( SCIPincludeReaderStp(scip) );       /* include event handler for printing primal solution development */       SCIP_CALL_ABORT( SCIPincludeEventHdlrBestsol(scip) );       /* include branching rule */       SCIP_CALL_ABORT( SCIPincludeBranchruleStp(scip) );       /* include propagator */       SCIP_CALL_ABORT( SCIPincludePropStp(scip) ); : } }; SCIP solver Copy from cmain.c Solver 1 Firste intenational UG workshop 2019, Berlin, Germany

  44. User can control algorithms flexibly LoadCoordinator SCIP solver I/O , presolve presolving settings Each solver settings Global root solving settings SCIP solver Three independent parameter sets can be specified Solver 1 Firste intenational UG workshop 2019, Berlin, Germany

  45. Built parallel solvers with ug[SCIP,*] libraries • ug[SCIP-Jack,*] • Parallel solvers for solving Steiner Tree Problems • ug[SCIP-SDP,*] • Parallel solvers for solving Mixed Integer Semidefinite Programs • New features (UG-0.8.6) • The availability of C++11 threads for communication, • customized racing ramp-up, which allows to use a set of user defined parameters for racing ramp-up, • branching on constraints for ug[SCIP,*], • user interfaces to initialize subproblems for ug[SCIP,*] • more aggressive layered presolving Firste intenational UG workshop 2019, Berlin, Germany

  46. The glue code would be small Firste intenational UG workshop 2019, Berlin, Germany

  47. Outline • What is Ubiquity Generator (UG) Framework • Brief history of the development • Computational behavior of state-of-the-art Branch-and-bound based solvers • Example: Mixed Integer Linear Program (MILP) solver • Anomalies of Parallel Branch-and-Bound • UG and how UG does parallel tree search • Ramp-up mechanism • Dynamic load balancing • Checkpoint and restart mechanism • Deterministic mode implementation • What is ug[SCIP, *] libraries • UG synthesizer Firste intenational UG workshop 2019, Berlin, Germany

  48. UG Synthesizer (UGS) • Run different solvers with different configurations in parallel as MPMD(Multiple Program, Multiple Data) model MPI program … ugsParaXpress config1 ugsPAC_Xpress config1 UGS ugsXpress config1 ugsXpress config2 ugsGurobi config1 ugsCPLEX config1 Mediate solution sharing can be distributed memory solvers : B&B solvers : Heuristic solvers mpirun-np 1 ../bin/ugstest-run-ugsc ../config/test.cfg: -np1 ../bin/ugs_gurobi1_1 ../IP/miplib2003/ds.mps.gz-ugsc ../config/test.cfg-paramgrb1.prm: -np1 ../bin/ugs_gurobi2_1../IP/miplib2003/ds.mps.gz-ugsc ../config/test.cfg-paramgrb2.prm:-np1 ../bin/ugs_xpress1_1../IP/miplib2003/ds.mps.gz-ugsc ../config/test.cfg-paramxpress1.prm: -np 2000 ../bin/paraxpress1_2000 ../settings/ds.set ../IP/miplib2003/ds.mps.gz–ugsc ../config/test.config -s paraxpress.set–srparaxpress.set–slparaxpress.set Firste intenational UG workshop 2019, Berlin, Germany

  49. Concluding remarks • Computational results for each parallel solver are presented during this workshop Firste intenational UG workshop 2019, Berlin, Germany

More Related