Derivative-Free Methods Using Linesearch Techniques: An Overview

Derivative-free Methods using Linesearch Techniques Stefano Lucidi

joint works with L. Grippo (the father linesearch approach) F. Lampariello M. Sciandrone P. Tseng G. Fasano G. Liuzzi V. Piccialli F. Rinaldi (in order of appearance in this research activity)

PROBLEM DEFINITION: are not available

MOTIVATIONS: In many engineering problems the objective and constraint function values are obtained by • direct measurements • complex simulation programs first order derivatives can be often neither explicitly calculated nor approximated

MOTIVATIONS: In fact • the mathematical representations of the objective function and the constraints are not available • the source codes of the programs are not available • the evaluations of the objective function and the constraints can be very expensive • the values of the objective function and theconstraintscan be affected by the presence of noise

MOTIVATIONS: the mathematical representations of the objective function and the constraints are not available the first order derivatives the objective function and the constraints can not be computed analytically

MOTIVATIONS: the source codes of the programs are not available the automatic differentiation techniques can not be applied

MOTIVATIONS: the evaluations of the objective function and the constraints can be very expensive the finite difference approximations can be too expensive (they need n function evaluations at least)

MOTIVATIONS: the values of the objective function and theconstraintscan be affected by the presence of noise finite difference approximations can produce very wrong estimates of the first order derivatives

where denotes a Gaussian distributed random number with zero mean and variance NUMERICAL EXPERIENCE: we considered 41 box constrained standard test problems we perturbed such problems in the following way:

NUMERICAL EXPERIENCE: we considered two codes: DF_box = derivative-free method E04UCF = NAG subroutine using finite-differences gradients Number of Failures DF_box E04UCF

Direct search methods use only function values Modelling methods approximate the functions by suitable models which are progressively built and updated GLOBALLY CONVERGENT DF METHODS - pattern search methods where the function is evaluated on specified geometric patterns - line search methods which use one-dimensional minimization along suitable search directions

UNCONSTRAINED MINIMIZATION PROBLEMS is not available is compact

THE ROLE OF THE GRADIENT characterizes accurately the local behaviour of f allows us to determine an "efficient" descent direction to determine a "good" step length along the direction

provides the rates of change of along the 2n directions THE ROLE OF THE GRADIENT is the directional derivatives of along characterizes accurately the local behaviour of

aset of directions can be associated at each the local behaviour of along should be indicative of the whole local behaviour of HOW TO OVERCOME THE LACK OF GRADIENT

Given , the bounded sequences are such that ASSUMPTION D

are linearly independent and bounded EXAMPLES OF SETS OF DIRECTIONS

are bounded EXAMPLES OF SETS OF DIRECTIONS (Lewis,Torczon)

EXAMPLES OF SETS OF DIRECTIONS

Assumption D ensures that, performing finer and finer sampling of along it is possible: • either to realize that the point is a good approximation of a stationary point of • or to find a point where is decreased UNCONSTRAINED MINIMIZATION PROBLEMS

GLOBAL CONVERGENCE By Assumption D we have:

By using satisfying Assumption D it is possible: to characterize the global convergence of a sequence of points the existence of suitable sequences of failuresin decreasing the objective function along the directions GLOBAL CONVERGENCE by means

GLOBAL CONVERGENCE By Assumption D we have:

PROPOSITION Let and be such that: - - satisfy Assumption D • there exist sequences of points and scalars such that then

GLOBAL CONVERGENCE • the Proposition characterizes in “some sense” the requirements on the accettable samplings of along the directions that guarantee the global convergence • it is not necessary to perform at each point a sampling of along all the directions • the sampling of along all the directions can be distributed along the iterations of the algorithm

Thedirect search methodscan divided in - pattern search methods - line search methods GLOBAL CONVERGENCE The use of directions satisfying Condition D and the result of producing sequences of points satisfying the hypothesis of the Proposition are the common elements of all the globally convergent direct search methods

PATTERN SEARCH METHODS Pros: they require that the new point produces a simple decrease of (in the line search methods the new point must guarantees a “sufficient” decrease of ) Cons: all the points produced must lie in a suitable lattice this implies - additional assumptions on the search directions - restrictions on the choiches of the steplenghts (in the line search methods no additional requiriments respect to Assumption D and the assumptions of the Proposition)

LINESEARCH TECHNIQUES

STEP 1 Computesatisfying AssumptionD ALGORITHM DF STEP 2 Minimization of along STEP 3 Compute and set k=k+1

STEP 2 The aim of this step is: - to detect the “promising” directions, the direction along which the function decreases “sufficiently” - to compute steplenghts along these directions which guarantee both a “sufficiently” decrease of the function and a “sufficient” moving from the previous point

LINESEARCH TECHNIQUE

The value of the initial step along the i-th direction derives from the linesearch performed along the i-th direction at the previuos iteration If the set of search directions does not depend on the iteration namely the scalar should be representative of the behaviour of the objective function along the i-th direction STEP 2

Find such that otherwise set STEP 3 set k=k+1 and go to Step 1 At Step 3, every approximation technique can be used to produce a new better point

THEOREM Let be the sequence of points produced by DF Algorithm then there exists an accomulation point of and every accumulation points of is a stationary point of the objective function GLOBAL CONVERGENCE

(LCP) is not available is compact is compact LINEARLY CONSTRAINED MINIMIZATION PROBLEMS

Given a feasible point it is possible to define • the set of the indeces of the active constraints • the set of the feasible directions LINEARLY CONSTRAINED MINIMIZATION PROBLEMS

is a stationary point for Problem (LCP) is a stationary point for Problem (LCP) LINEARLY CONSTRAINED MINIMIZATION PROBLEMS

is a stationary point for Problem (LCP) LINEARLY CONSTRAINED MINIMIZATION PROBLEMS

an estimate of the set of the indeces of the active constraints • an estimate of the set of the feasible directions LINEARLY CONSTRAINED MINIMIZATION PROBLEMS Given and it is possible to define • has good properties which allow us to define globally convergent algorithms

Given and the set of directions with satisfies: is uniformly bounded ASSUMPTION D2 (an example)

STEP 1 Computesatisfying AssumptionD2 STEP 2 Minimization of along STEP 3 Compute the new point and set k=k+1 ALGORITHM DFL

THEOREM Let be the sequence of points produced by DFL Algorithm then there exists an accomulation point of and every accumulation points of is a stationary point for Problem (LCP) GLOBAL CONVERGENCE

(BCP) is not available is compact satisfies Assumption D2 the set BOX CONSTRAINED MINIMIZATION PROBLEMS

(NCP) is not available are not available NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS

We define and given a point NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS

ASSUMPTION A1 The set is compact ASSUMPTION A2 there exists a vector such that For every boundeness of the iterates Assumption A1 existence and boundeness of the Lagrange multipliers Assumption A2 NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS

where (penalty parameter) NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS We consider the following continuously differentiable penalty function:

Derivative-Free Methods Using Linesearch Techniques: An Overview