optim-results

Benchmarking

The point of the tests here is deliberately not to precondition, tune, and/or bound the optimizations for better performance. Instead, it is to see how robust the minimizers are when not tuned.

It is often the case that users do not know the properties of the objective function—which is why they need a numerical optimizer to begin with. On some level, this has to be true. If they know enough about the function, they can just derive the optimum in closed form. Moreover, in many economics applications, there are non-linear non-box constraints, which are difficult or impossible to derive; and difficult to describe to the program, but which limited the valid domain of the objective function.

The designed test functions are deliberately heavy in terms of not having nice quadratic shapes. The simple V, in particular, is linear and thus should give gradient based optimizers a headache (although most do just fine). The other functions tend to be concave, like funnels. In economics applications, ill-behaved surfaces can often contain such shapes (a tradeoff between two forces), even though the test functions may overemphasize the extreme shapes.
The full code is at the chapter end.

The Optim.jl package is not yet fully debugged and consistent. I am sure it will get better in due time. For example, some optimizers (ParticleSwarm and NewtonTrustRegion) barf when I try to use `Fminbox` to box-bound. Waiting for the next draft with more consistent interface…

Particle Swarm can try to feed NaN's.

The bounded version does not seem to work with Optim.options().

`optimize( f, [1.0, 2.0], NelderMead(), show_trace= true )` flunks

Test Functions

The first five test functions (the first three of which are standard, though modified to shift the optimum to [0.0,0.0]) are designed not to be separable, and thus require 3-D plots for insight:

 Rosenbrock Bukin Easom Polynomial LogMeanVar `f^*([0,0])=0` `f([-10,1]=0` `f([pi,pi])=0` `f([2,3])=0` `f([0,0])=0`

The remaining eight functions are separable (sums or products of single-dimensional cuts). Thus, it is easier to present their one-dimensional cuts. All are strictly positive, so that products are still monotonic.

 SqAtan V VSqr Cliff Kinky Super Wiggler Asym Limited ——– — —– —– —– —– ——- —- ——- `f^*([0,0])=1` `f^*([0,0])>0` `f^*([0,0])=1` `f^*([0,0])=0` `f^*([0,0])=0` `f^*([0,0])=0` `f^*([0,0])=4` `f^*([0,0])=2.828`

Common Optimization Test Functions

These algorithms can be found here.

Rosenbrock

snippet.juliarepl
```function rosenbrock(x::Vector{Float64})::Float64
assert( length(x) == 2 )
(1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2  ## min=[0,0]
end```

Bukin #6

snippet.juliarepl
```function bukin(x::Vector{Float64})::Float64  ## min( –10, 1) = 0
assert( length(x) == 2)
xt= [ truncme( x[1], –15.0, –5.0 ), truncme( x[2], –3.0, 3.0 ) ]
100.0*(sqrt( abs( xt[2]–0.01*xt[1]^2 ) ) + 0.01*abs( xt[1]+10.0 )) + exp(5.0+sum(abs.( xt-x )))
end#function```

Easom

snippet.juliarepl
```function easom(x::Vector{Float64})::Float64  ## min(pi,pi) = –1
assert( (eltype(x) == Float64) && (length(x) == 2) )
xt= [ truncme( x[1], –100.0, 100.0 ), truncme( x[2], –100.0, 100.0 ) ]
1.0 -cos(xt[1])*cos(xt[2])*exp( - ((xt[1]-pi)^2 + (xt[2]-pi)^2) ) + exp(5.0+sum(abs.( xt-x )))
end#function```

Self-Designed Test functions

snippet.juliarepl
```function myfun(x::Vector{Float64})::Float64   ## min f(2.0,3.0,...,)=0
sum( (x .- [ i+1 for i=1:length(x) ]).^2 ) + prod( (x .- [ i+1 for i=1:length(x) ]) )
end#function```

The optimal solution is 0 at [2,3]. Most gradient optimizers were more or less designed to work functions like this simply polynomial. 3D-Plot

The remaining test cases are n-dimensional.

Squared Atan

snippet.juliarepl
```function sqatan(x::Vector{Float64})::Float64
onesqatan(x::Float64)= atan( x^2 )^(2+(x.>0)) end#function
prod( (1.0 .+ onesqatan.(x)) )
end#function```

Mildly asymmetric. The optimal solution is 0 at [0,0…]. 2D-Plot

LogMeanVar

A non-separable but simple function of mean and variance.

snippet.juliarepl
```function logmeanvar(x::Vector{Float64})::Float64
am= abs(mean(x)); v= var(x)
log( 1.0 + v + am + sqrt(am*v) )
end#function```

The optimal solution is 0 at [0,0…]. 3D-Plot

The remaining test cases are all separable in their arguments. This should make optimization simpler than complex depending relationships.

Linear V-Function

snippet.juliarepl
```function vfun(x::Vector{Float64})::Float64
onevfun(x::Float64) = abs(x)*(1.0+(x>0))
prod( 1.0 .+ onevfun.(x) )
end#function```

The optimal solution is 0 at [0,0…]. The function is asymmetric. 2D-Plot 3D-Plot

V-Squareroot Function

snippet.juliarepl
```function vsqrtfun(x::Vector{Float64})::Float64
assert( eltype(x) == Float64 )
onesqrtvfun(x::Float64) = sqrt(abs(x)*(1.0+(x>0)))
prod( 1.0 .+ onesqrtvfun.(x) )
end#function```

The optimal solution is 0 at [0,0…]. The function is asymmetric. 2D-Plot 3D-Plot

Cliff (Discontinuous) Funnel

snippet.juliarepl
```function cliffunnel(x::Vector{Float64})::Float64
onecliffunnel(x::Float64) =  10.01 + ((x<0.5) ? 1.0/(x–1) : –10.0/(x+1))

kc= onecliffunnel.( x .+ (0.5) )
@assert( all( x ->(x >= 0), kc ), "dc received bad input request \$kc at \$x" )
prod( kc )
end#function```

The optimal solution is 0 at [0,0…]. The function is asymmetric. 2D-Plot 3D-Plot

Kinky Funnel

snippet.juliarepl
```function kinkyfunnel(x::Vector{Float64})::Float64
onekinkyfunnel(x::Float64) = (1.01 + ((x<0) ? 1.0/(x–1) : –1.0/(x+1)))*10.0

kc= onekinkyfunnel.(x)
@assert( all( x ->(x >= 0), kc ), "dc received bad input request \$kc at \$x" )
prod( kc )
end#function```

The optimal solution is 0 at [0,0…]. The function is asymmetric. 2D-Plot 3D-Plot

Super(-kinky) Funnel

snippet.juliarepl
```function superfunnel(x::Vector{Float64})::Float64
onesuperfunnel(x::Float64) = log(  1.0 / max( abs(x) , 1e–5 ) )

15*length(x) - sum( onesuperfunnel.(x) )
end#function```

The optimal solution is 0 at [0,0…]. The function is asymmetric and very steep until 1e-5, where it becomes flat for a tiny region of width 2e-5. 2D-Plot 3D-Plot

Wiggler

snippet.juliarepl
```function wiggler(x::Vector{Float64})::Float64
function onewig(x::Float64)::Float64
xform= sqrt(abs(x*10))
1.5*(1+(x>0)) * (xform + 0.9*sin(xform))+0.01     ## +0.01 to assure positive, or setting any to 0 is optimal
end#function

prod( onewig.(x) )  ## only min at 0, but many saddle points right and many local minima left
end#function```

The optimal solution is 0 at [0,0…]. The function is asymmetric. 2D-Plot 3D-Plot

Very Asymmetric

snippet.juliarepl
```function veryasym(x::Vector{Float64})::Float64
function oneveryasym(x::Float64)::Float64;   ## asymptotes on the left, does not on the right
const xm = –0.42385379906978327137804006262551523367638819718517754082301
const yac = –0.40089940449996159253408195381069828929801311039560261672969
const ysq = –0.17965204298588821036533019640026627767324063903383614189857
-( (x) ? atan(x) : -x^2 - ysq + yac) - 0.22124736151407337 + 1.0
end#function

prod(1.0 .+ oneveryasym.(x) )   ## min is at 0,0,0...
end#function```

The optimal solution is 0 at [0,0…]. The function is asymmetric. 2D-Plot 3D-Plot

Domain Limited

snippet.juliarepl
```function limited_domain(x::Vector)::Float64
sum( sqrt.(1 ./ (x+1) .+ (x+1) ) )  ## min is at 1,1,1...
end#function```

Used to test if the optimizers have some sense to pull back from invalid regions. 2D-Plot 3D-Plot

Performance

Overview of Optimizers By Test Function

The starting point are always a set of uniform random floating point values from 0 to 10. Only the SAMIN (Simulated Annealing) is bounded at -100 to +100 because it cannot be run unbounded. SAMIN and Prtcl are global search algorithms. Prtle (Particle Swarm) runs for as long as the user desires, ending only when the maximum number of iterations is reached. Here, it was set to 1 million. The remaining search algorithms are happy at any local optimum. The numbers in the table below are function evaluations.

testfun dim SAMIN Prtcl NM GrdDsct CnjgGrd BFGS LBFGS Nwt NwTst
 rosenbrock 2 24,851 - 254 109,645 137 153 103 121 35 easom 2 13,801 - - - - - - - - bukin 2 - - - - - - - - -
 testfun dim SAMIN Prtcl NM GrdDsct CnjgGrd BFGS LBFGS Nwt NwTst mypolynom 2 20,951 4,000,000 83 18 20 38 20 20 8 atansq 5 42,376 6,000,000 - 939 2,143 81 97 104 24 18 vfun 2 17,951 4,000,000 182 - 132 170 118 649 - vfun 5 60,751 6,000,000 - - - - 323 994 - vfun 10 174,001 11,000,000 - - - 1,071 729 - - vsqrtfun logmeanvar 2 15,051 4,000,000 112 - 199 91 133 206 390,173 logmeanvarn 5 35,376 - - - - 145 - 224 283,296 logmeanvar 10 71,001 11,000,000 - 10,581 - 123 145 277 86,901 cliff funnel 2 16,051 4,000,000 502 - - - - 187 7,396 cliff funnel 5 49,001 - - - - - - - - cliff funnel 10 120,751 - - - - - - - kinky funnel 2 16,151 - 149 - 406 153 83 136 - kinky funnel 5 52,876 - - - - - - 600 - kinky funnel 10 139,751 11,000,000 - - - 602 - 1,591 - super funnel 2 1,000,000 4,000,000 134 180 313 144 104 11 106 super funnel 5 1,000,000 6,000,000 - 437 - 241 - 64 - super funnel 10 1,000,000 11,000,000 - 922 - - - 56 - wiggler 2 17,501 4,000,000 - - - - - - - wiggler 5 58,751 6,000,000 - - - - - 181 - wiggler 10 205,751 11,000,000 - - - - - - - asym 2 18,351 4,000,000 84 - 74 79 78 16 10 asym 5 54,626 6,000,000 1,080 - - - - 29 21 asym 10 153,751 11,000,000 1,625 - - - - 69 30 limited_domain 2 - - - - - - - - -

The most surprising aspect to me was not how fast or slow the algorithms are, but how often algorithms simply fail to find the optimum, and sometimes they even claimed they did when they did not.

Detailed Output from the Maximizations

myfun Polynomial

myfun: f( x^* )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 42 83 f([2.0, 3.0])= 0.0 30.586 μs (571 allocations: 15.14 KiB)
SAMIN 2 found no 20951 20951 f([2.0, 3.0])= 0.0 4.787 ms (173105 allocations: 5.28 MiB)
Prtcl 2 found no 1000000 4000000 f([2.0, 3.0])= 0.0 2.353 s (38000068 allocations: 1022.34 MiB)
GrdDsct 2 found ok 5 18 f([2.0, 3.0])= 0.0 19.068 μs (272 allocations: 9.02 KiB)
CnjgGrd 2 found ok 9 20 f([2.0, 3.0])= 0.0 20.512 μs (302 allocations: 10.80 KiB)
BFGS 2 found ok 12 38 f([2.0, 3.0])= 0.0 40.766 μs (565 allocations: 17.47 KiB)
LBFGS 2 found ok 6 20 f([2.0, 3.0])= 0.0 28.401 μs (345 allocations: 13.19 KiB)
Nwt 2 found ok 7 20 f([2.0, 3.0])= 0.0 33.194 μs (435 allocations: 19.23 KiB)
NwTst 2 found ok 7 8 f([2.0, 3.0])= 0.0 51.419 μs (634 allocations: 40.66 KiB)

Modified Rosenbrock

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 137 254 f([1.0, 1.0])= 0.0 81.755 μs (1434 allocations: 36.05 KiB)
SAMIN 2 found no 24851 24851 f([1.0, 1.0])= 0.0 4.952 ms (175240 allocations: 5.40 MiB)
Prtcl 2 not no 1000000 4000000 f([3.885, 15.096])= 8.3253 2.427 s (38006260 allocations: 1022.44 MiB)
GrdDsct 2 found no 43454 109645 f([1.0, 1.0])= 0.0 115.483 ms (1701597 allocations: 46.69 MiB)
CnjgGrd 2 found ok 48 137 f([1.0, 1.0])= 0.0 136.907 μs (2114 allocations: 60.44 KiB)
BFGS 2 found ok 56 153 f([1.0, 1.0])= 0.0 167.991 μs (2394 allocations: 68.94 KiB)
LBFGS 2 found ok 35 103 f([1.0, 1.0])= 0.0 132.633 μs (1586 allocations: 48.09 KiB)
Nwt 2 found ok 44 121 f([1.0, 1.0])= 0.0 186.693 μs (2574 allocations: 106.48 KiB)
NwTst 2 found ok 34 35 f([1.0, 1.0])= 0.0 229.502 μs (2509 allocations: 165.39 KiB)

Modified Easom

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 not ok 0 4 f([8.236, 9.104])= 1.0 5.265 μs (104 allocations: 5.50 KiB)
SAMIN 2 found no 13801 13801 f([3.142, 3.142])= 0.0 4.923 ms (156878 allocations: 7.49 MiB)
Prtcl 2 not no 1000000 4000000 f([19.526, 69.33])= 1.0 3.021 s (58000044 allocations: 2.19 GiB)
GrdDsct 2 not ok 0 1 f([8.236, 9.104])= 1.0 4.220 μs (73 allocations: 4.25 KiB)
CnjgGrd 2 not ok 0 1 f([8.236, 9.104])= 1.0 4.331 μs (77 allocations: 4.66 KiB)
BFGS 2 not ok 0 1 f([8.236, 9.104])= 1.0 4.251 μs (78 allocations: 4.78 KiB)
LBFGS 2 not ok 0 1 f([8.236, 9.104])= 1.0 8.911 μs (112 allocations: 7.59 KiB)
Nwt 2 not ok 0 1 f([8.236, 9.104])= 1.0 7.239 μs (130 allocations: 8.11 KiB)
NwTst 2 not ok 0 1 f([8.236, 9.104])= 1.0 7.303 μs (131 allocations: 8.14 KiB)

Modified Bukin

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 not ok 160 313 f([-5.0, 0.25])= 5.0 129.186 μs (3369 allocations: 141.44 KiB)
SAMIN 2 not no 10951 10951 f([-9.958, 0.992])= 0.0418 3.691 ms (126516 allocations: 5.32 MiB)
Prtcl 2 not no 1000000 4000000 f([-5.0, 0.25])= 5.0 2.945 s (58000080 allocations: 2.19 GiB)
GrdDsct 2 not no 14 124 f([-2.728, 0.25])= 7.272 178.542 μs (4855 allocations: 239.91 KiB)
CnjgGrd 2 not no 13 133 f([-3.306, 0.25])= 6.694 194.574 μs (5285 allocations: 250.30 KiB)
BFGS 2 not no 11 156 f([-5.0, 0.25])= 5.0479 219.807 μs (5955 allocations: 299.45 KiB)
LBFGS 2 not no 17 220 f([-1.002, 0.25])= 8.9981 332.933 μs (8783 allocations: 425.70 KiB)
Nwt 2 not no 11 125 f([-5.071, 0.257])= 4.9317 217.024 μs (5936 allocations: 293.77 KiB)
NwTst 2 not no 1000000 1000001 f([-2.399, 0.25])= 7.6008 7.217 s (148626965 allocations: 8.87 GiB)

vfun

vfun: f( x^*=[0] )= 1.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 94 182 f([-0.0, -0.0])= 1.0 66.178 μs (1326 allocations: 45.22 KiB)
SAMIN 2 found no 17951 17951 f([-0.0, 0.0])= 1.0 4.920 ms (160497 allocations: 5.96 MiB)
Prtcl 2 found no 1000000 4000000 f([-0.0, -0.0])= 1.0 2.552 s (42000788 allocations: 1.36 GiB)
GrdDsct 2 not no 15 57 f([-5.057, 0.0])= 6.0566 269.225 μs (1416 allocations: 59.34 KiB)
CnjgGrd 2 found ok 26 132 f([-0.0, -0.0])= 1.0 555.100 μs (3263 allocations: 123.09 KiB)
BFGS 2 found no 24 170 f([-0.0, 0.0])= 1.0 759.124 μs (3294 allocations: 149.39 KiB)
LBFGS 2 found ok 31 118 f([-0.0, -0.0])= 1.0 584.274 μs (3059 allocations: 125.05 KiB)
Nwt 2 found no 32 649 f([-0.0, -0.0])= 1.0 3.171 ms (16935 allocations: 687.03 KiB)
NwTst 2 not ok 50 51 f([-0.0, 5.041])= 11.0816 472.729 μs (5170 allocations: 334.50 KiB)

vfun: f( x^*=[0] )= 1.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 5 not ok 1020 1847 f(-0.002 … 7.964)= 16.9939 700.025 μs (12906 allocations: 487.05 KiB)
SAMIN 5 found no 60751 60751 f(-0.0 … 0.0)= 1.0 16.666 ms (519183 allocations: 24.53 MiB)
Prtcl 5 found no 1000000 6000000 f(-0.0 … 0.0)= 1.0 3.292 s (55003308 allocations: 2.19 GiB)
GrdDsct 5 not no 28 120 f(-0.0 … 8.65)= 1058.4901 551.155 μs (2826 allocations: 213.44 KiB)
CnjgGrd 5 not no 159 976 f(-2.489 … -0.0)= 3.4892 4.326 ms (28373 allocations: 1.59 MiB)
BFGS 5 not no 39 208 f(-0.131 … 0.173)= 1.5527 1.044 ms (6283 allocations: 394.23 KiB)
LBFGS 5 found no 88 323 f(-0.0 … -0.0)= 1.0 1.669 ms (9890 allocations: 620.66 KiB)
Nwt 5 found no 49 994 f(-0.0 … -0.0)= 1.0 5.273 ms (35399 allocations: 2.22 MiB)
NwTst 5 not ok 953 954 f(-0.0 … 5.156)= 11.3114 20.128 ms (158356 allocations: 17.82 MiB)

vfun: f( x^*=[0] )= 1.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 10 not ok 2188 3754 f(-0.0 … 5.463)= 2548.9504 1.742 ms (27950 allocations: 1.11 MiB)
SAMIN 10 found no 174001 174001 f(-0.0 … 0.0)= 1.0 47.799 ms (1452761 allocations: 86.49 MiB)
Prtcl 10 found no 1000000 11000000 f(-0.0 … 0.0)= 1.0 5.960 s (85009240 allocations: 4.34 GiB)
GrdDsct 10 not no 59 179 f(-1.617 … 9.081)= 148482.4095 947.524 μs (6984 allocations: 683.22 KiB)
CnjgGrd 10 not no 218 1073 f(-0.0 … 8.367)= 4096.311 4.902 ms (38863 allocations: 3.31 MiB)
BFGS 10 found no 200 1071 f(-0.0 … 0.0)= 1.0002 5.730 ms (40980 allocations: 3.93 MiB)
LBFGS 10 found no 203 729 f(-0.0 … 0.0)= 1.0 4.027 ms (28447 allocations: 2.70 MiB)
Nwt 10 not no 160 3027 f(-1.798 … 0.0)= 2.8011 18.863 ms (164023 allocations: 16.52 MiB)
NwTst 10 not ok 144 145 f(-0.0 … 9.446)= 181642.4673 6.261 ms (38296 allocations: 6.48 MiB)

logfun

logfun: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 57 112 f([-0.0, 0.0])= 0.0 48.529 μs (1076 allocations: 32.95 KiB)
SAMIN 2 found no 15051 15051 f([-0.0, 0.0])= 0.0 4.538 ms (152658 allocations: 5.07 MiB)
Prtcl 2 found no 1000000 4000000 f([-0.01, 0.01])= 0.0002 2.774 s (50000092 allocations: 1.48 GiB)
GrdDsct 2 not no 12 38 f([8.431, 8.431])= 2.244 64.156 μs (1327 allocations: 44.13 KiB)
CnjgGrd 2 found no 27 199 f([0.017, -0.017])= 0.0006 275.932 μs (5788 allocations: 188.77 KiB)
BFGS 2 found ok 29 91 f([-0.0, 0.0])= 0.0 153.539 μs (3061 allocations: 102.31 KiB)
LBFGS 2 found no 24 133 f([-0.0, 0.0])= 0.0 200.918 μs (3803 allocations: 133.27 KiB)
Nwt 2 found no 29 206 f([-0.0, 0.0])= 0.0 371.247 μs (7682 allocations: 282.80 KiB)
NwTst 2 found no 390172 390173 f([-0.0, -0.0])= 0.0 2.138 s (40187380 allocations: 2.22 GiB)

logfun: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 5 not ok 469 897 f(-0.331 … 0.712)= 0.178 390.233 μs (8429 allocations: 242.11 KiB)
SAMIN 5 found no 35376 35376 f(-0.003 … 0.002)= 0.0 10.650 ms (357451 allocations: 13.61 MiB)
Prtcl 5 not no 1000000 6000000 f(-0.079 … 0.229)= 0.0167 3.713 s (67000068 allocations: 2.19 GiB)
GrdDsct 5 not no 20 60 f(4.018 … 4.018)= 1.6131 131.087 μs (3136 allocations: 113.80 KiB)
CnjgGrd 5 not no 25 154 f(-0.099 … 0.069)= 0.0068 269.514 μs (6412 allocations: 236.41 KiB)
BFGS 5 found no 23 145 f(-0.0 … 0.0)= 0.0 274.467 μs (6509 allocations: 249.52 KiB)
LBFGS 5 not no 16 127 f(-0.119 … 0.083)= 0.0097 254.100 μs (5930 allocations: 222.83 KiB)
Nwt 5 found no 32 224 f(-0.001 … 0.0)= 0.0 676.867 μs (16821 allocations: 667.91 KiB)
NwTst 5 found no 283295 283296 f(-0.0 … -0.0)= 0.0 4.292 s (71671384 allocations: 4.28 GiB)

logfun: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 10 not ok 2476 3703 f(-0.113 … 0.148)= 0.0064 2.086 ms (36320 allocations: 1.03 MiB)
SAMIN 10 found no 71001 71001 f(-0.003 … 0.004)= 0.0 22.021 ms (692016 allocations: 29.86 MiB)
Prtcl 10 found no 1000000 11000000 f(-0.004 … 0.007)= 0.0 6.990 s (107001188 allocations: 4.01 GiB)
GrdDsct 10 found no 4750 10581 f(-0.001 … 0.01)= 0.0 35.682 ms (847344 allocations: 32.14 MiB)
CnjgGrd 10 not no 26 209 f(-0.074 … 0.046)= 0.0021 561.816 μs (14804 allocations: 568.27 KiB)
BFGS 10 found ok 35 123 f(-0.0 … 0.0)= 0.0 399.082 μs (10228 allocations: 388.25 KiB)
LBFGS 10 found no 26 145 f(-0.0 … 0.0)= 0.0 417.485 μs (10942 allocations: 436.73 KiB)
Nwt 10 found no 39 277 f(-0.007 … 0.012)= 0.0 1.715 ms (47741 allocations: 1.91 MiB)
NwTst 10 found ok 86900 86901 f(-0.0 … 0.0)= 0.0 4.764 s (65813391 allocations: 3.81 GiB)

discfunnel

discfunnel: f( x^*=[0] )= 11.178

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 250 502 f([0.0, 0.0])= 11.1779 154.525 μs (2894 allocations: 106.91 KiB)
SAMIN 2 found no 16051 16051 f([0.0, 0.0])= 11.1779 4.842 ms (131128 allocations: 4.89 MiB)
Prtcl 2 found no 1000000 4000000 f([0.0, 0.0])= 11.1779 — internal error AssertionError(“dc received bad input request [NaN] —
GrdDsct 2 not no 4 126 f([0.0, 7.31])= 29.6723 811.513 μs (3013 allocations: 120.66 KiB)
CnjgGrd 2 not no 4 179 f([-2.582, 0.0])= 32.382 817.644 μs (3412 allocations: 149.97 KiB)
BFGS 2 not no 2 93 f([-0.997, 0.0])= 31.2333 654.342 μs (2294 allocations: 91.09 KiB)
LBFGS 2 not no 4 121 f([0.0, -0.902])= 31.0816 550.470 μs (2263 allocations: 105.83 KiB)
Nwt 2 found no 4 187 f([0.0, 0.0])= 11.1781 1.245 ms (4283 allocations: 180.73 KiB)
NwTst 2 found ok 7399 7396 f([0.0, 0.0])= 11.1782 112.330 ms (605959 allocations: 42.72 MiB)

discfunnel: f( x^*=[0] )= 417.733

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 5 not ok 1307 2467 f(0.0 … 49.601)= 3288.2925 927.573 μs (16633 allocations: 635.52 KiB)
SAMIN 5 found no 49001 49001 f(-0.0 … 0.0)= 417.7326 16.917 ms (440809 allocations: 21.26 MiB)
Prtcl 5 FAILED WITH ERROR! AssertionError(“dc received bad input request [NaN, NaN, NaN, NaN, NaN] at [NaN, NaN, NaN, NaN, NaN]”)
GrdDsct 5 not no 1 88 f(-0.309 … 8.974)= 16738.9271 655.923 μs (2654 allocations: 164.56 KiB)
CnjgGrd 5 not no 2 154 f(-0.309 … 8.974)= 16738.8469 747.465 μs (3793 allocations: 267.22 KiB)
BFGS 5 not no 1 88 f(-0.309 … 8.974)= 16738.9271 662.672 μs (2659 allocations: 165.44 KiB)
LBFGS 5 not no 1 88 f(-0.309 … 8.974)= 16738.9271 671.836 μs (2693 allocations: 168.69 KiB)
Nwt 5 not no 6 157 f(-0.341 … 0.21)= 1371.9827 1.126 ms (4735 allocations: 333.53 KiB)
NwTst 5 not ok 2651 2640 f(0.0 … 0.0)= 417.8715 64.929 ms (359591 allocations: 41.02 MiB)

discfunnel: f( x^*=[0] )= 174500.543

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 10 not ok 6911 12448 f(0.0 … 31.544)= 4.5364731487e6 5.510 ms (83248 allocations: 3.53 MiB)
SAMIN 10 found no 120751 120751 f(-0.0 … 0.0)= 174500.5431 43.769 ms (1016278 allocations: 66.26 MiB)
Prtcl 10 FAILED WITH ERROR! AssertionError(“dc received bad input request [NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN] at [NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]”)
GrdDsct 10 not no 1 119 f(0.0 … 9.726)= 2.362840786419e8 887.289 μs (4345 allocations: 438.75 KiB)
CnjgGrd 10 not no 1 119 f(0.0 … 9.726)= 2.36284078642e8 889.420 μs (4352 allocations: 439.53 KiB)
BFGS 10 not no 1 119 f(0.0 … 9.726)= 2.362840786419e8 892.231 μs (4350 allocations: 440.30 KiB)
LBFGS 10 not no 1 119 f(0.0 … 9.726)= 2.362840786419e8 902.717 μs (4384 allocations: 443.66 KiB)
Nwt 10 not no 4 210 f(-1.371 … 10.81)= 4.09198211248e7 1.640 ms (8952 allocations: 920.47 KiB)
NwTst 10 not ok 152 141 f(0.0 … 9.68)= 3.31713791228e7 6.904 ms (34123 allocations: 5.87 MiB)

kinkyfunnel

kinkyfunnel: f( x^*=[0] )= 0.01

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 78 149 f([0.0, 0.0])= 0.01 58.626 μs (1138 allocations: 38.45 KiB)
SAMIN 2 found no 16151 16151 f([-0.0, 0.0])= 0.01 4.833 ms (137414 allocations: 5.12 MiB)
Prtcl 2 FAILED WITH ERROR! AssertionError(“dc received bad input request [NaN, NaN] at [NaN, NaN]”)
GrdDsct 2 not no 13 114 f([-2.973, 0.0])= 0.7583 110.594 μs (1836 allocations: 87.70 KiB)
CnjgGrd 2 found ok 49 406 f([0.0, 0.0])= 0.01 493.096 μs (9242 allocations: 347.03 KiB)
BFGS 2 found ok 24 153 f([-0.0, 0.0])= 0.01 213.767 μs (3963 allocations: 147.95 KiB)
LBFGS 2 found ok 23 83 f([0.0, 0.0])= 0.01 129.218 μs (1968 allocations: 81.73 KiB)
Nwt 2 found ok 23 136 f([0.0, -0.0])= 0.01 234.932 μs (4380 allocations: 182.25 KiB)
NwTst 2 not no 1000000 1000001 f([0.0, 5.783])= 0.8626 6.228 s (101361826 allocations: 6.46 GiB)

kinkyfunnel: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 5 not ok 313 621 f(-0.0 … 12.628)= 379.037 251.835 μs (4724 allocations: 169.88 KiB)
SAMIN 5 found no 52876 52876 f(-0.0 … 0.0)= 0.0 14.864 ms (425737 allocations: 20.43 MiB)
Prtcl 5 FAILED WITH ERROR! AssertionError(“dc received bad input request [NaN, NaN, NaN, NaN, NaN] at [NaN, NaN, NaN, NaN, NaN]”)
GrdDsct 5 not no 10 117 f(-5.219 … 8.777)= 568.0043 151.400 μs (2657 allocations: 197.48 KiB)
CnjgGrd 5 not no 11 159 f(-142.169 … -0.0)= 201.6754 208.813 μs (3704 allocations: 260.30 KiB)
BFGS 5 not no 26 188 f(-9.585 … 0.0)= 0.0367 279.398 μs (4924 allocations: 330.27 KiB)
LBFGS 5 not no 41 224 f(0.0 … 9.026)= 1.1885 352.144 μs (5791 allocations: 397.02 KiB)
Nwt 5 found ok 72 600 f(-0.0 … 0.0)= 0.0 1.357 ms (25951 allocations: 1.68 MiB)
NwTst 5 not no 1000000 1000001 f(0.0 … 9.08)= 286.3696 15.769 s (153717831 allocations: 16.96 GiB)

kinkyfunnel: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 10 not ok 969 1853 f(0.0 … 13.869)= 148915.2376 896.983 μs (14175 allocations: 566.98 KiB)
SAMIN 10 found no 139751 139751 f(-0.0 … 0.0)= 0.0 43.681 ms (1207611 allocations: 71.13 MiB)
Prtcl 10 found no 1000000 11000000 f(-0.0 … 0.0)= 0.0 — internal error AssertionError(“dc received bad input request [NaN] —
GrdDsct 10 not no 14 148 f(-18.283 … 9.621)= 5.4257644926e6 300.370 μs (4945 allocations: 533.25 KiB)
CnjgGrd 10 not no 40 341 f(-18.607 … 9.604)= 114126.3874 697.288 μs (11494 allocations: 1.12 MiB)
BFGS 10 found no 94 602 f(-45.121 … 20.748)= 0.0001 1.392 ms (22624 allocations: 2.17 MiB)
LBFGS 10 not no 59 388 f(-5.295 … 9.373)= 2519.1319 899.672 μs (14310 allocations: 1.39 MiB)
Nwt 10 found ok 221 1591 f(-0.0 … 0.0)= 0.0 7.361 ms (122664 allocations: 13.22 MiB)
NwTst 10 not no 1000000 1000001 f(-0.0 … 9.73)= 3.1841418085e6 49.026 s (296960809 allocations: 48.65 GiB)

superkinkyfunnel

superkinkyfunnel: f( x^*=[0] )= 6.974

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 63 134 f([-0.0, 0.0])= 6.9741 51.524 μs (964 allocations: 34.80 KiB)
SAMIN 2 found no 1000000 1000000 f([0.0, -0.0])= 6.9741 276.578 ms (7606164 allocations: 315.95 MiB)
Prtcl 2 found no 1000000 4000000 f([0.0, -0.0])= 6.9741 2.869 s (42000224 allocations: 1.36 GiB)
GrdDsct 2 found ok 13 180 f([-0.0, -0.0])= 6.9741 231.521 μs (3872 allocations: 156.77 KiB)
CnjgGrd 2 found ok 41 313 f([0.0, 0.0])= 6.9741 385.744 μs (6248 allocations: 257.28 KiB)
BFGS 2 found ok 14 144 f([-0.0, 0.0])= 6.9741 199.684 μs (3250 allocations: 128.70 KiB)
LBFGS 2 found ok 15 104 f([-0.0, 0.0])= 6.9741 162.491 μs (2533 allocations: 100.30 KiB)
Nwt 2 found ok 2 11 f([-0.0, -0.0])= 6.9741 25.689 μs (333 allocations: 17.08 KiB)
NwTst 2 found ok 110 106 f([-0.0, -0.0])= 6.9741 505.708 μs (8944 allocations: 616.70 KiB)

superkinkyfunnel: f( x^*=[0] )= 17.435

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 5 not ok 898 1751 f(-0.0 … 12.793)= 43.9004 737.321 μs (11837 allocations: 466.72 KiB)
SAMIN 5 found no 1000000 1000000 f(-0.0 … 0.0)= 17.4354 279.827 ms (7254614 allocations: 383.10 MiB)
Prtcl 5 found no 1000000 6000000 f(-0.0 … 0.0)= 17.4354 3.612 s (55001028 allocations: 2.19 GiB)
GrdDsct 5 found ok 38 437 f(-0.0 … 0.0)= 17.4354 873.435 μs (11780 allocations: 770.89 KiB)
CnjgGrd 5 not no 27 319 f(-0.0 … 7.031)= 47.9639 600.601 μs (7716 allocations: 529.58 KiB)
BFGS 5 found ok 39 241 f(-0.0 … 0.0)= 17.4354 521.285 μs (6942 allocations: 438.05 KiB)
LBFGS 5 not no 35 202 f(-0.0 … 1.715)= 29.4876 452.635 μs (5871 allocations: 371.27 KiB)
Nwt 5 found ok 7 64 f(-0.0 … 0.0)= 17.4354 184.612 μs (2308 allocations: 174.11 KiB)
NwTst 5 not ok 47 48 f(-0.779 … 8.944)= 68.2972 616.365 μs (5721 allocations: 648.38 KiB)

superkinkyfunnel: f( x^*=[0] )= 34.871

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 10 not ok 7113 12670 f(-0.0 … 18.495)= 113.376 6.864 ms (86488 allocations: 3.74 MiB)
SAMIN 10 found no 1000000 1000000 f(-0.0 … 0.0)= 34.8707 340.746 ms (7150095 allocations: 456.85 MiB)
Prtcl 10 found no 1000000 11000000 f(-0.0 … 0.0)= 34.8707 6.913 s (85002772 allocations: 4.34 GiB)
GrdDsct 10 found ok 93 922 f(-0.0 … 0.0)= 34.8707 3.825 ms (34127 allocations: 3.30 MiB)
CnjgGrd 10 not no 71 665 f(-7.021 … 9.133)= 120.9157 2.567 ms (22769 allocations: 2.22 MiB)
BFGS 10 not ok 19 140 f(-0.124 … 8.769)= 127.8611 599.526 μs (5407 allocations: 522.52 KiB)
LBFGS 10 not no 14 87 f(-2.664 … 9.49)= 134.6787 385.063 μs (3497 allocations: 330.94 KiB)
Nwt 10 found ok 5 56 f(-0.0 … 0.0)= 34.8707 405.450 μs (3273 allocations: 379.59 KiB)
NwTst 10 not ok 44 45 f(0.0 … 9.724)= 148.4712 1.632 ms (7225 allocations: 1.28 MiB)

wiggler

wiggler: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 not ok 120 228 f([1.838, -0.0])= 0.1041 90.927 μs (1710 allocations: 57.31 KiB)
SAMIN 2 found no 17501 17501 f([-0.0, -0.0])= 0.0001 6.404 ms (164533 allocations: 6.22 MiB)
Prtcl 2 found no 1000000 4000000 f([0.0, -0.0])= 0.0001 2.662 s (42001844 allocations: 1.36 GiB)
GrdDsct 2 not no 12 136 f([-0.0, -0.698])= 0.1197 147.803 μs (2301 allocations: 108.25 KiB)
CnjgGrd 2 not no 11 139 f([-0.0, 3.246])= 0.3986 152.450 μs (2313 allocations: 108.19 KiB)
BFGS 2 not no 10 140 f([-0.0, -9.276])= 0.3754 157.535 μs (2350 allocations: 111.95 KiB)
LBFGS 2 not no 9 131 f([-6.065, -0.0])= 0.3301 155.155 μs (2244 allocations: 106.58 KiB)
Nwt 2 not no 7 94 f([-1.492, -0.0])= 0.127 108.049 μs (1503 allocations: 87.61 KiB)
NwTst 2 not no 57 58 f([-0.0, 3.144])= 0.3442 308.309 μs (5845 allocations: 377.94 KiB)

wiggler: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 5 not ok 1086 2118 f(-0.0 … 22.364)= 0.5353 918.823 μs (15334 allocations: 561.42 KiB)
SAMIN 5 found no 58751 58751 f(-0.0 … 0.0)= 0.0 21.513 ms (512273 allocations: 24.27 MiB)
Prtcl 5 found no 1000000 6000000 f(-0.0 … 0.0)= 0.0 3.600 s (55007628 allocations: 2.19 GiB)
GrdDsct 5 not no 6 121 f(-2.57 … 8.93)= 607.7639 226.611 μs (2684 allocations: 203.72 KiB)
CnjgGrd 5 not no 13 154 f(-1.353 … 8.846)= 947.9015 291.566 μs (3548 allocations: 257.38 KiB)
BFGS 5 not no 16 155 f(-36.712 … -0.0)= 37.7409 306.848 μs (3731 allocations: 269.20 KiB)
LBFGS 5 not no 11 125 f(-0.51 … 8.922)= 385.8313 248.080 μs (2774 allocations: 214.45 KiB)
Nwt 5 found no 24 181 f(-0.0 … 0.0)= 0.0 459.366 μs (6157 allocations: 515.88 KiB)
NwTst 5 not ok 58 59 f(-1.282 … 8.744)= 173.233 780.756 μs (7271 allocations: 824.09 KiB)

wiggler: f( x^*=[0] )= 0.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 10 not ok 2289 5033 f(-0.0 … 12.21)= 236582.9799 2.876 ms (37718 allocations: 1.90 MiB)
SAMIN 10 found no 205751 205751 f(-0.0 … 0.0)= 0.0 78.092 ms (1743046 allocations: 98.36 MiB)
Prtcl 10 found no 1000000 11000000 f(-0.0 … 0.0)= 0.0 6.679 s (85024200 allocations: 4.34 GiB)
GrdDsct 10 not no 16 236 f(-0.708 … 9.696)= 2.835355338993e8 1.005 ms (8027 allocations: 853.94 KiB)
CnjgGrd 10 not no 15 196 f(-0.811 … 9.551)= 377291.2466 780.651 μs (6396 allocations: 683.14 KiB)
BFGS 10 not no 15 533 f(-8.777 … 3.073)= 2900.4907 2.328 ms (18804 allocations: 1.89 MiB)
LBFGS 10 not no 6 177 f(-0.856 … 9.7)= 2.443615166477e8 786.508 μs (5895 allocations: 640.02 KiB)
Nwt 10 not no 26 184 f(-2.346 … 5.519)= 0.0048 1.552 ms (12450 allocations: 1.55 MiB)
NwTst 10 not ok 74 64 f(-0.0 … 9.725)= 5.702770166704e8 2.946 ms (13548 allocations: 2.40 MiB)

f1asym

f1asym: f( x^*=[0] )= 4.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 2 found ok 42 84 f([0.0, -0.0])= 4.0 33.770 μs (662 allocations: 23.13 KiB)
SAMIN 2 found no 18351 18351 f([0.0, 0.0])= 4.0 3.650 ms (128236 allocations: 4.74 MiB)
Prtcl 2 found no 1000000 4000000 f([0.0, 0.0])= 4.0 2.575 s (42000428 allocations: 1.36 GiB)
GrdDsct 2 not no 1000000 2500056 f([-344.806, -0.0])= 6.6933 3.364 s (51501117 allocations: 2.16 GiB)
CnjgGrd 2 found ok 8 74 f([-0.0, 0.0])= 4.0 85.770 μs (1399 allocations: 60.52 KiB)
BFGS 2 found ok 6 79 f([-0.0, -0.0])= 4.0 92.619 μs (1512 allocations: 68.03 KiB)
LBFGS 2 found ok 6 78 f([0.0, 0.0])= 4.0 104.079 μs (1669 allocations: 71.80 KiB)
Nwt 2 found ok 7 16 f([0.0, 0.0])= 4.0 38.061 μs (567 allocations: 32.47 KiB)
NwTst 2 found ok 9 10 f([0.0, -0.0])= 4.0 66.898 μs (953 allocations: 64.28 KiB)

f1asym: f( x^*=[0] )= 32.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 5 found ok 645 1080 f(-0.0 … 0.0)= 32.0 448.484 μs (7741 allocations: 293.16 KiB)
SAMIN 5 found no 54626 54626 f(-0.0 … 0.0)= 32.0 16.036 ms (445940 allocations: 21.03 MiB)
Prtcl 5 found no 1000000 6000000 f(-0.0 … 0.0)= 32.0 3.451 s (55001388 allocations: 2.19 GiB)
GrdDsct 5 not ok 1 3 f(-981343.044 … -301130.74)= 421.6289 8.676 μs (118 allocations: 8.13 KiB)
CnjgGrd 5 not ok 9 146 f(-95239.098 … 0.0)= 150.3192 252.139 μs (3548 allocations: 243.41 KiB)
BFGS 5 not ok 1 3 f(-981343.044 … -301130.74)= 421.6289 9.031 μs (123 allocations: 9.00 KiB)
LBFGS 5 not ok 1 3 f(-981343.044 … -301130.74)= 421.6289 14.378 μs (157 allocations: 12.25 KiB)
Nwt 5 found ok 10 29 f(-0.0 … 0.0)= 32.0 100.201 μs (1568 allocations: 137.84 KiB)
NwTst 5 found ok 20 21 f(-0.0 … 0.0)= 32.0 302.175 μs (3092 allocations: 354.50 KiB)

f1asym: f( x^*=[0] )= 1024.0

Optmzr dim act cnvgd `#`iter `#`calls timing soln
NM 10 found ok 1035 1625 f(-0.0 … 0.0)= 1024.0 844.320 μs (12607 allocations: 511.36 KiB)
SAMIN 10 found no 153751 153751 f(-0.0 … 0.0)= 1024.0 47.975 ms (1237123 allocations: 72.00 MiB)
Prtcl 10 found no 1000000 11000000 f(-0.0 … 0.0)= 1024.0 6.546 s (85005280 allocations: 4.34 GiB)
GrdDsct 10 not ok 1 2 f(-4.726072023758e10 … -1.3605993580249e10)= 177771.9797 10.327 μs (125 allocations: 10.78 KiB)
CnjgGrd 10 not ok 1 3 f(-4.726072022414e9 … -1.360599349305e9)= 177771.9795 9.939 μs (120 allocations: 11.36 KiB)
BFGS 10 not ok 1 2 f(-4.726072023758e10 … -1.3605993580249e10)= 177771.9797 10.574 μs (130 allocations: 12.33 KiB)
LBFGS 10 not ok 1 2 f(-4.726072023758e10 … -1.3605993580249e10)= 177771.9797 15.529 μs (164 allocations: 15.69 KiB)
Nwt 10 found ok 19 69 f(-0.0 … 0.0)= 1024.0 470.498 μs (7025 allocations: 905.23 KiB)
NwTst 10 found ok 29 30 f(-0.0 … 0.0)= 1024.0 1.183 ms (8527 allocations: 1.42 MiB)

limited

limited_domain: f( x^* )= Inf

Optmzr dim wth?
NM 2 FAILED WITH ERROR! DomainError()
SAMIN 2 FAILED WITH ERROR! DomainError()
Prtcl 2 FAILED WITH ERROR! DomainError()
GrdDsct 2 FAILED WITH ERROR! DomainError()
CnjgGrd 2 FAILED WITH ERROR! DomainError()
BFGS 2 FAILED WITH ERROR! DomainError()
LBFGS 2 FAILED WITH ERROR! DomainError()
Nwt 2 FAILED WITH ERROR! DomainError()
NwTst 2 FAILED WITH ERROR! DomainError()

Code

snippet.juliarepl
```using BenchmarkTools
const ITERS=1_000_000
const WITHBOXES=false      ## if (true), dies in various optimization algorithms, like ParticleSwarm and NewtonTrust

using Optim
using DataStructures

###############################################################################################################################
############### Optimizers ####################################################################################################

optimizers= OrderedDict(
SAMIN => "SAMIN",
ParticleSwarm => "Prtcl",
BFGS => "BFGS",
LBFGS => "LBFGS",
Newton => "Nwt",
NewtonTrustRegion => "NwTst"
)

###############
function optall( testfnname, zl::Int, optval=NaN )

testfn= eval(Symbol(testfnname))
allzeros= (isnan(optval)) ? "=[0]" : ""

optval= isnan(optval) ? optval=testfn(fill(0.0, zl)) : optval

println("\n#### \$testfnname:  f( x^*\$allzeros )= \$(round(optval,3))\n")
println("|  **Optmzr**  |  **dim**  |  **act**  |  **cnvgd**  |  **`#`iter**  |  **`#`calls**  |  **timing**  |  **soln**  |")

for optimizer in keys(optimizers)
optme( optimizer, testfn, zl, optval )
end#for

end#function

###############
using Random

function optme(optimizer, testfn, zl::Int, optval )
Random.seed!(0); zin= 10.0*rand( zl )

optname= optimizers[optimizer]

print( @sprintf("| %12.12s", optname), " | ", zl, " |  ")

try
if (optname == "SAMIN")
r= optimize( testfn, zin, fill(–100.0,zl), fill(+100.0,zl), SAMIN(verbosity=0), Optim.Options(iterations=ITERS) );
else
r= (WITHBOXES) ? optimize( testfn, zin, fill(–100.0,zl), fill(+100.0,zl), Fminbox{optimizer}(), iterations=ITERS ) :
optimize( testfn, zin, optimizer(), Optim.Options(iterations=ITERS) );
end#if

xstar= round.(Optim.minimizer(r),3)
xstarrange= "\$(minimum(xstar)) … \$(maximum(xstar))"
xstarrangestring= (length(xstar) >= 5) ? xstarrange : string(xstar)

## generous criterion
print( (Optim.minimum(r) <= optval+1e–3) ? "  found  " : " **not** ", " | ",
Optim.converged(r)?"   ok   " : " **no** ", " |   ", @sprintf("%7d", Optim.iterations(r)), " |  ",  @sprintf("%7d", Optim.f_calls(r)), " | ",
"f(", xstarrangestring, ")= ", round.(Optim.minimum(r),4), "\t |" )
catch y
println("FAILED WITH ERROR! \$y |")
return
end#try

try
if (optname == "SAMIN")
@btime begin
r=optimize( \$testfn, \$zin, fill(–100.0,\$zl), fill(+100.0,\$zl), SAMIN(verbosity=0), Optim.Options(iterations=ITERS) )
end
else
@btime begin
## unstable:  r=optimize( \$testfn, \$zin, fill(–100.0,\$zl), fill(+100.0,\$zl), Fminbox{\$optimizer}(), iterations=ITERS )
r= (WITHBOXES) ? optimize( \$testfn, \$zin, fill(–100.0,zl), fill(+100.0,zl), Fminbox{\$optimizer}(), iterations=ITERS ) :
optimize( \$testfn, \$zin, \$optimizer(), Optim.Options(iterations=ITERS) );
end
end#if
catch y
println(" --- internal error \$y --- ")
end#try
nothing
end;##function##

###############################################################################################################################
############### Test Functions To Be Optimized ################################################################################

# standard test functions, all 2-dim
truncme(x,mini,maxi) = (x)?mini:(x>maxi)?maxi:x

function rosenbrock(x::Vector{Float64})::Float64
assert(length(x) == 2)
(1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2  ## min=[0,0]
end

function bukin(x::Vector{Float64})::Float64  ## min( –10, 1) = 0
assert(length(x) == 2)
xt= [ truncme( x[1], –15.0, –5.0 ), truncme( x[2], –3.0, 3.0 ) ]
100.0*(sqrt( abs( xt[2]–0.01*xt[1]^2 ) ) + 0.01*abs( xt[1]+10.0 )) + exp(5.0+sum(abs.( xt-x )))
end#function

function easom(x::Vector{Float64})::Float64  ## min(pi,pi) = –1
assert( (eltype(x) == Float64) && (length(x) == 2) )
xt= [ truncme( x[1], –100.0, 100.0 ), truncme( x[2], –100.0, 100.0 ) ]
1.0 -cos(xt[1])*cos(xt[2])*exp( - ((xt[1]-pi)^2 + (xt[2]-pi)^2) ) + exp(5.0+sum(abs.( xt-x )))
end#function

# custom functions, all arbitrary dimensions

function myfun(x::Vector{Float64})::Float64   ## min f(2.0,3.0,...,)=0
sum( (x .- [ i+1 for i=1:length(x) ]).^2 ) + prod( (x .- [ i+1 for i=1:length(x) ]) )
end#function

function logmeanvar(x::Vector{Float64})::Float64
am= abs(mean(x)); v= var(x)
log( 1.0 + v + am + sqrt(am*v) )
end#function

function sqatan(x::Vector{Float64})::Float64
onesqatan(x::Float64)= atan( x^2 )^(2+(x.>0)) end#function
prod( (1.0 .+ onesqatan.(x)) )
end#function

function vfun(x::Vector{Float64})::Float64
onevfun(x::Float64) = abs(x)*(1.0+(x>0))
prod( 1.0 .+ onevfun.(x) )
end#function

function vsqrtfun(x::Vector{Float64})::Float64
assert(eltype(x) == Float64)
onesqrtvfun(x::Float64) = sqrt(abs(x)*(1.0+(x>0)))
prod( 1.0 .+ onesqrtvfun.(x) )
end#function

function cliffunnel(x::Vector{Float64})::Float64
onecliffunnel(x::Float64) =  10.01 + ((x<0.5) ? 1.0/(x–1) : –10.0/(x+1))

kc= onecliffunnel.( x .+ (0.5) )
@assert( all( x ->(x >= 0), kc ), "dc received bad input request \$kc at \$x" )
prod( kc )
end#function

function kinkyfunnel(x::Vector{Float64})::Float64
onekinkyfunnel(x::Float64) = (1.01 + ((x<0) ? 1.0/(x–1) : –1.0/(x+1)))*10.0

kc= onekinkyfunnel.(x)
@assert( all( x ->(x >= 0), kc ), "dc received bad input request \$kc at \$x" )
prod( kc )
end#function

function superfunnel(x::Vector{Float64})::Float64
onesuperfunnel(x::Float64) = log(  1.0 / max( abs(x) , 1e–5 ) )

15*length(x) - sum( onesuperfunnel.(x) )
end#function

function wiggler(x::Vector{Float64})::Float64
function onewig(x::Float64)::Float64
xform= sqrt(abs(x*10))
1.5*(1+(x>0)) * (xform + 0.9*sin(xform))+0.01     ## +0.01 to assure positive, or setting any to 0 is optimal
end#function

prod( onewig.(x) )  ## only min at 0, but many saddle points right and many local minima left
end#function

function veryasym(x::Vector{Float64})::Float64
function oneveryasym(x::Float64)::Float64;   ## asymptotes on the left, does not on the right
const xm = –0.42385379906978327137804006262551523367638819718517754082301
const yac = –0.40089940449996159253408195381069828929801311039560261672969
const ysq = –0.17965204298588821036533019640026627767324063903383614189857
-( (x) ? atan(x) : -x^2 - ysq + yac) - 0.22124736151407337 + 1.0
end#function

prod(1.0 .+ oneveryasym.(x) )   ## min is at 0,0,0...
end#function

function limited_domain(x::Vector)::Float64
sum( sqrt.(1 ./ (x+1) .+ (x+1) ) )  ## min is at 1,1,1...
end#function

###############################################################################################################################

println("\n## Sample Maximizations\n")

println("\n### rosenbrock2\n")
optall( "rosenbrock", 2, rosenbrock([1.0,1.0]) )

println("\n### Easom")
optall( "easom", 2, easom([pi,pi]) )

println("\n### Bukin")
optall( "bukin", 2, bukin([–10.0,1.0]) )
println()

println("\n### myfun2")
optall( "myfun", 2, myfun([2.0,3.0]) )  ## deliberately offset solutions for checking

for fnname in [ "sqatan", "vfun", "vsqrtfun", "logmeanvar", "cliffunnel", "kinkyfunnel", "superfunnel", "wiggler", "veryasym" ]
println("\n### \$fnname")
for d in [2,5,10]
optall( fnname, d )
end#for
println()
end#for

println("\n### limited\n"); optall( limited_domain, 2, limited_domain(fill(0.0, 2)) )
println()```