lmfit minimize fails with ValueError: array is too big

Question

我正在尝试使用 "brute" 方法来最小化 20 个变量的函数。它因神秘错误而失败。这是完整的代码：

import random
import numpy as np
import lmfit

def progress_update(params, iter, resid, *args, **kws):
    pass
    #print(resid)

def score(params, data = None):
    parvals = params.valuesdict()
    M = data
    X_params = []
    Y_params = []
    for i in range(M.shape[0]):
        X_params.append(parvals['x'+str(i)])
    for j in range(M.shape[1]):
        Y_params.append(parvals['y'+str(i)])
    return diff(M, X_params, Y_params)


def diff(M, X_params, Y_params):
    total = 0
    for i in range(M.shape[0]):
        for j in range(M.shape[1]):
            total += abs(M[i,j] - (X_params[i] - Y_params[j])**2)
    return total

dim = 10
random.seed(0)
M = np.empty((dim, dim))

for i in range(M.shape[0]):
    for j in range(M.shape[1]):
        M[i,j] = i*random.random()+j**2

params = lmfit.Parameters()
for i in range(M.shape[0]):
    params.add('x'+str(i), value=random.random()*10, min=0, max=10)
for j in range(M.shape[1]):
    params.add('y'+str(j), value=random.random()*10, min=0, max=10)

result = lmfit.minimize(score, params, method='brute', kws={'data': M},  iter_cb=progress_update)

然而，这失败了：

ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.

是什么导致了这个问题？

Answer 1

"What is causing this problem"

Math

你不能暴力破解高维问题，因为暴力破解方法 require exponential work（时间和内存，如果实现得天真）。

更直接地说，lmfit 在底层使用 numpy (*)，它具有可以分配的数据量的最大大小。您的初始数据结构不是太大 (10x10)，它是导致问题的蛮力所需的组合 table。

如果您愿意破解实现，您可以切换到稀疏内存结构。但这并不能解决数学问题。

关于高维优化

尝试不同的最小化器，但要注意：在高维中全局最小化是非常困难的 space。 "Local minima" 方法如 fixed point / gradient descent 可能更有效率。

我不想悲观，但一般情况下，高级优化非常困难，恐怕超出了 SO 问题的范围。 Here is a survey.

实用的选择

梯度下降是 supported a little in sklearn but more for machine learning than general optimization; scipy actually has pretty good optimization coverage, and great documentation. I'd start there. It's possible to do gradient descent there too，但不是必需的。

从 scipy 关于无约束最小化的文档中，您有很多选择：

Method Nelder-Mead uses the Simplex algorithm [], []. This algorithm is robust in many applications. However, if numerical computation of derivative can be trusted, other algorithms using the first and/or second derivatives information might be preferred for their better performance in general.

Method Powell is a modification of Powell’s method [], [] which is a conjugate direction method. It performs sequential one-dimensional minimizations along each vector of the directions set (direc field in options and info), which is updated at each iteration of the main minimization loop. The function need not be differentiable, and no derivatives are taken.

还有更多基于导数的方法可用。（一般来说，当你有可用的衍生信息时，你会做得更好。）

Footnotes/Looking 源代码

(*) 这里的实际错误 is thrown，基于您的 numpy 实现。引用:

`if (npy_mul_with_overflow_intp(&nbytes, nbytes, dim)) {
    PyErr_SetString(PyExc_ValueError,
        "array is too big; `arr.size * arr.dtype.itemsize` "
        "is larger than the maximum possible size.");
    Py_DECREF(descr);
    return NULL;`

lmfit minimize fails with ValueError: array is too big

lmfit minimize fails with ValueError: array is too big

python

optimization

lmfit