如何从 python 中已知似然函数的最大似然估计中获取参数误差？

Question

我有一些数据，想要拟合给定的心理测量函数 p。我对拟合参数和错误也很感兴趣。通过使用 scipy 包中的 curve_fit 函数的 'classical' 方法，很容易获得 p 的参数和错误。但是，我想使用最大似然估计 (MLE) 来做同样的事情。从输出和图中，您可以看到这两种方法提供的参数略有不同。实施 MLE 不是问题，但我不知道如何使用此方法获取错误。有没有简单的方法来获得它们？我的似然函数 L 是：我无法调整此处描述的代码 http://rlhick.people.wm.edu/posts/estimating-custom-mle.html 但这可能是一个解决方案。我该如何实施？或者这个还有别的办法吗？

这里使用 scipy 统计模型安装了一个类似的函数：https://stats.stackexchange.com/questions/66199/maximum-likelihood-curve-model-fitting-in-python。但是参数的误差也不计算。

负对数似然函数是正确的，因为它提供了正确的参数，但我想知道这个函数是否依赖于 y 数据？负对数似然函数 l 显然是 l = -ln(L)。这是我的代码：

#!/usr/bin/env python
# -*- coding: utf-8 -*- 

## libary
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from scipy.optimize import minimize


def p(x,x50,s50):
    """return y value of psychometric function p"""
    return 1./(1+np.exp(4.*s50*(x50-x)))

def initialparams(x,y):
    """return initial fit parameters for function p with given dataset"""
    midpoint = np.mean(x)
    slope = (np.max(y)-np.min(y))/(np.max(x)-np.min(x))
    return [midpoint, slope]

def cfit_error(pcov):
    """return errors of fir from covariance matrix"""
    return np.sqrt(np.diag(pcov))

def neg_loglike(params):
    """analytical negative log likelihood function. This function is dependend on the dataset (x and y) and the two parameters x50 and s50."""
    x50 = params[0]
    s50 = params[1]
    i = len(xdata)
    prod = 1.
    for i in range(i):
        #print prod
        prod *= p(xdata[i],x50,s50)**(ydata[i]*5) * (1-p(xdata[i],x50,s50))**((1.-ydata[i])*5)
    return -np.log(prod)


xdata = [0.,-7.5,-9.,-13.500001,-12.436171,-16.208617,-13.533123,-12.998025,-13.377527,-12.570075,-13.320075,-13.070075,-11.820075,-12.070075,-12.820075,-13.070075,-12.320075,-12.570075,-11.320075,-12.070075]
ydata = [1.,0.6,0.8,0.4,1.,0.,0.4,0.6,0.2,0.8,0.4,0.,0.6,0.8,0.6,0.2,0.6,0.,0.8,0.6]

intparams = initialparams(xdata, ydata)## guess some initial parameters


## normal curve fit using least squares algorithm
popt, pcov = curve_fit(p, xdata, ydata, p0=intparams)
print('scipy.optimize.curve_fit:')
print('x50 = {:f} +- {:f}'.format(popt[0], cfit_error(pcov)[0]))
print('s50 = {:f} +- {:f}\n'.format(popt[1], cfit_error(pcov)[1]))



## fitting using maximum likelihood estimation
results = minimize(neg_loglike, initialparams(xdata,ydata), method='Nelder-Mead')
print('MLE with self defined likelihood-function:')
print('x50 = {:f}'.format(results.x[0]))
print('s50 = {:f}'.format(results.x[1]))
#print results


## ploting the data and results
xfit = np.arange(-20,1,0.1)

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(xdata, ydata, 'xb', label='measured data')
ax.plot(xfit, p(xfit, *popt), '-r', label='curve fit')
ax.plot(xfit, p(xfit, *results.x), '-g', label='MLE')
plt.legend()
plt.show()

输出为：

scipy.optimize.curve_fit:
x50 = -12.681586 +- 0.252561
s50 = 0.264371 +- 0.117911

MLE with self defined likelihood-function:
x50 = -12.406544
s50 = 0.107389

拟合和测量数据都可以在这里看到：我的 Python Debian Stretch 版本是 2.7。谢谢你的帮助。

Answer 1

Rob Hicks (http://rlhick.people.wm.edu/posts/estimating-custom-mle.html) 描述的方法终于成功了。安装numdifftools后，我可以从hessian矩阵计算估计参数的误差。

使用 su 权限在 Linux 上安装 numdifftools：

apt-get install python-pip
pip install numdifftools

此处是我的程序的完整代码示例：

#!/usr/bin/env python
# -*- coding: utf-8 -*- 

## libary
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from scipy.optimize import minimize
import numdifftools as ndt



def p(x,x50,s50):
    """return y value of psychometric function p"""
    return 1./(1+np.exp(4.*s50*(x50-x)))

def initialparams(x,y):
    """return initial fit parameters for function p with given dataset"""
    midpoint = np.mean(x)
    slope = (np.max(y)-np.min(y))/(np.max(x)-np.min(x))
    return [midpoint, slope]

def cfit_error(pcov):
    """return errors of fir from covariance matrix"""
    return np.sqrt(np.diag(pcov))

def neg_loglike(params):
    """analytical negative log likelihood function. This function is dependend on the dataset (x and y) and the two parameters x50 and s50."""
    x50 = params[0]
    s50 = params[1]
    i = len(xdata)
    prod = 1.
    for i in range(i):
        #print prod
        prod *= p(xdata[i],x50,s50)**(ydata[i]*5) * (1-p(xdata[i],x50,s50))**((1.-ydata[i])*5)
    return -np.log(prod)


xdata = [0.,-7.5,-9.,-13.500001,-12.436171,-16.208617,-13.533123,-12.998025,-13.377527,-12.570075,-13.320075,-13.070075,-11.820075,-12.070075,-12.820075,-13.070075,-12.320075,-12.570075,-11.320075,-12.070075]
ydata = [1.,0.6,0.8,0.4,1.,0.,0.4,0.6,0.2,0.8,0.4,0.,0.6,0.8,0.6,0.2,0.6,0.,0.8,0.6]



intparams = initialparams(xdata, ydata)## guess some initial parameters


## normal curve fit using least squares algorithm
popt, pcov = curve_fit(p, xdata, ydata, p0=intparams)
print('scipy.optimize.curve_fit:')
print('x50 = {:f} +- {:f}'.format(popt[0], cfit_error(pcov)[0]))
print('s50 = {:f} +- {:f}\n'.format(popt[1], cfit_error(pcov)[1]))



## fitting using maximum likelihood estimation
results = minimize(neg_loglike, initialparams(xdata,ydata), method='Nelder-Mead')
## calculating errors from hessian matrix using numdifftools
Hfun = ndt.Hessian(neg_loglike, full_output=True)
hessian_ndt, info = Hfun(results.x)
se = np.sqrt(np.diag(np.linalg.inv(hessian_ndt)))

print('MLE with self defined likelihood-function:')
print('x50 = {:f} +- {:f}'.format(results.x[0], se[0]))
print('s50 = {:f} +- {:f}'.format(results.x[1], se[1]))

生成以下输出：

scipy.optimize.curve_fit:
x50 = -18.702375 +- 1.246728
s50 = 0.063620 +- 0.041207

MLE with self defined likelihood-function:
x50 = -18.572181 +- 0.779847
s50 = 0.078935 +- 0.028783

然而，在使用 numdifftools 计算 hessian 矩阵时会出现一些运行时错误。有一些被零除。这可能是因为我自定义的 neg_loglike 函数。最后有一些错误的结果。使用"Extending Statsmodels"的方法可能更优雅，但我想不通。

如何从 python 中已知似然函数的最大似然估计中获取参数误差？

How to get errors of parameters from maximum likelihood estimation with known likelihood function in python?

python

statistics

statsmodels

mle

function-fitting