对数回归损失的梯度
Gradient for log regression loss
我正在尝试为对数回归编写小批量梯度下降。
给定 numpy 矩阵 X_batch
(形状 (n_samples, n_features)
)和 y_batch
(形状 (n_samples,)
)。
天真的方法是写循环:
def calc_loss_grad(self, X_batch, y_batch):
n_samples, n_features = X_batch.shape
loss_grad = np.zeros((n_features,))
for i in range(n_samples):
sigm = sigmoid(X_batch[i] @ self.weights)
loss_grad += - (y_batch[i] - sigm) * X_batch[i]
return loss_grad
但似乎使用循环不是一个好主意 w.r.t。速度。有什么更好的方法吗?没有循环的纯 numpy?以某种方式重写梯度表达式?
请注意,此算法受内存带宽限制。如果您在更大的上下文(实际应用程序)中对此进行优化,则很可能会有更高的加速比。[=11=]
例子
import numpy as np
#
def sigmoid(x):
return np.exp(-np.logaddexp(0, -x))
def calc_loss_grad_1(weights, X_batch, y_batch):
n_samples, n_features = X_batch.shape
loss_grad = np.zeros((n_features,))
for i in range(n_samples):
sigm = sigmoid(X_batch[i,:] @ weights)
loss_grad += - (y_batch[i] - sigm) * X_batch[i]
return loss_grad
def calc_loss_grad_2(weights, X_batch, y_batch):
sigm =-y_batch+sigmoid(X_batch@weights)
return sigm@X_batch
weights=np.random.rand(n_features)
X_batch=np.random.rand(n_samples,n_features)
y_batch=np.random.rand(n_samples)
#n_samples=int(1e5)
#n_features=int(1e4)
%timeit res=calc_loss_grad_1(weights, X_batch, y_batch)
#1.79 s ± 35.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit res=calc_loss_grad_2(weights, X_batch, y_batch)
#539 ms ± 21.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#n_samples=int(1e3)
#n_features=int(1e2)
%timeit res=calc_loss_grad_1(weights, X_batch, y_batch)
#3.68 ms ± 44.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit res=calc_loss_grad_2(weights, X_batch, y_batch)
#49.1 µs ± 503 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
我正在尝试为对数回归编写小批量梯度下降。
给定 numpy 矩阵 X_batch
(形状 (n_samples, n_features)
)和 y_batch
(形状 (n_samples,)
)。
天真的方法是写循环:
def calc_loss_grad(self, X_batch, y_batch):
n_samples, n_features = X_batch.shape
loss_grad = np.zeros((n_features,))
for i in range(n_samples):
sigm = sigmoid(X_batch[i] @ self.weights)
loss_grad += - (y_batch[i] - sigm) * X_batch[i]
return loss_grad
但似乎使用循环不是一个好主意 w.r.t。速度。有什么更好的方法吗?没有循环的纯 numpy?以某种方式重写梯度表达式?
请注意,此算法受内存带宽限制。如果您在更大的上下文(实际应用程序)中对此进行优化,则很可能会有更高的加速比。[=11=]
例子
import numpy as np
#
def sigmoid(x):
return np.exp(-np.logaddexp(0, -x))
def calc_loss_grad_1(weights, X_batch, y_batch):
n_samples, n_features = X_batch.shape
loss_grad = np.zeros((n_features,))
for i in range(n_samples):
sigm = sigmoid(X_batch[i,:] @ weights)
loss_grad += - (y_batch[i] - sigm) * X_batch[i]
return loss_grad
def calc_loss_grad_2(weights, X_batch, y_batch):
sigm =-y_batch+sigmoid(X_batch@weights)
return sigm@X_batch
weights=np.random.rand(n_features)
X_batch=np.random.rand(n_samples,n_features)
y_batch=np.random.rand(n_samples)
#n_samples=int(1e5)
#n_features=int(1e4)
%timeit res=calc_loss_grad_1(weights, X_batch, y_batch)
#1.79 s ± 35.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit res=calc_loss_grad_2(weights, X_batch, y_batch)
#539 ms ± 21.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#n_samples=int(1e3)
#n_features=int(1e2)
%timeit res=calc_loss_grad_1(weights, X_batch, y_batch)
#3.68 ms ± 44.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit res=calc_loss_grad_2(weights, X_batch, y_batch)
#49.1 µs ± 503 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)