为 word2vec 梯度下降实现 numba 但得到 LoweringError

Question

我是运行 word2vec 的梯度下降，我想实施 numba 来加速训练。

编辑：看来真正的错误是这个

NotImplementedError: unsupported nested memory-managed object

这是后续错误：

raise NotImplementedError("%s: %s" % (root_type, str(e)))

numba.errors.LoweringError: Failed at nopython (nopython mode backend)
reflected list(reflected list(int64)): unsupported nested memory-managed object
File "test.py", line 36
[1] During: lowering "negative_indices = arg(6, name=negative_indices)" at test.py (36)

我搜索了 numba documentation 并用谷歌搜索了这个错误，但没有成功。

这是一个可复制的代码片段：

import numpy as np
import random
from numba import jit

random.seed(10)
np.random.seed(10)

@jit(nopython=True)
def sigmoid(x):
    return float(1)/(1+np.exp(-x))

@jit(nopython=True)
def listsum(list1):
    ret=0
    for i in list1:
        ret += i
    return ret


num_samples = 2
learning_rate = 0.05
center_token = 50
hidden_size = 100
sequence_chars = [2000, 1500, 400, 600]
W1 = np.random.uniform(-.5, .5, size=(11000, hidden_size))
W2 = np.random.uniform(-.5, .5, size=(11000, hidden_size))
negative_indices = [[800,1000], [777,950], [650,300], [10000,9999]]

@jit(nopython=True)
def performDescent(num_samples, learning_rate, center_token, sequence_chars,W1,W2,negative_indices):
    nll_new = 0
    neg_idx = 0
    for k in range(0, len(sequence_chars)):
        w_c = sequence_chars[k]
        W_neg = negative_indices[k]
        w_j = [w_c] + W_neg
        t_j = [1] + [0]*len(W_neg)
        h = W1[center_token]

        update_i = np.zeros((hidden_size,len(w_j)))
        for i in range(0,len(w_j)):
            v_j = W2[w_j[i]]
            update_i[:,i] = (sigmoid(np.dot(v_j.T,h))-t_j[i])*v_j
            W2[w_j[i]] = v_j - learning_rate*(sigmoid(np.dot(v_j.T,h))-t_j[i])*h #creates v_j_new
        W1[center_token] = h - learning_rate*np.sum(update_i, axis=1)

        update_nll = []
        for i in range(1,len(w_j)):
            update_nll.append(np.log(sigmoid(-np.dot(W2[w_j[i]].T,h))))  #h is updated in memory
        nll = -np.log(sigmoid(np.dot(W2[w_j[0]].T,h))) - listsum(update_nll)
        print("nll:",nll)
        nll_new += nll
    return [nll_new]

performDescent(num_samples, learning_rate, center_token, sequence_chars,W1,W2,negative_indices)

我不明白为什么 negative_indices 会出现问题。

Answer 1

正如错误消息所暗示的，numba 中的列表仅得到部分支持。它们不能包含 "memory-managed" 对象，这意味着它们只能包含标量、原始类型 - 例如：

@njit
def list_first(l):
    return l[0]

list_first([1, 2, 3])
# Out[65]: 1

list_first([[1], [2]])
# LoweringError: Failed at nopython (nopython mode backend)
# reflected list(reflected list(int64)): unsupported nested memory-managed object

假设您的示例具有代表性，在您使用列表的地方似乎没有必要，即使支持也会损害性能，因为您提前知道分配大小。

这是 numba 可以处理的潜在重构。

sequence_chars = np.array([2000, 1500, 400, 600], dtype=np.int64)
negative_indices = np.array([[800,1000], [777,950], [650,300], [10000,9999]], dtype=np.int64)

@jit(nopython=True)
def performDescent2(num_samples, learning_rate, center_token, sequence_chars, W1, W2 ,negative_indices):
    nll_new = 0
    neg_idx = 0

    neg_ind_length = len(negative_indices[0])
    w_j = np.empty(neg_ind_length + 1, dtype=np.int64)
    t_j = np.zeros(neg_ind_length + 1, dtype=np.int64)
    t_j[0] = 1

    for k in range(0, len(sequence_chars)):
        w_j[0] = sequence_chars[k]
        w_j[1:] = negative_indices[k]

        h = W1[center_token]

        update_i = np.zeros((hidden_size,len(w_j)))
        for i in range(0,len(w_j)):
            v_j = W2[w_j[i]]
            update_i[:,i] = (sigmoid(np.dot(v_j.T, h)) - t_j[i]) * v_j
            W2[w_j[i]] = v_j - learning_rate * (sigmoid(np.dot(v_j.T, h)) - t_j[i]) * h #creates v_j_new
        W1[center_token] = h - learning_rate * np.sum(update_i, axis=1)

        update_nll = np.zeros(len(w_j))

        for i in range(1, len(w_j)):
            update_nll[i-1] = np.log(sigmoid(-np.dot(W2[w_j[i]].T, h)))  #h is updated in memory
        nll = -np.log(sigmoid(np.dot(W2[w_j[0]].T,h))) - update_nll.sum()
        print("nll:",nll)
        nll_new += nll
    return nll_new

为 word2vec 梯度下降实现 numba 但得到 LoweringError

Implementing numba for word2vec gradient descent but getting LoweringError

python

gradient-descent

numba

word2vec