PyTorch：传递 numpy 数组进行权重初始化

Question

我想用np数组初始化RNN的参数。

在下面的例子中，我想将w传递给rnn的参数。我知道pytorch提供了很多像Xavier、uniform等初始化方法，但是有没有办法通过传递numpy数组来初始化参数呢？

import numpy as np
import torch as nn
rng = np.random.RandomState(313)
w = rng.randn(input_size, hidden_size).astype(np.float32)

rnn = nn.RNN(input_size, hidden_size, num_layers)

Answer 1

首先，我们要注意 nn.RNN 有多个权重变量 c.f。 documentation:

Variables:

weight_ih_l[k] – the learnable input-hidden weights of the k-th layer, of shape (hidden_size * input_size) for k = 0. Otherwise, the shape is (hidden_size * hidden_size)

weight_hh_l[k] – the learnable hidden-hidden weights of the k-th layer, of shape (hidden_size * hidden_size)

bias_ih_l[k] – the learnable input-hidden bias of the k-th layer, of shape (hidden_size)

bias_hh_l[k] – the learnable hidden-hidden bias of the k-th layer, of shape (hidden_size)

现在，这些变量（Parameter 个实例）中的每一个都是您的 nn.RNN 个实例的属性。您可以通过两种方式访问和编辑它们，如下所示：

解决方案 1：按名称（rnn.weight_hh_lK、rnn.weight_ih_lK 等）访问所有 RNN Parameter 属性：

import torch
from torch import nn
import numpy as np

input_size, hidden_size, num_layers = 3, 4, 2
use_bias = True
rng = np.random.RandomState(313)

rnn = nn.RNN(input_size, hidden_size, num_layers, bias=use_bias)

def set_nn_parameter_data(layer, parameter_name, new_data):
    param = getattr(layer, parameter_name)
    param.data = new_data

for i in range(num_layers):
    weights_hh_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    weights_ih_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    set_nn_parameter_data(rnn, "weight_hh_l{}".format(i), 
                          torch.from_numpy(weights_hh_layer_i))
    set_nn_parameter_data(rnn, "weight_ih_l{}".format(i), 
                          torch.from_numpy(weights_ih_layer_i))

    if use_bias:
        bias_hh_layer_i = rng.randn(hidden_size).astype(np.float32)
        bias_ih_layer_i = rng.randn(hidden_size).astype(np.float32)
        set_nn_parameter_data(rnn, "bias_hh_l{}".format(i), 
                              torch.from_numpy(bias_hh_layer_i))
        set_nn_parameter_data(rnn, "bias_ih_l{}".format(i), 
                              torch.from_numpy(bias_ih_layer_i))

解决方案2：通过rnn.all_weights列表属性访问所有RNN Parameter属性：

import torch
from torch import nn
import numpy as np

input_size, hidden_size, num_layers = 3, 4, 2
use_bias = True
rng = np.random.RandomState(313)

rnn = nn.RNN(input_size, hidden_size, num_layers, bias=use_bias)

for i in range(num_layers):
    weights_hh_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    weights_ih_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    rnn.all_weights[i][0].data = torch.from_numpy(weights_ih_layer_i)
    rnn.all_weights[i][1].data = torch.from_numpy(weights_hh_layer_i)

    if use_bias:
        bias_hh_layer_i = rng.randn(hidden_size).astype(np.float32)
        bias_ih_layer_i = rng.randn(hidden_size).astype(np.float32)
        rnn.all_weights[i][2].data = torch.from_numpy(bias_ih_layer_i)
        rnn.all_weights[i][3].data = torch.from_numpy(bias_hh_layer_i)

Answer 2

回答的很详细，我再补充一句。一个nn.Module的参数是Tensors（以前是autograd变量，which is deperecated in Pytorch 0.4）。因此，本质上您需要使用 torch.from_numpy() 方法将 Numpy 数组转换为 Tensor，然后使用它们来初始化 nn.Module 参数。

PyTorch：传递 numpy 数组进行权重初始化

PyTorch: passing numpy array for weight initialization

python

numpy

initialization

pytorch

rnn