如何修复 MatMul Op 的 float64 类型与 float32 类型不匹配 TypeError？

Question

我正在尝试将神经网络权重保存到一个文件中，然后通过初始化网络而不是随机初始化来恢复这些权重。我的代码适用于随机初始化。但是，当我从文件初始化权重时，它向我显示错误 TypeError: Input 'b' of 'MatMul' Op has type float64 that does not match type float32 of argument 'a'. 我不知道如何解决这个问题 issue.Here 是我的代码：

模型初始化

# Parameters
training_epochs = 5
batch_size = 64
display_step = 5
batch = tf.Variable(0, trainable=False)
regualarization =  0.008

# Network Parameters
n_hidden_1 = 300 # 1st layer num features
n_hidden_2 = 250 # 2nd layer num features

n_input = model.layer1_size # Vector input (sentence shape: 30*10)
n_classes = 12 # Sentence Category detection total classes (0-11 categories)

#History storing variables for plots
loss_history = []
train_acc_history = []
val_acc_history = []

# tf Graph input
x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes])

模型参数

#loading Weights
def weight_variable(fan_in, fan_out, filename):
    stddev = np.sqrt(2.0/fan_in)
    if (filename == ""):
        initial  = tf.random_normal([fan_in,fan_out], stddev=stddev)
    else:
        initial  = np.loadtxt(filename)
    print initial.shape
    return tf.Variable(initial)

#loading Biases
def bias_variable(shape, filename):
    if (filename == ""):
     initial = tf.constant(0.1, shape=shape)
    else:
     initial  = np.loadtxt(filename)  
    print initial.shape
    return tf.Variable(initial)

# Create model
def multilayer_perceptron(_X, _weights, _biases):
    layer_1 = tf.nn.relu(tf.add(tf.matmul(_X, _weights['h1']), _biases['b1'])) 
    layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, _weights['h2']), _biases['b2'])) 
    return tf.matmul(layer_2, weights['out']) + biases['out']  

# Store layers weight & bias
weights = {
'h1':  w2v_utils.weight_variable(n_input, n_hidden_1,    filename="weights_h1.txt"),
'h2':  w2v_utils.weight_variable(n_hidden_1, n_hidden_2, filename="weights_h2.txt"),
'out': w2v_utils.weight_variable(n_hidden_2, n_classes,  filename="weights_out.txt") 
}

 biases = {
'b1': w2v_utils.bias_variable([n_hidden_1], filename="biases_b1.txt"),
'b2': w2v_utils.bias_variable([n_hidden_2], filename="biases_b2.txt"),
'out': w2v_utils.bias_variable([n_classes], filename="biases_out.txt")
}

# Define loss and optimizer
#learning rate
# Optimizer: set up a variable that's incremented once per batch and
# controls the learning rate decay.
learning_rate = tf.train.exponential_decay(
    0.02*0.01,           # Base learning rate. #0.002
    batch * batch_size,  # Current index into the dataset.
    X_train.shape[0],    # Decay step.
    0.96,                # Decay rate.
    staircase=True)


# Construct model
pred = tf.nn.relu(multilayer_perceptron(x, weights, biases))

#L2 regularization
l2_loss = tf.add_n([tf.nn.l2_loss(v) for v in tf.trainable_variables()])

#Softmax loss
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y)) 

#Total_cost
cost = cost+ (regualarization*0.5*l2_loss)

# Adam Optimizer
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost,global_step=batch)


# Add ops to save and restore all the variables.
saver = tf.train.Saver()

# Initializing the variables
init = tf.initialize_all_variables()

print "Network Initialized!"

错误详情

Answer 1

tf.matmul() op does not perform automatic type conversions, so both of its inputs must have the same element type. The error message you are seeing indicates that you have a call to tf.matmul() where the first argument has type tf.float32, and the second argument has type tf.float64. You must convert one of the inputs to match the other, for example using tf.cast(x, tf.float32).

查看您的代码，我没有在任何地方看到显式创建 tf.float64 张量（TensorFlow Python [= 中浮点值的默认 dtype 30=]——例如 tf.constant(37.0)——是 tf.float32）。我猜这些错误是由 np.loadtxt(filename) 调用引起的，它可能正在加载一个 np.float64 数组。您可以显式更改它们以加载 np.float32 数组（转换为 tf.float32 张量），如下所示：

initial = np.loadtxt(filename).astype(np.float32)

Answer 2

虽然这是一个老问题，但我希望你能包括我遇到过同样的问题。我使用 dtype=tf.float64 解决了这个问题，用于参数初始化以及创建 X 和 Y 占位符。

这是我的代码截图。

X = tf.placeholder(shape=[n_x, None],dtype=tf.float64)
Y = tf.placeholder(shape=[n_y, None],dtype=tf.float64)

和

parameters['W' + str(l)] = tf.get_variable('W' + str(l), [layers_dims[l],layers_dims[l-1]],dtype=tf.float64, initializer = tf.contrib.layers.xavier_initializer(seed = 1))
parameters['b' + str(l)] = tf.get_variable('b' + str(l), [layers_dims[l],1],dtype=tf.float64, initializer = tf.zeros_initializer())

使用 float64 数据类型声明所有占位符和参数将解决此问题。

Answer 3

对于 Tensorflow 2

您可以投射其中一个张量，例如：

_X = tf.cast(_X, dtype='float64')

如何修复 MatMul Op 的 float64 类型与 float32 类型不匹配 TypeError？

How to fix MatMul Op has type float64 that does not match type float32 TypeError?

python

machine-learning

neural-network

tensorflow