AdamOptimizer 问题

Question

我正在使用一个简单的网络，我正在尝试使用 AdamOptimizer 来 最小化 Q 学习上下文中的损失 。

这里是代码：

### DATASET IMPORT
from DataSet import *

### NETWORK
state_size      = STATE_SIZE
stack_size      = STACK_SIZE
action_size     = ACTION_SIZE
learning_rate   = LEARNING_RATE
hidden_tensors  = HIDDEN_TENSORS
gamma           = GAMMA

import tensorflow as tf
import numpy as np

class NNetwork:       
    def __init__(self, name='NNetwork'):

        # Initialisations
        self.state_size     = state_size
        self.action_size    = action_size
        self.model          = tf.keras.models.Sequential()
        self.optimizer      = tf.keras.optimizers.Adam(learning_rate)

        # Network shaping
        self.model.add(tf.keras.layers.Dense(self.state_size,   activation='relu',      kernel_initializer='glorot_uniform'))
        self.model.add(tf.keras.layers.Dense(hidden_tensors,    activation='relu',      kernel_initializer='glorot_uniform'))
        self.model.add(tf.keras.layers.Dense(action_size,       activation='linear',    kernel_initializer='glorot_uniform'))

    # Prediction function (return Q_values)
    def get_outputs(self, inputs):
        inputs = tf.convert_to_tensor(inputs, dtype=tf.float32)
        return self.model.predict(inputs)

    # Optimization of the network
    def optimize(self, state, action, reward, next_state):
        next_Q_values   = self.get_outputs(next_state)
        target_Q        = reward + gamma * np.max(next_Q_values)
        curent_Q        = tf.reduce_sum(tf.multiply(self.get_outputs(state), action))
        loss           = tf.square(target_Q - curent_Q)
        self.optimizer.minimize(tf.convert_to_tensor(loss), self.model.trainable_variables)



B = NNetwork('b')
print(B.get_outputs([[0.12, 0.59]]))

B.optimize([[0.12, 0.59]], [1, 0, 0, 0, 0, 0, 0], 100000000, [[0.13, 0.58]])
print(B.get_outputs([[0.12, 0.59]]))

所以我的问题是：

当我执行这段代码时，我得到了这个：

[[-0.00105272 0.02356465 -0.01908724 -0.03868931 0.01585
0.02427034 0.00203115]] Traceback (most recent call last): File ".\DQNet.py", line 69, in B.optimize([[0.12, 0.59]], [1, 0, 0, 0, 0, 0, 0], 100000000, [[0.13, 0.58]]) File ".\DQNet.py", line 62, in optimize tf.keras.optimizers.Adam(learning_rate).minimize(tf.convert_to_tensor(10), self.model.trainable_variables) File "C:\Users\Odeven poste 1\Documents[Python-3.6.8\python-3.6.8.amd64\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py", line 296, in minimize loss, var_list=var_list, grad_loss=grad_loss) File "C:\Users\Odeven poste 1\Documents[Python-3.6.8\python-3.6.8.amd64\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py", line 328, in _compute_gradients loss_value = loss() TypeError: 'tensorflow.python.framework.ops.EagerTensor' object is not callable

这意味着我的网络可以正常工作，因为我得到了 Q 值，但是当我尝试调用我的 'optimize' 函数时，我在线路上遇到了一个错误：

self.optimizer.minimize(tf.convert_to_tensor(loss), self.model.trainable_variables)

而且我真的不明白为什么会出现此错误：

'tensorflow.python.framework.ops.EagerTensor' object is not callable

因为我很确定我必须给最小化函数的 'loss' 参数应该是张量...

Answer 1

在 TF2 中，最小化方法的 loss 参数必须是 Python 可调用的。

因此，您可以将损失定义更改为：

def loss():
    return tf.square(target_Q - curent_Q)

并在不将其转换为 Tensor 的情况下使用它：

self.optimizer.minimize(loss, self.model.trainable_variables)

AdamOptimizer 问题

Issue with AdamOptimizer

python-3.x

tensorflow

tensorflow2.0