如何在 Django 中训练 Keras 模型：弱引用 'gevent._local.local' 对象错误

Question

在我的 Django 应用程序中，我允许用户使用 Tensorflow Hub 训练他们自己的二元分类模型。训练任务如下所示：

import tensorflow as tf
import tensorflow_hub as hub
import numpy as np

def classification_model_train_binary():
    x_train = np.array(["Some test text",
                        "Testing this text",
                        "This is relevant to my test",
                        "Cows don't fly",
                        "One two three",
                        "some text"])
    y_train = np.array([1, 1, 1, 0, 0, 0])

    model = "https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1"

    def my_iterator(x, y):
        while True:
            for _x, _y in zip(x, y):
                yield np.array([_x]), np.array([_y])

    hub_layer = hub.KerasLayer(model, output_shape=[20], input_shape=[],
                               dtype=tf.string, trainable=True)
    model = tf.keras.Sequential()
    model.add(hub_layer)
    model.add(tf.keras.layers.Dense(16, activation='relu', input_shape=[20]))
    model.add(tf.keras.layers.Dense(1, activation='sigmoid'))

    model.summary()

    model.compile(optimizer='adam',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])

    model.fit_generator(my_iterator(x_train, y_train), epochs=5, steps_per_epoch=len(x_train))
    print("THE END")

我运行上面的代码有以下几种方式（所有测试都使用相同的虚拟环境）：

作为独立脚本：成功
从 Django 视图：错误（见下文）
来自 Django Shell：错误（见下文）
从 Jupyter Notebook 使用：python manage.py shell_plus --notebook：成功

错误

Exception Type: TypeError at /classify/classifier/binary/
Exception Value: cannot create weak reference to 'gevent._local.local' object

为什么我能够运行作为一个独立的脚本并通过 jupyter notebook（使用 Django shell！）而不是使用独立的 Django shell？我需要做什么才能在独立的 Django shell 中完成这项工作？

回溯:

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\backend.py" in eager_learning_phase_scope
  425.     _GRAPH_LEARNING_PHASES[_DUMMY_EAGER_GRAPH] = value

File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\weakref.py" in __setitem__
  409.         self.data[ref(key, self._remove)] = value

During handling of the above exception (cannot create weak reference to 'gevent._local.local' object), another exception occurred:

File "C:\Users\me\myproject\env\lib\site-packages\django\core\handlers\exception.py" in inner
  34.             response = get_response(request)

File "C:\Users\me\myproject\env\lib\site-packages\django\core\handlers\base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "C:\Users\me\myproject\env\lib\site-packages\django\core\handlers\base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\contextlib.py" in inner
  74.                 return func(*args, **kwds)

File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\contextlib.py" in inner
  74.                 return func(*args, **kwds)

File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\contextlib.py" in inner
  74.                 return func(*args, **kwds)

File "C:\Users\me\myproject\env\lib\site-packages\django\contrib\auth\decorators.py" in _wrapped_view
  21.                 return view_func(request, *args, **kwargs)

File "C:\Users\me\myproject\classify\views.py" in binary
  169.                 classify_tasks.classification_model_train_binary(workspace_id, dataset_id, classifier_id)

File "C:\Users\me\myproject\classify\tasks.py" in classification_model_train_binary
  86.     model.fit_generator(my_iterator(x_train, y_train), epochs=5, steps_per_epoch=len(x_train), verbose=3)

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\engine\training.py" in fit_generator
  1297.         steps_name='steps_per_epoch')

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\engine\training_generator.py" in model_iteration
  265.       batch_outs = batch_function(*batch_data)

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\engine\training.py" in train_on_batch
  973.           class_weight=class_weight, reset_metrics=reset_metrics)

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\engine\training_v2_utils.py" in train_on_batch
  264.       output_loss_metrics=model._output_loss_metrics)

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\engine\training_eager.py" in train_on_batch
  311.           output_loss_metrics=output_loss_metrics))

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\engine\training_eager.py" in _process_single_batch
  241.   with backend.eager_learning_phase_scope(1 if training else 0):

File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\contextlib.py" in __enter__
  112.             return next(self.gen)

File "C:\Users\me\myproject\env\lib\site-packages\tensorflow_core\python\keras\backend.py" in eager_learning_phase_scope
  432.       del _GRAPH_LEARNING_PHASES[_DUMMY_EAGER_GRAPH]

File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\weakref.py" in __delitem__
  393.         del self.data[ref(key)]

Exception Type: TypeError at /classify/classifier/binary/
Exception Value: cannot create weak reference to 'gevent._local.local' object

Answer 1

看起来 可能是环境配置问题 - 你在 django 项目中有 virtualenv 而 Visual Studio 可能正在使用它自己的。

尝试在项目目录中重新创建虚拟python环境，并在[=25]中重新配置项目设置=] 以确保它使用项目目录中存在的 python 虚拟环境。

(P.S。已尝试运行仅提供的任务 - 在所有情况下都运行正常)

Answer 2

在我的例子中，我在我的 manage.py 文件中运行猴子补丁。

from gevent import monkey
monkey.patch_all()

删除它解决了我的问题。

如何在 Django 中训练 Keras 模型：弱引用 'gevent._local.local' 对象错误

How to train a Keras model in Django: weak reference to 'gevent._local.local' object error

django

python-3.x

keras

tensorflow2.0

tensorflow-hub