Python 运行 Macbook pro m1 max 错误(运行 在 Tensorflow 上)

Python Running Error on Macbook pro m1 max (Running on Tensorflow)

我正尝试在我的新 macbook pro max M1 芯片上 运行 来自 github binary-bot 的代码:

金属设备设置为:

Apple M1 Max
systemMemory: 32.00 GB
maxCacheSize: 10.67 GB

我收到以下错误。有什么建议吗?

2021-12-19 17:26:25.248041: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-12-19 17:26:25.248181: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
1 Physical GPUs, 1 Logical GPUs
1 Physical GPUs, 1 Logical GPUs
Trying to connect to IqOption
Successfully Connected!
/Users/abdallahmohamed/Downloads/binary-bot-master/training.py:35: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  df = df.drop("future", 1)
/Users/abdallahmohamed/Downloads/binary-bot-master/training.py:35: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  df = df.drop("future", 1)
train data: 836 validation: 68
sells: 418, buys: 418
VALIDATION sells: 34, buys : 34
0.001-5-SEQ-2-40-16-PRED-1639927591
1 Physical GPUs, 1 Logical GPUs
2021-12-19 17:26:32.262259: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Epoch 1/40
2021-12-19 17:26:33.223096: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.592036: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.646351: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.684523: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.786763: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.866171: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.932667: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
53/53 [==============================] - ETA: 0s - loss: 0.8687 - accuracy: 0.52272021-12-19 17:26:35.527128: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:35.652789: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:35.681668: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:35.710022: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 4s 34ms/step - loss: 0.8687 - accuracy: 0.5227 - val_loss: 0.6928 - val_accuracy: 0.5000
Epoch 2/40
53/53 [==============================] - ETA: 0s - loss: 0.7844 - accuracy: 0.5335WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7844 - accuracy: 0.5335 - val_loss: 0.6890 - val_accuracy: 0.5000
Epoch 3/40
51/53 [===========================>..] - ETA: 0s - loss: 0.7257 - accuracy: 0.5588WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7293 - accuracy: 0.5562 - val_loss: 0.6836 - val_accuracy: 0.5735
Epoch 4/40
52/53 [============================>.] - ETA: 0s - loss: 0.7421 - accuracy: 0.5649WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7411 - accuracy: 0.5658 - val_loss: 0.7035 - val_accuracy: 0.4412
Epoch 5/40
52/53 [============================>.] - ETA: 0s - loss: 0.7205 - accuracy: 0.5565WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7193 - accuracy: 0.5586 - val_loss: 0.7327 - val_accuracy: 0.4412
Epoch 6/40
52/53 [============================>.] - ETA: 0s - loss: 0.7233 - accuracy: 0.5637WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7228 - accuracy: 0.5634 - val_loss: 0.7023 - val_accuracy: 0.5441
Epoch 7/40
51/53 [===========================>..] - ETA: 0s - loss: 0.7192 - accuracy: 0.5588WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7187 - accuracy: 0.5586 - val_loss: 0.8523 - val_accuracy: 0.4559
Epoch 8/40
51/53 [===========================>..] - ETA: 0s - loss: 0.7111 - accuracy: 0.5613WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7105 - accuracy: 0.5634 - val_loss: 0.7727 - val_accuracy: 0.4559
Epoch 9/40
53/53 [==============================] - ETA: 0s - loss: 0.7151 - accuracy: 0.5514WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.7151 - accuracy: 0.5514 - val_loss: 0.7105 - val_accuracy: 0.5147
Epoch 10/40
53/53 [==============================] - ETA: 0s - loss: 0.7046 - accuracy: 0.5371WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.7046 - accuracy: 0.5371 - val_loss: 0.6940 - val_accuracy: 0.5588
Epoch 11/40
53/53 [==============================] - ETA: 0s - loss: 0.7064 - accuracy: 0.5455WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.7064 - accuracy: 0.5455 - val_loss: 0.7433 - val_accuracy: 0.3971
Epoch 12/40
51/53 [===========================>..] - ETA: 0s - loss: 0.6991 - accuracy: 0.5784WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.6988 - accuracy: 0.5778 - val_loss: 0.6902 - val_accuracy: 0.5147
Epoch 13/40
52/53 [============================>.] - ETA: 0s - loss: 0.6812 - accuracy: 0.5757WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6818 - accuracy: 0.5754 - val_loss: 0.8100 - val_accuracy: 0.4118
Epoch 14/40
52/53 [============================>.] - ETA: 0s - loss: 0.6876 - accuracy: 0.5673WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6888 - accuracy: 0.5658 - val_loss: 0.7208 - val_accuracy: 0.5294
Epoch 15/40
52/53 [============================>.] - ETA: 0s - loss: 0.6815 - accuracy: 0.5505WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 23ms/step - loss: 0.6809 - accuracy: 0.5502 - val_loss: 0.6965 - val_accuracy: 0.5441
Epoch 16/40
51/53 [===========================>..] - ETA: 0s - loss: 0.6886 - accuracy: 0.5711WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6900 - accuracy: 0.5670 - val_loss: 0.6529 - val_accuracy: 0.6029
Epoch 17/40
53/53 [==============================] - ETA: 0s - loss: 0.6959 - accuracy: 0.5598WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6959 - accuracy: 0.5598 - val_loss: 0.7832 - val_accuracy: 0.4118
Epoch 18/40
52/53 [============================>.] - ETA: 0s - loss: 0.7002 - accuracy: 0.5325WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6999 - accuracy: 0.5335 - val_loss: 0.7270 - val_accuracy: 0.3676
WARNING:tensorflow:Layer lstm_3 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
WARNING:tensorflow:Layer lstm_4 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
WARNING:tensorflow:Layer lstm_5 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
Trying to connect to IqOption
Successfully Connected!
2021-12-19 17:27:01.637577: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:27:01.792362: F tensorflow/core/framework/tensor.cc:681] Check failed: IsAligned() ptr = 0x2d0efcee0
zsh: abort      /Users/abdallahmohamed/miniforge3/envs/env_tensorflow/bin/python 

我删除了存储的模型并保存了一个新模型后,它起作用了

我的感觉是这里有两个明显的问题:

  1. 使用 Metal 引擎在 M1 机器上安装 Tensorflow 的步骤。
  2. 要使用的包的requirements.txt文件。

背景:

在新的 M1 处理器上工作时,由于某些命名约定和版本控制假设往往包含在包 requirement.txt 文件中,因此在安装 M1 之前的 python 包时往往会遇到问题.

另外需要注意的是,M1 处理器仅支持 Python 3.8 及更高版本。 las,Python 3.7 和 python 的所有先前版本都是在 M1 处理器发布之前开发的,并且(显然)没有向后移植关键补丁的计划。

Tensorflow-macos 和 Tensorflow-metal 安装

安装Tensorflow-macosTensorFlow-metal的步骤很详细here,使用mini-forge可以总结如下:

conda create -n tf python=3.8 -y
conda activate tf
conda install -c apple tensorflow-deps -y
# Navigate the issue with conda environments
# built against pre-macOS 11 SDK use (Just in case)
# this solves the Intel / M1 install issues:
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-macos 
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-metal

所以我们可以测试它是否有效:

#!/usr/bin/env bash

PYCMD=$(cat <<EOF
import tensorflow as tf

tf.config.list_physical_devices()
with tf.device('/GPU'):
  a = tf.random.normal(shape=(2,), dtype=tf.float32)
  b = tf.nn.relu(a)

print(a)
print(b)

EOF
)

python3 -c "$PYCMD" 2>/dev/null

我的测试机器的预期输出:

>./val.sh
Metal device set to: Apple M1

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

tf.Tensor([-0.4044231   0.08157467], shape=(2,), dtype=float32)
tf.Tensor([0.         0.08157467], shape=(2,), dtype=float32)

好的,一切(假设)都在工作...

二进制 Bot 安装问题说明:

现在我们来看看 Binary Bot...

首先,我们需要通过 git hub

安装
git clone https://github.com/ItamarRocha/binary-bot

好的,现在在安装之前让我们看一下 requirements.txt 文件...

iqoptionapi @ git+git://github.com/Lu-Yi-Hsun/iqoptionapi.git@e96ba2c5b905a139a4765167b08c5df48cf57773
numpy==1.18.0
pandas==1.1.1
tensorflow==2.3.1
scikit-learn==0.23.2
grpcio==1.24.3

重要的是要注意两个关键问题 ==tensorflow。需求文件中的 == 表示与包版本完全匹配。同时我们还没有安装tensorflow,我们安装了tensorflow-macostensorflow-metal。我们可以检查一下:

>pip show numpy pandas tensorflow scikit-learn grpcio | egrep 'Name:|Version:'
WARNING: Package(s) not found: pandas, scikit-learn, tensorflow
Name: numpy
Version: 1.19.5
Name: grpcio
Version: 1.43.0

因此如果我们 运行:

python3 pip install -r requirements.txt

我们收到大量错误。为什么?好吧,因为 requirements.txt 文件需要 exact == 包匹配而不是兼容的 ~= 包匹配或下载大于或等于规定版本的任何版本>=

为了解决这个问题,我们可以更改为 >=,这(希望)应该允许我们安装与 Apple M1 芯片兼容的软件包,但如果开发人员依赖于旧的假设,我们可能会破坏兼容性包。

此外,requirements.txt 将尝试安装 tensorflow 包,而不是 tensorflow-macosxtensorflow-metal 包,正如我们所看到的那样,当我们 运行 requirements.txt 文件不变。

那么如何导航?

好吧,一种选择是简单地将 == 替换为 >=,并将 tensorflow 替换或删除为 tensorflow-macosx

sed -i.bak 's/==/>=/g;s/tensorflow/tensorflow-macos/g' requirements.txt

导致:

cat requirements.txt
iqoptionapi @ git+git://github.com/Lu-Yi-Hsun/iqoptionapi.git@e96ba2c5b905a139a4765167b08c5df48cf57773
numpy>=1.18.0
pandas>=1.1.1
tensorflow-macos>=2.3.1
scikit-learn>=0.23.2
grpcio>=1.24.3

现在如果我们 运行:

pip install -r requirements.txt

我们安装了最新版本的软件包 >= 导致:

pip show numpy pandas tensorflow-macos tensorflow-metal scikit-learn grpcio | egrep 'Name:|Version:'
Name: numpy
Version: 1.22.0
Name: pandas
Version: 1.3.5
Name: tensorflow-macos
Version: 2.7.0
Name: tensorflow-metal
Version: 0.3.0
Name: scikit-learn
Version: 1.0.2
Name: grpcio
Version: 1.43.0

所以,我们现在应该能够测试一切是否正常...

python3 testing.py
Metal device set to: Apple M1

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2022-01-08 20:40:40.591430: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-01-08 20:40:40.591754: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
1 Physical GPUs, 1 Logical GPUs
1 Physical GPUs, 1 Logical GPUs
Trying to connect to IqOption
Error when trying to connect
<iqoptionapi.stable_api.IQ_Option object at 0x1031074c0>
Retrying

所以包现在安装,所有需要的是将您的名称和密码添加到 iq.py 文件,并检查结果是否符合预期。

  19   if iq == None:
  20       print("Trying to connect to IqOption")
  21       iq=IQ_Option('USERNAME','PASSWORD') # YOU HAVE TO ADD YOUR USERNAME AND PASSWORD
  22       iq.connect()

参考 'deleting the model' 上的观点,这是公认的答案。我推测这是指 binary-bot 目录中 models 目录中的 LSTM-best.model

tree binary-bot                                                  binary-bot
├── README.md
├── __pycache__
│   ├── iq.cpython-38.pyc
│   └── training.cpython-38.pyc
├── imgs
│   ├── mac_forex.jpg
│   └── result_binary.png
├── iq.py
├── models
│   └── LSTM-best.model
├── requirements.txt
├── requirements.txt.bak
├── result_binary.png
├── testing.py
└── training.py

唉,我目前不在允许我注册服务 https://iqoption.net/ 的国家/地区,以进行全面测试,但我的感觉是,事情应该可以达到可以根据以下条件验证结果的程度以上。

希望这能为人们指明正确的方向,让他们思考让 python 软件包与使用 TensorFlow 金属插件的 Apple-M1 / Apple-Intel 笔记本电脑一起工作所涉及的障碍。

注意安全。