GCP Composer - ftplib timeouterror errno 110 连接超时

Question

我试图通过 GCP Composer Tasks 从 FTP 服务器的 txt 文件中获取数据。

所以我在代码中导入并使用了 ftplib 包。

像这样。

ftp = FTP()
ftp.connect(host=HOST,port=PORT, timeout=600)
ftp.login(user=USER,passwd=PSWD)
ftp.set_pasv(True)

ftp.sendcmd('TYPE A')
conn = ftp.transfercmd(F"RETR {PATH}")
fp = conn.makefile('rb')

但是。此 (conn = ftp.transfercmd(F"RETR {PATH}")) 代码造成 TimeoutError: [Errno 110] Connection timed out 错误。

ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1166, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 150, in execute
    return_value = self.execute_callable()
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 161, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/home/airflow/gcs/dags/FTP_ZIPCODE_to_BQ_DAG.py", line 91, in replace_BQ_table
    conn = ftp.transfercmd(F"RETR {PATH}")
  File "/opt/python3.8/lib/python3.8/ftplib.py", line 389, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/opt/python3.8/lib/python3.8/ftplib.py", line 350, in ntransfercmd
    conn = socket.create_connection((host, port), self.timeout,
  File "/opt/python3.8/lib/python3.8/socket.py", line 808, in create_connection
    raise err
  File "/opt/python3.8/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

你知道原因吗？？？？

并且...这是我的 gcp 作曲家环境。

图片版本：composer-1.17.7-airflow-2.1.4
python 版本：3
网络 VPC 原生：启用

Answer 1

可能发生的一件事是连接不工作。为了使连接正常工作，您需要将 Cloud NAT 与 Cloud Composer 一起设置，以便为工作人员提供 public 互联网访问权限，如 documentation.

中所述

GCP Composer - ftplib timeouterror errno 110 连接超时

GCP Composer - ftplib timeouterror errno 110 connection timed out

ftplib

google-cloud-composer