Airflow mysql 到 google 云存储，需要类似字节的对象错误

Question

我正在研究查询 MySQL 数据库、提取数据并将其加载到 Google 云存储的 DAG。

我尝试导出的 table 包括 text、int、float、varchar(20) 和 varchar(32) 数据。

我正在使用 airflow v1.8.0。

default_args = {
    'owner' : 'tia',
    'start_date' : datetime(2018, 1, 4),
    'depends_on_past' : False,
    'retries' : 1,
    'retry_delay':timedelta(minutes=5),
}

dag = DAG('mysql_to_gcs', default_args=default_args)

export_waybills = MySqlToGoogleCloudStorageOperator(
   task_id='extract_waybills',
   mysql_conn_id = 'podiotestmySQL',
   sql = 'SELECT * FROM podiodb.logistics_waybills',
   bucket='podio-reader-storage',
   filename= 'podio-data/waybills{}.json',
   schema_filename='podio-data/schema/waybills.json',
   dag=dag)

我遇到了以下错误，似乎与此类似 Airflow mysql to gcp Dag error

[2018-01-04 11:12:23,372] {models.py:1342} INFO - Executing on 2018-01-04 00:00:00

[2018-01-04 11:12:23,400] {base_hook.py:67} INFO - Using connection to: 35.189.207.140

[2018-01-04 11:12:24,903] {models.py:1417} ERROR - a bytes-like object is required, not 'str'

Traceback (most recent call last):

File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/models.py", line 1374, in run

result = task_copy.execute(context=context)

File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py", line 91, in execute

files_to_upload = self._write_local_data_files(cursor)

File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py", line 136, in _write_local_data_files

json.dump(row_dict, tmp_file_handle)

File "/usr/lib/python3.5/json/init.py", line 179, in dump fp.write(chunk)

File "/usr/lib/python3.5/tempfile.py", line 622, in func_wrapper return func(*args, **kwargs)

TypeError: a bytes-like object is required, not 'str'

[2018-01-04 11:12:24,907] {models.py:1433} INFO - Marking task as UP_FOR_RETRY

[2018-01-04 11:12:25,037] {models.py:1462} ERROR - a bytes-like object is required, not 'str'

有谁知道为什么抛出异常？

Answer 1

您在使用 python 3 吗？因为它接缝 mysql_to_gcs 最新版本 (1.90) 不 python 兼容它接缝。

看来这个运算符已更改 here to make mysql_to_gcs py3 compatible. But the latest version(1.90) doesn't include this change

Airflow mysql 到 google 云存储，需要类似字节的对象错误

Airflow mysql to google cloud storage, byte-like object required error

python

mysql

google-cloud-storage

google-cloud-platform

airflow