无法加载具有相同列名但顺序不同的镶木地板文件

Question

无法加载具有相同列名但顺序不同的镶木地板文件。

场景：

ABD-MacBook-Pro:ttt abd$ tree
.
├── testing1.paquet
└── testing2.paquet

如上所述，我有两个镶木地板文件。两个文件中的列名相同，只是顺序不同，我能够使用 Spark 加载这些文件。如果我在这里遗漏了什么，你能告诉我吗？还是 pyarrow 不支持？

我正在尝试使用以下命令加载这些 parquet 文件。

pandas_df = pq.ParquetDataset('ttt', filesystem=file_system).read_pandas().to_pandas()

在运行以上命令中出现以下错误。

ValueError: Schema in ttt//testing2.paquet was different.

C1: string
C2: string
C3: string
C4: string
Unnamed: 4: double
Unnamed: 5: double
Unnamed: 6: double
__index_level_0__: int64
metadata
--------
{b'pandas': b'{"index_columns": ["__index_level_0__"], "column_indexes": [{"na'
            b'me": null, "field_name": null, "pandas_type": "unicode", "numpy_'
            b'type": "object", "metadata": {"encoding": "UTF-8"}}], "columns":'
            b' [{"name": "C1", "field_name": "C1", "pandas_type": "unicode", "'
            b'numpy_type": "object", "metadata": null}, {"name": "C2", "field_'
            b'name": "C2", "pandas_type": "unicode", "numpy_type": "object", "'
            b'metadata": null}, {"name": "C3", "field_name": "C3", "pandas_typ'
            b'e": "unicode", "numpy_type": "object", "metadata": null}, {"name'
            b'": "C4", "field_name": "C4", "pandas_type": "unicode", "numpy_ty'
            b'pe": "object", "metadata": null}, {"name": "Unnamed: 4", "field_'
            b'name": "Unnamed: 4", "pandas_type": "float64", "numpy_type": "fl'
            b'oat64", "metadata": null}, {"name": "Unnamed: 5", "field_name": '
            b'"Unnamed: 5", "pandas_type": "float64", "numpy_type": "float64",'
            b' "metadata": null}, {"name": "Unnamed: 6", "field_name": "Unname'
            b'd: 6", "pandas_type": "float64", "numpy_type": "float64", "metad'
            b'ata": null}, {"name": null, "field_name": "__index_level_0__", "'
            b'pandas_type": "int64", "numpy_type": "int64", "metadata": null}]'
            b', "pandas_version": "0.23.0"}'}

vs

C1: string
C2: string
C4: string
C3: string
Unnamed: 4: double
Unnamed: 5: double
Unnamed: 6: double
__index_level_0__: int64
metadata
--------
{b'pandas': b'{"index_columns": ["__index_level_0__"], "column_indexes": [{"na'
            b'me": null, "field_name": null, "pandas_type": "unicode", "numpy_'
            b'type": "object", "metadata": {"encoding": "UTF-8"}}], "columns":'
            b' [{"name": "C1", "field_name": "C1", "pandas_type": "unicode", "'
            b'numpy_type": "object", "metadata": null}, {"name": "C2", "field_'
            b'name": "C2", "pandas_type": "unicode", "numpy_type": "object", "'
            b'metadata": null}, {"name": "C4", "field_name": "C4", "pandas_typ'
            b'e": "unicode", "numpy_type": "object", "metadata": null}, {"name'
            b'": "C3", "field_name": "C3", "pandas_type": "unicode", "numpy_ty'
            b'pe": "object", "metadata": null}, {"name": "Unnamed: 4", "field_'
            b'name": "Unnamed: 4", "pandas_type": "float64", "numpy_type": "fl'
            b'oat64", "metadata": null}, {"name": "Unnamed: 5", "field_name": '
            b'"Unnamed: 5", "pandas_type": "float64", "numpy_type": "float64",'
            b' "metadata": null}, {"name": "Unnamed: 6", "field_name": "Unname'
            b'd: 6", "pandas_type": "float64", "numpy_type": "float64", "metad'
            b'ata": null}, {"name": null, "field_name": "__index_level_0__", "'
            b'pandas_type": "int64", "numpy_type": "int64", "metadata": null}]'
            b', "pandas_version": "0.23.0"}'}

Answer 1

pyarrow 目前尚不支持此功能。更具体地说，当前的限制是不同片段/文件的所有模式都需要相同（不仅是顺序，还有类型）。

当然计划改善这种情况并在读取镶木地板文件时进行一些模式规范化（参见涵盖这种情况的例如https://issues.apache.org/jira/browse/ARROW-2659 about different types). For this specific issue, there is this JIRA issue https://issues.apache.org/jira/browse/ARROW-2366。

无法加载具有相同列名但顺序不同的镶木地板文件

Unable to load parquet files with same columns names but with different order

pandas

python-3.6

pyarrow