打开远程 zarr 存储时出现 KeyError '.zmetadata'

Question

正在尝试使用 xarray 从 s3 读取 zarr 存储。获取密钥错误。有什么想法

import fsspec
import xarray as xr


uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr"
ds = xr.open_zarr(fsspec.get_mapper(uri, anon=True), consolidated=True)

我先下载本地打开就ok了:

import s3fs
fs = s3fs.S3FileSystem(anon=True)
fs.get("s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr/*", "eastward_wind_at_10_metres.zarr", recursive=True)

这是与 top 命令关联的 Traceback

Traceback (most recent call last):
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 234, in _call_s3
    return await method(**additional_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py", line 132, in __getitem__
    result = self.fs.cat(k)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/asyn.py", line 241, in cat
    raise ex
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 737, in _cat_file
    resp = await self._call_s3(
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 252, in _call_s3
    raise translate_boto_error(err) from err
FileNotFoundError: The specified key does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py", line 675, in open_zarr
    ds = open_dataset(
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py", line 572, in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py", line 294, in open_group
    zarr_group = zarr.open_consolidated(store, **open_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/convenience.py", line 1178, in open_consolidated
    meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/storage.py", line 2680, in __init__
    meta = json_loads(store[metadata_key])
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py", line 136, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

Answer 1

问题是这个 zarr 数据集没有统一的元数据。错误实际上是告诉你这个 (KeyError: '.zmetadata').

打开远程 zarr 存储时出现 KeyError '.zmetadata'

getting KeyError '.zmetadata' when opening remote zarr store

python-xarray

zarr

fsspec