对于bigtable和python，出现google.api_core.exceptions.Aborted这样的异常的原因是什么：409 Error while reading table？

Question

我在 table 上使用 read_rows 时遇到此异常。 table 有文档特征的行，每个文档有 300 到 800 个特征，大约有 200 万个文档。 row_key 是特征，列是具有该特征的文档 ID。有数十亿行。

我运行在 GCP 上的 16 CPU VM 上，平均负载在 6 到 10 之间。我正在使用 python bigtable SDK 和 python 3.6.8 和 google-cloud-bigtable 2.3.3.

我在使用 table.read_rows(start_key=foo#xy, end_key=foo#xz) 读取行时遇到这种异常。 foo#xy 和 foo#xy 来自 table.sample_row_keys()。我从 sample_row_keys 获得了 200 个前缀，并且在我收到此错误之前成功处理了前 5 个左右。我是运行 ThreadPool 中的 table.read_rows() 调用。

如果您遇到过这样的异常并对其进行了调查，它的原因是什么以及您采取了什么措施来防止它？

Traceback (most recent call last):
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/google/api_core/grpc_helpers.py", line 106, in __next__
    return next(self._wrapped)
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/grpc/_channel.py", line 809, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.ABORTED
        details = "Error while reading table 'projects/arxiv-production/instances/docsim/tables/docsim' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)"
        debug_error_string = "{"created":"@1635477504.521060666","description":"Error received from peer ipv4:172.217.0.42:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Error while reading table 'projects/arxiv-production/instances/docsim/tables/docsim' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)","grpc_status":10}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/bdc34/docsim/docsim/loading/all_common_hashes.py", line 53, in <module>
    for hash, n, c, dt in pool.imap_unordered( do_prefix, jobs ):
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/bdc34/docsim/docsim/loading/all_common_hashes.py", line 33, in do_prefix
    for hash, common, papers in by_prefix(db, start, end):
  File "/home/bdc34/docsim/docsim/loading/all_common_hashes.py", line 15, in by_prefix
    for row in db.table.read_rows(start_key=start, end_key=end):
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 485, in __iter__
    response = self._read_next_response()
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 474, in _read_next_response
    return self.retry(self._read_next, on_error=self._on_error)()
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
    on_error=on_error,
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 470, in _read_next
    return six.next(self.response_iterator)
  File "/home/bdc34/docsim/venv/lib64/python3.6/site-packages/google/api_core/grpc_helpers.py", line 109, in __next__
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.Aborted: 409 Error while reading table 'projects/testproject/instances/testinstance/tables/testtable' : 
Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)

Answer 1

此错误可能有不同的原因。您可能需要确保您没有遇到 hotspotting 情况。

另外，您可以检查您是否在 table 中读取许多不同的行，以及您是否正在创建尽可能少的客户端。如果您正在读取仅包含少量行的大范围行键，性能也会受到影响。您会找到有关解决性能问题的更多一般性建议 here。

Answer 2

我通过使用更小的范围调用 read_rows 来解决这个问题。来自 table.sample_row_keys() 的前缀跨越大约 1.5B 行。将每个范围平分 5 次以产生更小的范围。

我通过将开始和结束 row_keys 填充到相同的长度来平分，将它们转换为整数并找到中点。

对于bigtable和python，出现google.api_core.exceptions.Aborted这样的异常的原因是什么：409 Error while reading table？

With bigtable and python, what are the causes of an exception like google.api_core.exceptions.Aborted: 409 Error while reading table?

python

google-cloud-bigtable

bigtable