Cassandra：批量写入优化
Cassandra : Batch write optimisation
我从客户那里收到了大约 20 个密钥的批量写入请求。
我可以将它们批量写入 C* 或以异步方式单独写入它们并等待将来完成它们。
根据文档，批量写入似乎不是一个很好的选择，因为我的插入率会很高，如果键属于不同的分区，协调员将不得不做额外的工作。
Is there a way in datastax java driver with which I can group keys
which could belong to same partition and then club them into small
batches and then do invidual unlogged batch write in async. IN that
way i make less rpc calls to server at the same time coordinator will
have to write locally. I will be using token aware policy.
你的想法是对的，但是没有built-in的办法，一般都是手动的。
这里的主要规则是使用 TokenAwarePolicy，因此在 driver 端会发生一些协调。
然后，您可以根据分区键的相等性对您的请求进行分组，这可能就足够了，具体取决于您的工作量。
我所说的“按分区键相等分组”的意思是，例如你有一些数据看起来像
MyData { partitioningKey, clusteringKey, otherValue, andAnotherOne }
然后在插入多个这样的 objects 时，将它们按 MyData.partitioningKey 分组。它是，对于所有现有的 paritioningKey 值，您使用相同的 partitioningKey 获取所有 objects，并将它们包装在 BatchStatement 中。现在你有几个BatchStatements，所以就执行它们。
如果您想更进一步并模仿 cassandra 哈希，那么您应该通过 com.datastax.driver.core.Cluster class 中的 getMetadata 方法查看集群元数据，方法 getTokenRanges并将它们与 Murmur3Partitioner.getToken 或您在 cassandra.yaml 中配置的任何其他分区程序的结果进行比较。不过我自己从来没有尝试过。
因此，我建议实施第一种方法，然后对您的应用程序进行基准测试。我自己也在使用这种方法，在我的工作负载上，它比没有批处理要好得多，更不用说没有分组的批处理了。
Logged 批处理在 Cassandra 中应谨慎使用，因为它们会带来额外的开销。它还取决于分区键分布。如果您的批量写入目标是单个分区，那么使用 Unlogged 批处理会导致单个插入操作。
一般来说，以异步方式单独编写它们似乎是一个很好的方法，如下所示：
https://medium.com/@foundev/cassandra-batch-loading-without-the-batch-the-nuanced-edition-dd78d61e9885
您可以在上面的网站上找到如何处理多个异步写入的示例代码：
https://gist.github.com/rssvihla/26271f351bdd679553d55368171407be#file-bulkloader-java
https://gist.github.com/rssvihla/4b62b8e5625a805583c1ce39b1260ff4#file-bulkloader-java
编辑：
另请阅读：
https://inoio.de/blog/2016/01/13/cassandra-to-batch-or-not-to-batch/#14
What does a single partition batch cost?
There’s no batch log written for single partition batches. The
coordinator doesn’t have any extra work (as for multi partition
writes) because everything goes into a single partition. Single
partition batches are optimized: they are applied with a single
RowMutation [10].
In a few words: single partition batches don’t put much more load on
the server than normal writes.
What does a multi partition batch cost?
Let me just quote Christopher Batey, because he has summarized this
very well in his post “Cassandra anti-pattern: Logged batches” [3]:
Cassandra [is first] writing all the statements to a batch log. That
batch log is replicated to two other nodes in case the coordinator
fails. If the coordinator fails then another replica for the batch log
will take over. [..] The coordinator has to do a lot more work than
any other node in the cluster.
Again, in bullets what has to be done:
- serialize the batch statements
- write the serialized batch to the batch log system table
- replicate of this serialized batch to 2 nodes
- coordinate writes to nodes holding the different partitions
- on success remove the serialized batch from the batch log (also on the 2 replicas)
Remember that unlogged batches for multiple partitions are deprecated since Cassandra 2.1.6
我从客户那里收到了大约 20 个密钥的批量写入请求。 我可以将它们批量写入 C* 或以异步方式单独写入它们并等待将来完成它们。
根据文档，批量写入似乎不是一个很好的选择，因为我的插入率会很高，如果键属于不同的分区，协调员将不得不做额外的工作。
Is there a way in datastax java driver with which I can group keys which could belong to same partition and then club them into small batches and then do invidual unlogged batch write in async. IN that way i make less rpc calls to server at the same time coordinator will have to write locally. I will be using token aware policy.
你的想法是对的，但是没有built-in的办法，一般都是手动的。
这里的主要规则是使用 TokenAwarePolicy，因此在 driver 端会发生一些协调。
然后，您可以根据分区键的相等性对您的请求进行分组，这可能就足够了，具体取决于您的工作量。
我所说的“按分区键相等分组”的意思是，例如你有一些数据看起来像
MyData { partitioningKey, clusteringKey, otherValue, andAnotherOne }
然后在插入多个这样的 objects 时，将它们按 MyData.partitioningKey 分组。它是，对于所有现有的 paritioningKey 值，您使用相同的 partitioningKey 获取所有 objects，并将它们包装在 BatchStatement 中。现在你有几个BatchStatements，所以就执行它们。
如果您想更进一步并模仿 cassandra 哈希，那么您应该通过 com.datastax.driver.core.Cluster class 中的 getMetadata 方法查看集群元数据，方法 getTokenRanges并将它们与 Murmur3Partitioner.getToken 或您在 cassandra.yaml 中配置的任何其他分区程序的结果进行比较。不过我自己从来没有尝试过。
因此，我建议实施第一种方法，然后对您的应用程序进行基准测试。我自己也在使用这种方法，在我的工作负载上，它比没有批处理要好得多，更不用说没有分组的批处理了。
Logged 批处理在 Cassandra 中应谨慎使用，因为它们会带来额外的开销。它还取决于分区键分布。如果您的批量写入目标是单个分区，那么使用 Unlogged 批处理会导致单个插入操作。
一般来说，以异步方式单独编写它们似乎是一个很好的方法，如下所示： https://medium.com/@foundev/cassandra-batch-loading-without-the-batch-the-nuanced-edition-dd78d61e9885
您可以在上面的网站上找到如何处理多个异步写入的示例代码： https://gist.github.com/rssvihla/26271f351bdd679553d55368171407be#file-bulkloader-java https://gist.github.com/rssvihla/4b62b8e5625a805583c1ce39b1260ff4#file-bulkloader-java
编辑：
另请阅读：
https://inoio.de/blog/2016/01/13/cassandra-to-batch-or-not-to-batch/#14
What does a single partition batch cost?
There’s no batch log written for single partition batches. The coordinator doesn’t have any extra work (as for multi partition writes) because everything goes into a single partition. Single partition batches are optimized: they are applied with a single RowMutation [10].
In a few words: single partition batches don’t put much more load on the server than normal writes.
What does a multi partition batch cost?
Let me just quote Christopher Batey, because he has summarized this very well in his post “Cassandra anti-pattern: Logged batches” [3]:
Cassandra [is first] writing all the statements to a batch log. That batch log is replicated to two other nodes in case the coordinator fails. If the coordinator fails then another replica for the batch log will take over. [..] The coordinator has to do a lot more work than any other node in the cluster.
Again, in bullets what has to be done:
- serialize the batch statements
- write the serialized batch to the batch log system table
- replicate of this serialized batch to 2 nodes
- coordinate writes to nodes holding the different partitions
- on success remove the serialized batch from the batch log (also on the 2 replicas)
Remember that unlogged batches for multiple partitions are deprecated since Cassandra 2.1.6