AWS Kinesis 如何限制写入吞吐量？
How does AWS Kinesis throttle write throughput?
AWS Kinesis 的写入吞吐量相当低，为 1000 writes/sec 和 1MB/写入秒。 Kinesis 如何执行此限制？如果我尝试在一秒钟内执行 1500 次写入，额外的 500 次写入是否会被放入某种队列中，还是它们会失败？
看起来它只是失败并抛出异常。
An unsuccessfully processed record includes ErrorCode and ErrorMessage values. ErrorCode reflects the type of error and can be one of the following values: ProvisionedThroughputExceededException or InternalFailure. ErrorMessage provides more detailed information about the ProvisionedThroughputExceededException exception including the account ID, stream name, and shard ID of the record that was throttled. For more information about partially successful responses, see Adding Multiple Records with PutRecords in the Amazon Kinesis Data Streams Developer Guide.
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html
速率限制是如何完成的
Rate Limiting
The KPL includes a rate limiting feature, which limits per-shard throughput sent from a single producer. Rate limiting is implemented using a token bucket algorithm with separate buckets for both Kinesis Data Streams records and bytes. Each successful write to an Kinesis data stream adds a token (or multiple tokens) to each bucket, up to a certain threshold. This threshold is configurable but by default is set 50% higher than the actual shard limit, to allow shard saturation from a single producer.
You can lower this limit to reduce spamming due to excessive retries. However, the best practice is for each producer is to retry for maximum throughput aggressively and to handle any resulting throttling determined as excessive by expanding the capacity of the stream and implementing an appropriate partition key strategy.
https://docs.aws.amazon.com/streams/latest/dev/kinesis-producer-adv-retries-rate-limiting.html
这取决于您写入数据的方式。
如果您使用的是 PutRecord，则任何超过限制的请求都将失败并返回 ProvisionedThroughputExceededException，您必须重试该请求。但是，由于单个请求的往返时间大约为 20-30 毫秒，因此您需要有大量客户端才能受到限制。
PutRecords 调用被限制的可能性更高，因为您可以在单个请求中发送最多 500 条记录。如果它受到限制，限制可能会影响整个请求或请求中的单个记录（如果一个分片接受记录但另一个不接受记录，则可能会发生这种情况）。
要处理这个问题，您需要检查 PutRecords 响应中的 Records 列表。该数组与请求中的 Records 列表完全对应，但包含 PutRecordsResultEntry 个值。
如果条目有 SequenceNumber 那么你没问题：该记录已写入碎片。但是，如果它有一个 ErrorCode 那么您需要从请求中复制记录并重新发送它（假设错误代码是超出吞吐量；如果是内部错误，您也可以尝试重新发送，但这可能不工作）。
您将需要循环调用 PutRecords 直到响应中没有任何未发送的消息。
请注意，由于个别记录可能会受到限制和重新发送，因此您无法保证记录在分片上的显示顺序（它们按照收到的顺序存储在分片中）。
AWS Kinesis 的写入吞吐量相当低，为 1000 writes/sec 和 1MB/写入秒。 Kinesis 如何执行此限制？如果我尝试在一秒钟内执行 1500 次写入，额外的 500 次写入是否会被放入某种队列中，还是它们会失败？
看起来它只是失败并抛出异常。
An unsuccessfully processed record includes ErrorCode and ErrorMessage values. ErrorCode reflects the type of error and can be one of the following values: ProvisionedThroughputExceededException or InternalFailure. ErrorMessage provides more detailed information about the ProvisionedThroughputExceededException exception including the account ID, stream name, and shard ID of the record that was throttled. For more information about partially successful responses, see Adding Multiple Records with PutRecords in the Amazon Kinesis Data Streams Developer Guide.
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html
速率限制是如何完成的
Rate Limiting The KPL includes a rate limiting feature, which limits per-shard throughput sent from a single producer. Rate limiting is implemented using a token bucket algorithm with separate buckets for both Kinesis Data Streams records and bytes. Each successful write to an Kinesis data stream adds a token (or multiple tokens) to each bucket, up to a certain threshold. This threshold is configurable but by default is set 50% higher than the actual shard limit, to allow shard saturation from a single producer.
You can lower this limit to reduce spamming due to excessive retries. However, the best practice is for each producer is to retry for maximum throughput aggressively and to handle any resulting throttling determined as excessive by expanding the capacity of the stream and implementing an appropriate partition key strategy.
https://docs.aws.amazon.com/streams/latest/dev/kinesis-producer-adv-retries-rate-limiting.html
这取决于您写入数据的方式。
如果您使用的是 PutRecord，则任何超过限制的请求都将失败并返回 ProvisionedThroughputExceededException，您必须重试该请求。但是，由于单个请求的往返时间大约为 20-30 毫秒，因此您需要有大量客户端才能受到限制。
PutRecords 调用被限制的可能性更高，因为您可以在单个请求中发送最多 500 条记录。如果它受到限制，限制可能会影响整个请求或请求中的单个记录（如果一个分片接受记录但另一个不接受记录，则可能会发生这种情况）。
要处理这个问题，您需要检查 PutRecords 响应中的 Records 列表。该数组与请求中的 Records 列表完全对应，但包含 PutRecordsResultEntry 个值。
如果条目有 SequenceNumber 那么你没问题：该记录已写入碎片。但是，如果它有一个 ErrorCode 那么您需要从请求中复制记录并重新发送它（假设错误代码是超出吞吐量；如果是内部错误，您也可以尝试重新发送，但这可能不工作）。
您将需要循环调用 PutRecords 直到响应中没有任何未发送的消息。
请注意，由于个别记录可能会受到限制和重新发送，因此您无法保证记录在分片上的显示顺序（它们按照收到的顺序存储在分片中）。