当 lambda 中的事件失败时,kinesis 如何保持偏移量并再次推送记录
How kinesis keep the offset and push the record again when an event fails in lambda
我是 AWS lambda 和 Kinesis 的新手。请帮忙解答以下问题
我有一个运动流作为 lambda 的源,目标又是运动。我有以下疑问。
系统不想丢失一条记录。
如果有一条记录在lambda中处理失败,如何重新拉入lambda?它如何保留未处理的记录? kinesis如何跟踪偏移量以处理下一条记录?
请更新。
来自AWS Lambda docs about using Lambda with Kinesis:
If your function returns an error, Lambda retries the batch until processing succeeds or the data expires. Until the issue is resolved, no data in the shard is processed. To avoid stalled shards and potential data loss, make sure to handle and record processing errors in your code.
在这种情况下,还要考虑 Retention Period
of Kinesis:
The retention period is the length of time that data records are accessible after they are added to the stream. A stream’s retention period is set to a default of 24 hours after creation. You can increase the retention period up to 168 hours (7 days)
如第一个引述中所述,AWS 将在保留期到期后删除该事件。这对你来说意味着:
a) 注意您的 Lambda 函数正确处理错误。
b) 如果保留所有记录很重要,也将它们存储在持久存储中,例如DynamoDB.
除此之外,您还应该了解重复的 Lambda 执行。另一个 Whosebug 问题和答案上有很棒的 blog post available explaining how you can achieve an idempotent implementation. And read 。
我是 AWS lambda 和 Kinesis 的新手。请帮忙解答以下问题
我有一个运动流作为 lambda 的源,目标又是运动。我有以下疑问。 系统不想丢失一条记录。
如果有一条记录在lambda中处理失败,如何重新拉入lambda?它如何保留未处理的记录? kinesis如何跟踪偏移量以处理下一条记录?
请更新。
来自AWS Lambda docs about using Lambda with Kinesis:
If your function returns an error, Lambda retries the batch until processing succeeds or the data expires. Until the issue is resolved, no data in the shard is processed. To avoid stalled shards and potential data loss, make sure to handle and record processing errors in your code.
在这种情况下,还要考虑 Retention Period
of Kinesis:
The retention period is the length of time that data records are accessible after they are added to the stream. A stream’s retention period is set to a default of 24 hours after creation. You can increase the retention period up to 168 hours (7 days)
如第一个引述中所述,AWS 将在保留期到期后删除该事件。这对你来说意味着:
a) 注意您的 Lambda 函数正确处理错误。
b) 如果保留所有记录很重要,也将它们存储在持久存储中,例如DynamoDB.
除此之外,您还应该了解重复的 Lambda 执行。另一个 Whosebug 问题和答案上有很棒的 blog post available explaining how you can achieve an idempotent implementation. And read