DynamoDB get_item 以毫秒读取 400kb 数据

Question

我有一个名为 events 的 dynamodb table，我在其中存储了所有 user event details，例如 product_view、add_to_cart 和 product_purchase

在这个eventstable中，我有一些items的存储容量达到了400kb

问题：

        response = self._table.get_item(
            Key={
                PARTITION_KEY: <pk>,
                SORT_KEY: <sk>,
            },
            ConsistentRead=False,
        )

当我想使用 dynamodb get_item 方法访问 item(400kb) 时，它需要大约 5 seconds 到 return 结果。

我已经用过DAX

目标

我想在 1 秒内阅读 400kb 项。

重要信息：

dynamodb中的数据将以此格式存储

{
 "partition_key": "user_id1111",
 "sort_key": "version_1",
 "attributes": {
  "events": [
   {
    "t": "1614712316",  
    "a": "product_view",   
    "i": "1275"
   },
   {
    "t": "1614712316",  
    "a": "product_add",   
    "i": "1275"
   },
   {
    "t": "1614712316",  
    "a": "product_purchase",   
    "i": "1275"
   },
    ...

  ]
 }
}

t是一个时间戳
a 可能是 product_view,product_add,product_purchase
i 是 product_id

如果您看到上面的项目 events 是一个列表，它将被新事件追加。

我有一个 400kb 的项目，其事件数在 events 列表中

我写了一些脚本来测量时间，结果如下

import boto3
import datetime

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('events')

pk = f"user_id1111"
sk = f"version_1"


t_load_start = datetime.datetime.now()


response = table.get_item(
    Key={
        "partition_key": pk,
        "sort_key": sk,
    },
    ReturnConsumedCapacity="TOTAL"
)
capacity_units = response["ConsumedCapacity"]["CapacityUnits"]

t_load_end = datetime.datetime.now()
seconds = (t_load_end - t_load_start).total_seconds()

print(f"Elapsed time is::{seconds}sec and {capacity_units} capacity units")

这是我得到的输出。

Elapsed time is::5.676799sec and 50.0 capacity units

有人可以为此提出解决方案吗？

Answer 1

听起来您遇到了一些问题。第一个问题是您运行违反了 400kb 的项目大小限制。虽然您没有说这是一个问题，但可能值得重新访问您的数据模型，以便您可以存储更多事件数据。

性能问题不太可能与您的数据模型有关。 get_item 操作的平均延迟应为个位数毫秒，特别是因为您指定的是最终一致性读取。这里发生了其他事情。

您如何测试和衡量此操作的性能？

AWS 文档有一些来自 about troubleshooting high latency DynamoDB operations 的建议可能有用。

Answer 2

tl;dr：将函数内存增加到至少 1024MB，请参阅更新 2

我很好奇，所以我做了一些测量。我创建了一个脚本，用于在新的 table.

中创建一个几乎恰好 400KB 大小的大 boi 项目

然后我测试来自 Python 的两次读取 - 一次使用资源 API，另一个使用较低级别的客户端 - 在两种情况下最终读取一致。

这是我测量的：

Reading Big Boi from a Table Resource took 0.366508s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.301585s and consumed 50.0 RCUs

如果我们从 RCU 推断，它读取的项目的大小约为 50 * 2 * 4KB = 400 KB（最终一致性读取消耗 0.5 个 RCU）。

我运行在德国本地与 eu-central-1（F运行kfurt，德国）进行了几次，我看到的最高延迟约为 900 毫秒。（这没有 DAX。）

因此，我认为您应该向我们展示您是如何进行测量的。

import uuid
from datetime import datetime, timedelta

import boto3
import boto3.dynamodb.conditions as conditions

TABLE_NAME = "big-boi-test"
BIG_BOI_PK = "f0ba8d6c"

TABLE_RESOURCE = boto3.resource("dynamodb").Table(TABLE_NAME)
DDB_CLIENT = boto3.client("dynamodb")

def create_table():
    DDB_CLIENT.create_table(
        AttributeDefinitions=[{"AttributeName": "PK", "AttributeType": "S"}],
        TableName=TABLE_NAME,
        KeySchema=[{"AttributeName": "PK", "KeyType": "HASH"}],
        BillingMode="PAY_PER_REQUEST"
    )

def create_big_boi_item() -> str:
    # based on calculations here: https://zaccharles.github.io/dynamodb-calculator/
    template = {
        "PK": {
            "S": BIG_BOI_PK
        },
        "bigBoi": {
            "S": ""
        }
    } # This is 16 bytes

    big_boi = "X" * (1024 * 400 - 16)
    template["bigBoi"]["S"] = big_boi
    return template

def store_big_boi():
    big_bio = create_big_boi_item()

    DDB_CLIENT.put_item(
        Item=big_bio,
        TableName=TABLE_NAME
    )

def get_big_boi_with_table_resource():

    start = datetime.now()
    response = TABLE_RESOURCE.get_item(
        Key={"PK": BIG_BOI_PK},
        ReturnConsumedCapacity="TOTAL"
    )
    end = datetime.now()
    seconds = (end - start).total_seconds()
    capacity_units = response["ConsumedCapacity"]["CapacityUnits"]

    print(f"Reading Big Boi from a Table Resource took {seconds}s and consumed {capacity_units} RCUs")

def get_big_boi_with_client():

    start = datetime.now()
    response = DDB_CLIENT.get_item(
        Key={"PK": {"S": BIG_BOI_PK}},
        ReturnConsumedCapacity="TOTAL",
        TableName=TABLE_NAME
    )
    end = datetime.now()
    seconds = (end - start).total_seconds()
    capacity_units = response["ConsumedCapacity"]["CapacityUnits"]

    print(f"Reading Big Boi from a Client took {seconds}s and consumed {capacity_units} RCUs")

if __name__ == "__main__":
    # create_table()
    # store_big_boi()
    get_big_boi_with_table_resource()
    get_big_boi_with_client()

更新

我用一个看起来更像你正在使用的项目再次进行了相同的测量，无论我以何种方式请求它们，我的平均时间仍然低于 1000 毫秒：

Reading Big Boi from a Table Resource took 1.492829s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.871583s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.857513s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.769432s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.690172s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.670099s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.633489s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.605999s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.598635s and consumed 50.0 RCUs
Reading Big Boi from a Table Resource took 0.606553s and consumed 50.0 RCUs
Reading Big Boi from a Client took 1.66636s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.921605s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.831735s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.707082s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.668602s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.648401s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.5695s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.592073s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.611436s and consumed 50.0 RCUs
Reading Big Boi from a Client took 0.553827s and consumed 50.0 RCUs
Average latency over 10 requests with the table resource: 0.7796304s
Average latency over 10 requests with the client: 0.7770621s

这是物品的样子：

这是供您验证的完整测试脚本：

import statistics
import uuid
from datetime import datetime, timedelta

import boto3
import boto3.dynamodb.conditions as conditions

TABLE_NAME = "big-boi-test"
BIG_BOI_PK = "NestedBoi"

TABLE_RESOURCE = boto3.resource("dynamodb").Table(TABLE_NAME)
DDB_CLIENT = boto3.client("dynamodb")

def create_table():
    DDB_CLIENT.create_table(
        AttributeDefinitions=[{"AttributeName": "PK", "AttributeType": "S"}],
        TableName=TABLE_NAME,
        KeySchema=[{"AttributeName": "PK", "KeyType": "HASH"}],
        BillingMode="PAY_PER_REQUEST"
    )

def create_big_boi_item() -> str:
    # based on calculations here: https://zaccharles.github.io/dynamodb-calculator/
    template = {
        "PK": {
            "S": "NestedBoi"
        },
        "bigBoiContainer": {
            "M": {
            "bigBoiList": {
                "L": [
                
                ]
            }
            }
        }
    } # 43 bytes

    item = {
        "M": {
        "t": {
            "S": "1614712316"
        },
        "a": {
            "S": "product_view"
        },
        "i": {
            "S": "1275"
        }
        }
    }  # 36 bytes

    number_of_items = int((1024 * 400 - 43) / 36)

    for _ in range(number_of_items):
        template["bigBoiContainer"]["M"]["bigBoiList"]["L"].append(item)

    return template

def store_big_boi():
    big_bio = create_big_boi_item()

    DDB_CLIENT.put_item(
        Item=big_bio,
        TableName=TABLE_NAME
    )

def get_big_boi_with_table_resource():

    start = datetime.now()
    response = TABLE_RESOURCE.get_item(
        Key={"PK": BIG_BOI_PK},
        ReturnConsumedCapacity="TOTAL"
    )
    end = datetime.now()
    seconds = (end - start).total_seconds()
    capacity_units = response["ConsumedCapacity"]["CapacityUnits"]

    print(f"Reading Big Boi from a Table Resource took {seconds}s and consumed {capacity_units} RCUs")

    return seconds

def get_big_boi_with_client():

    start = datetime.now()
    response = DDB_CLIENT.get_item(
        Key={"PK": {"S": BIG_BOI_PK}},
        ReturnConsumedCapacity="TOTAL",
        TableName=TABLE_NAME
    )
    end = datetime.now()
    seconds = (end - start).total_seconds()
    capacity_units = response["ConsumedCapacity"]["CapacityUnits"]

    print(f"Reading Big Boi from a Client took {seconds}s and consumed {capacity_units} RCUs")

    return seconds

if __name__ == "__main__":
    # create_table()
    # store_big_boi()

    n_experiments = 10
    experiments_with_table_resource = [get_big_boi_with_table_resource() for i in range(n_experiments)]
    experiments_with_client = [get_big_boi_with_client() for i in range(n_experiments)]
    print(f"Average latency over {n_experiments} requests with the table resource: {statistics.mean(experiments_with_table_resource)}s")
    print(f"Average latency over {n_experiments} requests with the client: {statistics.mean(experiments_with_client)}s")

如果我增加 n_experiments，它往往会变得更快，可能是因为 DDB 在内部缓存。

仍然：无法重现。

更新 2

了解你是运行 Lambda 函数后，我运行使用不同的内存配置在 Lambda 内部再次测试。

Memory	n_experiments	average time with resource	average time with client
128MB	10	6.28s	5.06s
256MB	10	3.26s	2.61s
512MB	10	1.62s	1.33s
1024MB	10	0.84s	0.68s
2048MB	10	0.52s	0.43s
4096MB	10	0.51s	0.41s

如评论中所述，CPU 和网络性能与分配给函数的内存量成比例。你可以通过砸钱来解决你的问题:-)

DynamoDB get_item 以毫秒读取 400kb 数据

DynamoDB get_item to read 400kb data in milliseconds

amazon-web-services

amazon-dynamodb

tl;dr：将函数内存增加到至少 1024MB，请参阅更新 2

更新

更新 2