DynamoDb : 扫描查询没有 return 所有数据

Question

我有一个包含数千条数据的 DynamoDb table。我正在使用 Scan 函数扫描 table，并且应用了 "Between" FilterExpression。然而，查询响应只给出了 3 条记录，而它应该 return 大约 100 条记录。

我已经使用 Node js 创建了 Lambda 函数。

Answer 1

另一个常见问题可能是扫描是否执行到 LastEvaluatedKey 为空。

如果您已经这样做但仍未获得所有物品，请出示您的代码以详细查看。

If the total number of scanned items exceeds the maximum data set size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation. The results also include the number of items exceeding the limit. A scan can result in no table data meeting the filter criteria.

If LastEvaluatedKey is empty, then the "last page" of results has been processed and there is no more data to be retrieved.

If LastEvaluatedKey is not empty, it does not necessarily mean that there is more data in the result set. The only way to know when you have reached the end of the result set is when LastEvaluatedKey is empty.

Answer 2

这是获取所有结果的示例代码：

 Map<String, AttributeValue> lastKeyEvaluated = null;
    do {
        ScanRequest sr = new ScanRequest()
                .withTableName("tableName")
                .withProjectionExpression("id")
                .withExclusiveStartKey(lastKeyEvaluated);
        ScanResult result = client.scan(sr);
        for (Map<String, AttributeValue> item : result.getItems()) {
            System.out.println(item.get("id").getS());
        }
        lastKeyEvaluated = result.getLastEvaluatedKey();
    } while (lastKeyEvaluated != null);

Answer 3

使用 Node.js 我实际上是在使用查询从数据库中检索项目。单个查询操作最多可以检索 1 MB 的数据。这就是为什么我创建了一个递归函数来从数据库中检索和连接数据，直到我们从响应中收到 LastEvaluatedKey。当我们收到 LastEvaluatedKey 作为 null 时，这意味着没有更多数据。我的函数使用索引从数据库中获取数据。使用查询函数比扫描更快更有效。

实际上，getItemByGSI函数有很多参数用于过滤和自定义查询，这些参数很有用。当然，您可以删除不适合您的情况的参数。

因此getAllItemsByGSI函数可用于从DynamoDB中检索所有数据，getItemByGSI可用于使用单个查询。

'use strict';
    
    const omitBy = require('lodash/omitBy');
    const isNil = require('lodash/isNil');
    const AWS = require('aws-sdk');
    
    const call = (action, params) => {
        return new Promise((resolve, reject) => {
            try {
                const dynamoDb = new AWS.DynamoDB.DocumentClient();
                resolve(dynamoDb[action](params).promise());
            } catch (error) {
                reject(error);
            }
        });
    };
    
    const getItemByGSI = ({
        TableName,
        IndexName,
        attribute,
        value,
        sortKey,
        sortValue,
        filter,
        filterValue,
        operator,
        filter1,
        filterValue1,
        LastEvaluatedKey,
        ScanIndexForward,
        Limit,
    }) => {
        return new Promise(async (resolve, reject) => {
            try {
                const params = {
                    TableName,
                    IndexName,
                    KeyConditionExpression: '#attrKey = :attrValue',
                    ExpressionAttributeValues: { ':attrValue': value },
                    ExpressionAttributeNames: { '#attrKey': attribute },
                    ExclusiveStartKey: LastEvaluatedKey,
                    Limit,
                    FilterExpression: null,
                };
                sortKey && sortValue
                    ? (params.KeyConditionExpression +=
                            ' and #sortKey = :sortValue' &&
                            (params.ExpressionAttributeNames['#sortKey'] = sortKey) &&
                            (params.ExpressionAttributeValues[':sortKey'] = sortValue))
                    : '';
                filter && filterValue
                    ? (params.FilterExpression = `#${filter} = :${filter}`) &&
                      (params.ExpressionAttributeNames[`#${filter}`] = filter) &&
                      (params.ExpressionAttributeValues[`:${filter}`] = filterValue)
                    : '';
                filter && filterValue && operator && filter1 && filterValue1
                    ? (params.FilterExpression += ` ${operator} #${filter1} = :${filter1}`) &&
                      (params.ExpressionAttributeNames[`#${filter1}`] = filter1) &&
                      (params.ExpressionAttributeValues[`:${filter1}`] = filterValue1)
                    : '';
                params = omitBy(params, isNil);
                if (ScanIndexForward === false)
                    params.ScanIndexForward = ScanIndexForward;
                const result = await call('query', params);
                resolve(result);
            } catch (error) {
                reject(error);
            }
        });
    };
    
    const getAllItemsByGSI = (data) => {
        return new Promise(async (resolve, reject) => {
            try {
                const finalData = [];
                const gettingData = await getItemByGSI(data);
                finalData = finalData.concat(gettingData.Items);
                if (gettingData.LastEvaluatedKey) {
                    const final2 = await getAllItemsByGSI({
                        ...data,
                        LastEvaluatedKey: gettingData.LastEvaluatedKey,
                    });
                    finalData = finalData.concat(final2);
                }
                resolve(finalData);
            } catch (err) {
                reject(err);
            }
        });
    };
    
    module.exports = {
        getItemByGSI,
        getAllItemsByGSI,
    };

DynamoDb : 扫描查询没有 return 所有数据

DynamoDb : Scan query does not return all the data

amazon-dynamodb

aws-lambda

aws-sdk

amazon-dynamodb-streams