将数据从 Google Cloud-SQL 移动到 Cloud Datastore
Move data from Google Cloud-SQL to Cloud Datastore
我正在尝试将我的数据从 Cloud SQL 移动到 Cloud Datastore。
SQL 数据库中的条目略低于 500 万条。
似乎我每天只能移动超过 100,000 个实体，然后才会出现配额错误。
我无法确定我超过了哪个配额，但是我有指数退避以确保我不会发送得太快。
最终它达到了 5 分钟，与 SQL 服务器的连接中断，但我认为每秒写入配额不是问题所在。在我的 API 页面或 App Engine API 页面中，我没有看到任何其他超出配额的情况。
我试过两种不同的API来写记录。
GCP 数据存储API
导入谷歌数据存储
这是代码：
https://gist.github.com/nburn42/d8b488da1d2dc53df63f4c4a32b95def
和数据流 API
从 apache_beam.io.gcp.datastore.v1.datastoreio 导入 WriteToDatastore
这是代码：
https://gist.github.com/nburn42/2c2a06e383aa6b04f84ed31548f1cb09
这是我写了一二十万次后看到的错误。
RPCError：数据存储调用提交 [while 运行 'Write To Datastore/Write Mutation to Datastore'] 失败：错误代码：RESOURCE_EXHAUSTED。消息：超出配额。
我在计算引擎上 运行。
非常感谢任何帮助！
谢谢，
内森
我要求增加配额，google 的人检查了我的帐户以发现问题。
这是他们的回复。
I understand that you want to know what specific quota you are
reaching whenever you try to backup your Cloud SQL to Cloud Datastore.
Upon checking your project, it seems that the problem is that your App
Engine application is at or near its spending limit. As of this time
of writing, the Datastore Write Operations you have executed costed
you 1.10$, which will be refreshed after 5 hours. It can definitely
cause your resources to become unavailable until the daily spending
limit is replenished. Kindly try to increase your spending limit as
soon as possible to avoid service interruption and then run or execute
your datastore write operations.
Give this a shot and let me know what happens. I will be looking
forward to your reply.
这解决了问题。我只需要进入 App Engine 并设置更高的每日支出限额。
希望我上面包含的代码对其他人有所帮助。
我正在尝试将我的数据从 Cloud SQL 移动到 Cloud Datastore。
SQL 数据库中的条目略低于 500 万条。
似乎我每天只能移动超过 100,000 个实体，然后才会出现配额错误。
我无法确定我超过了哪个配额，但是我有指数退避以确保我不会发送得太快。
最终它达到了 5 分钟，与 SQL 服务器的连接中断，但我认为每秒写入配额不是问题所在。在我的 API 页面或 App Engine API 页面中，我没有看到任何其他超出配额的情况。
我试过两种不同的API来写记录。
GCP 数据存储API
导入谷歌数据存储
这是代码：
https://gist.github.com/nburn42/d8b488da1d2dc53df63f4c4a32b95def
和数据流 API
从 apache_beam.io.gcp.datastore.v1.datastoreio 导入 WriteToDatastore
这是代码：
https://gist.github.com/nburn42/2c2a06e383aa6b04f84ed31548f1cb09
这是我写了一二十万次后看到的错误。
RPCError：数据存储调用提交 [while 运行 'Write To Datastore/Write Mutation to Datastore'] 失败：错误代码：RESOURCE_EXHAUSTED。消息：超出配额。
我在计算引擎上 运行。
非常感谢任何帮助！
谢谢，
内森
我要求增加配额，google 的人检查了我的帐户以发现问题。
这是他们的回复。
I understand that you want to know what specific quota you are reaching whenever you try to backup your Cloud SQL to Cloud Datastore.
Upon checking your project, it seems that the problem is that your App Engine application is at or near its spending limit. As of this time of writing, the Datastore Write Operations you have executed costed you 1.10$, which will be refreshed after 5 hours. It can definitely cause your resources to become unavailable until the daily spending limit is replenished. Kindly try to increase your spending limit as soon as possible to avoid service interruption and then run or execute your datastore write operations.
Give this a shot and let me know what happens. I will be looking forward to your reply.
这解决了问题。我只需要进入 App Engine 并设置更高的每日支出限额。
希望我上面包含的代码对其他人有所帮助。