如何优化这段代码？我想解决 DeadlineExceededError 问题

Question

我想从数据存储中提取数据。我对编程比较陌生，所以在页面上打印结果对我来说似乎最简单。不幸的是，大多数时候（并非总是如此！），我会收到以下错误。你能帮帮我吗？

错误信息：

500 Server Error The server encountered an error and could not complete your request.

日志：

http://pastebin.com/C2LQCArn

DeadlineExceededError

我的理解是我的查询太慢了： https://cloud.google.com/appengine/articles/deadlineexceedederrors

如何加快速度？

class ExpData(db.Model):

        # According to Datastore Admin, there are 20,557 entities in this model.

        created = db.DateTimeProperty(auto_now_add = True)
        username = db.StringProperty(required=True)

        task1n = db.StringProperty(required=False, indexed=False)
        task1a = db.StringProperty(required=False, indexed=False)
        task1c = db.StringProperty(required=False, indexed=False)

        task2n = db.StringProperty(required=False, indexed=False)
        task2a = db.StringProperty(required=False, indexed=False)
        task2c = db.StringProperty(required=False, indexed=False)

        task3n = db.StringProperty(required=False, indexed=False)
        task3a = db.StringProperty(required=False, indexed=False)
        task3c = db.StringProperty(required=False, indexed=False)

        task4n = db.StringProperty(required=False, indexed=False)
        task4a = db.StringProperty(required=False, indexed=False)
        task4c = db.StringProperty(required=False, indexed=False)

        q1 = db.StringProperty(required=False)
        q2 = db.StringProperty(required=False)

        ecus = db.StringProperty(required=False)


    class Viewer(Handler):
        def get(self):

            test = ExpData.all().order('-created')
            expDataDict = {}
            expDataList = []
            for entity in test:

                if (entity.q2 == "-1") or (entity.q2 == ""):
                    continue
                else:
                    expDataList.append(entity.created)
                    expDataList.append(entity.username)
                    expDataList.append(entity.ecus)
                    expDataList.append(entity.q1)
                    expDataList.append(entity.q2)
                    expDataList.append(entity.task1c)
                    expDataList.append(entity.task2c)
                    expDataList.append(entity.task3c)
                    expDataList.append(entity.task4c)
                    expDataDict[entity.created] = {}
                    expDataDict[entity.created]["username"] = entity.username
                    expDataDict[entity.created]["ecus"] = entity.ecus
                    expDataDict[entity.created]["q1"] = entity.q1
                    expDataDict[entity.created]["q2"] = entity.q2
                    expDataDict[entity.created]["task1c"] = entity.task1c
                    expDataDict[entity.created]["task2c"] = entity.task2c
                    expDataDict[entity.created]["task3c"] = entity.task3c
                    expDataDict[entity.created]["task4c"] = entity.task4c

            self.render("viewer.html", otherDict = expDataDict)

非常感谢您的帮助。

Answer 1

首先，尝试在检索结果 (entity.q2 == "-1") or (entity.q2 == "") 后将这些条件添加到您的查询中，而不是运行它们。其次，您应该查看结果的分页以查看那么多数据。您应该查看 Query Cursors 而不是

from google.appengine.api import memcache
from google.appengine.ext import db

# class Person(db.Model): ...

# Start a query for all Person entities
people = Person.all()

# If the application stored a cursor during a previous request, use it
person_cursor = memcache.get('person_cursor')
if person_cursor:
  people.with_cursor(start_cursor=person_cursor)

# Iterate over the results
for person in people:
  # Do something

# Get updated cursor and store it for next time
person_cursor = people.cursor()
memcache.set('person_cursor', person_cursor)

Answer 2

利用延迟。 https://cloud.google.com/appengine/articles/deferred

然后从您的延迟任务中创建提取物并将其作为电子邮件发送给您。

对于延迟任务，请求限制为 10 分钟。如果您在 10 分钟内提取完成，则很好。

伪代码：

为后台任务创建模块extract_task.py。

''' extract_task.py '''

from StringIO import StringIO
from google.appengine.api import mail

def extract():
    output = StringIO()
    test = ExpData.all().order('-created')
    for entity in test:
        # Just extracting 3 fileds per row for the example.
        output.write('%s,%s,%s\n' % (entity.created, entity.username, entity.ecus))

    mail.send_mail(sender="youremail@example.com",
                   to="Albert Johnson <youremail@example.com>",
                   subject="Extract",
                   body="""
Extract attached.
""",
                  attachments=[('extract.csv', output)])

写一个新的处理程序：

from extract_task import extract # import the module cretaed above.
from google.appengine.ext import deferred

class Viewer1(Handler):
    def get(self):
        deferred.defer(extract)
        self.response.write('Exatrcat will be emailed to you soon.')

app.yaml

将以下条目添加到 app.yaml 的内置部分：

- deferred: on

以及同一文件的处理程序部分的以下条目：

- url: /_ah/queue/deferred
    script: google.appengine.ext.deferred.deferred.application
    login: admin

如何优化这段代码？我想解决 DeadlineExceededError 问题

how to optimize this code? I want to fix the DeadlineExceededError issue

python

google-app-engine

google-cloud-datastore