Google 任务队列中的递归不遵守速率限制

Question

我在 Google App Engine 中使用递归方法循环访问分页页面。队列中只能同时有一个任务，因为你只能用当前页面获取下一页，即无法知道所有页面同时添加到队列中。

递归函数工作正常，但是，它不遵守 queue.yaml 文件中的说明。

我创建了一个测试示例来说明这个问题。我希望这个过程需要五分钟。原因是 queue.yaml 文件应该每分钟只执行一个任务，而我正在向队列发送五个任务。但是，所有任务都在同时执行。

我可以通过 test() 方法中的睡眠函数破解一些东西来减慢它的速度，但这违背了在 queue.yaml 文件中使用速率限制的目的。

如有任何帮助，我们将不胜感激！

main.py

@app.route('/test',methods=['POST'])
def test():     
    data = json.loads(request.data)
    current_num,max_num = data['current_num'],data['max_num']
    if current_num <= max_num:      
        current_num += 1
        task = taskqueue.add(
                            queue_name='test-queue',
                            method='POST',
                            url='/test',
                            payload=json.dumps({'current_num':current_num,
                                                'max_num':max_num}))
        message = "current_num: {}, max_num: {}".format(current_num,max_num)                
        print message
        return message
    else:
        print 'finished'
        return 'finished'

@app.route('/task')
def test_task():
    max_num = 5

    task = taskqueue.add(
        queue_name='test-queue',
        method='POST',
        url='/test',
        payload=json.dumps({'current_num':0,'max_num':max_num})
    )

    print "current_num: {}, max_num: {}".format(0,max_num)
    return 'submitted task'

queue.yaml

queue:

- name: test-queue
  target: app.main
  rate: 1/m
  bucket_size: 1
  max_concurrent_requests: 1
  retry_parameters:
    min_backoff_seconds: 5
    task_retry_limit: 1

日志文件：

WARNING  2018-04-18 12:24:19,824 urlfetch_stub.py:555] Stripped prohibited headers from URLFetch request: ['content-length']
current_num: 0, max_num: 5
INFO     2018-04-18 12:24:20,299 module.py:835] default: "GET /task HTTP/1.1" 200 14
WARNING  2018-04-18 12:24:20,338 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 1, max_num: 5
INFO     2018-04-18 12:24:20,350 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,350 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 2, max_num: 5
INFO     2018-04-18 12:24:20,362 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,363 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 3, max_num: 5
INFO     2018-04-18 12:24:20,380 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,381 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 4, max_num: 5
INFO     2018-04-18 12:24:20,399 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,400 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 5, max_num: 5
INFO     2018-04-18 12:24:20,415 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,416 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 6, max_num: 5
INFO     2018-04-18 12:24:20,441 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING  2018-04-18 12:24:20,441 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
finished
INFO     2018-04-18 12:24:20,490 module.py:835] default: "POST /test HTTP/1.1" 200 8

注意执行时间：

2018-04-18 12:24:20,299
2018-04-18 12:24:20,350
2018-04-18 12:24:20,362
2018-04-18 12:24:20,380
2018-04-18 12:24:20,399
2018-04-18 12:24:20,415
2018-04-18 12:24:20,441

Answer 1

这是开发服务器的预期行为，如 documentation:

中所述

The development server doesn't respect the rate and bucket-size attributes of your queues. As a result, tasks are executed as close to their ETA as possible. Setting a rate of 0 doesn't prevent tasks from being executed automatically.

Google 任务队列中的递归不遵守速率限制

Recursion in Google Tasks Queues not abiding by rate limiting

python

google-app-engine

task-queue