RQ: redis.exceptions.ResponseError: Command # 3 ... of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'
RQ: redis.exceptions.ResponseError: Command # 3 ... of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'
我 运行 我的 django 应用程序中的后台任务是 RQ，因为它应该是完成工作的最简单方法之一。该任务包括检查一些 API 是否有任何信息已更新，并将任何新信息插入我自己的数据库中。
直到几天前，它工作正常，但我现在遇到无法解决的错误。
从错误消息的最后几行开始：
OOM command not allowed when used memory > 'maxmemory'
一开始我以为我传递给worker的数据太多了。但是，我最终减少了传递给具有 5 个键值对的单个字典的数据，但我仍然收到错误消息（请参阅底部的完整消息）。然而，直到上周，我才通过了 20 多本词典，每本词典都有更多的元素，而且工作正常。
我查了here and here，不过好像和我的问题不一样
知道我为什么会收到此错误以及如何解决它吗？
错误信息：
Internal Server Error: /en/results/
Traceback (most recent call last):
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2518, in _execute_transaction
response = self.parse_response(connection, '_')
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
self, connection, command_name, **options)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ExecAbortError: Transaction discarded because of previous errors.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 149, in get_response
response = self.process_exception_by_middleware(e, request)
File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 147, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/my/path/app/results/views.py", line 259, in results_list
price_update_to_queue(outbound_trips)
File "/my/path/app/results/views.py", line 331, in price_update_to_queue
queue.enqueue(update_prices, queries)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 274, in enqueue
job_id=job_id, at_front=at_front, meta=meta)
File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 60, in enqueue_call
return self.original_enqueue_call(*args, **kwargs)
File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 56, in original_enqueue_call
return super(DjangoRQ, self).enqueue_call(*args, **kwargs)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 227, in enqueue_call
job = self.enqueue_job(job, at_front=at_front)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 298, in enqueue_job
pipe.execute()
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2626, in execute
return execute(conn, stack, raise_on_error)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2523, in _execute_transaction
raise errors[0][1]
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2510, in _execute_transaction
self.parse_response(connection, '_')
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
self, connection, command_name, **options)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ResponseError: Command # 3 (HMSET b'rq:job:8df23896-d52d-4585-aa60-9f5f9a39292a' created_at 2017-06-26T13:13:33Z timeout 500 origin default data b'\x80\x04\x95\xff\x00\x00\x00\x00\x00\x00\x00(\x8c\x1bresults.tasks.update_prices\x94N]\x94}\x94(\x8c\x03arr\x94\x8c\x0bDE-CGN-00-0\x94\x8c\x03dep\x94\x8c\x0bDE-BER-02-0\x94\x8c\x03rtn\x94K\x00\x8c\x06seller\x94\x8c\x04KIWI\x94\x8c\x06dep_dt\x94\x8c\x08datetime\x94\x8c\x08datetime\x94\x93\x94C\n\x07\xe1\x07\x18\x00\x00\x00\x00\x00\x00\x94\x8c\x0bpsycopg2.tz\x94\x8c\x13FixedOffsetTimezone\x94\x93\x94KxN\x86\x94R\x94}\x94\x8c\x07_offset\x94\x8c\x08datetime\x94\x8c\ttimedelta\x94\x93\x94K\x00M \x1cK\x00\x87\x94R\x94sb\x86\x94R\x94ua\x85\x94}\x94t\x94.' description results.tasks.update_prices([{'arr': 'DE-CGN-00-0', 'dep': 'DE-BER-02-0', 'rtn': 0, 'seller': 'KIWI', 'dep_dt': datetime.datetime(2017, 7, 24, 0, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=120, name=None))}]) enqueued_at 2017-06-26T13:13:33Z status queued) of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'.
[26/Jun/2017 13:13:33] "GET /en/results/?From=Berlin&To=Cologne&OutboundDate=24-07-2017&ReturnDate= HTTP/1.1" 500 174583
这就是我完成任务的方式
#views.py
import django_rq
from .tasks import update_prices
...
queue = django_rq.get_queue('default')
queue.enqueue(update_prices, queries)
和
#tasks.py
from django_rq import job
@job
def update_prices(queries):
...
return queries_to_background(queries)
之前遇到过类似的问题。我的 Flask 应用程序部署到 Heroku 上，我在 Heroku 上使用 redistogo，免费计划的内存非常少，这导致我出现超出最大内存的错误。然后换成heroku-redis，问题解决
我 运行 我的 django 应用程序中的后台任务是 RQ，因为它应该是完成工作的最简单方法之一。该任务包括检查一些 API 是否有任何信息已更新，并将任何新信息插入我自己的数据库中。
直到几天前，它工作正常，但我现在遇到无法解决的错误。
从错误消息的最后几行开始：
OOM command not allowed when used memory > 'maxmemory'
一开始我以为我传递给worker的数据太多了。但是，我最终减少了传递给具有 5 个键值对的单个字典的数据，但我仍然收到错误消息（请参阅底部的完整消息）。然而，直到上周，我才通过了 20 多本词典，每本词典都有更多的元素，而且工作正常。
我查了here and here，不过好像和我的问题不一样
知道我为什么会收到此错误以及如何解决它吗？
错误信息：
Internal Server Error: /en/results/
Traceback (most recent call last):
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2518, in _execute_transaction
response = self.parse_response(connection, '_')
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
self, connection, command_name, **options)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ExecAbortError: Transaction discarded because of previous errors.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 149, in get_response
response = self.process_exception_by_middleware(e, request)
File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 147, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/my/path/app/results/views.py", line 259, in results_list
price_update_to_queue(outbound_trips)
File "/my/path/app/results/views.py", line 331, in price_update_to_queue
queue.enqueue(update_prices, queries)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 274, in enqueue
job_id=job_id, at_front=at_front, meta=meta)
File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 60, in enqueue_call
return self.original_enqueue_call(*args, **kwargs)
File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 56, in original_enqueue_call
return super(DjangoRQ, self).enqueue_call(*args, **kwargs)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 227, in enqueue_call
job = self.enqueue_job(job, at_front=at_front)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 298, in enqueue_job
pipe.execute()
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2626, in execute
return execute(conn, stack, raise_on_error)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2523, in _execute_transaction
raise errors[0][1]
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2510, in _execute_transaction
self.parse_response(connection, '_')
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
self, connection, command_name, **options)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ResponseError: Command # 3 (HMSET b'rq:job:8df23896-d52d-4585-aa60-9f5f9a39292a' created_at 2017-06-26T13:13:33Z timeout 500 origin default data b'\x80\x04\x95\xff\x00\x00\x00\x00\x00\x00\x00(\x8c\x1bresults.tasks.update_prices\x94N]\x94}\x94(\x8c\x03arr\x94\x8c\x0bDE-CGN-00-0\x94\x8c\x03dep\x94\x8c\x0bDE-BER-02-0\x94\x8c\x03rtn\x94K\x00\x8c\x06seller\x94\x8c\x04KIWI\x94\x8c\x06dep_dt\x94\x8c\x08datetime\x94\x8c\x08datetime\x94\x93\x94C\n\x07\xe1\x07\x18\x00\x00\x00\x00\x00\x00\x94\x8c\x0bpsycopg2.tz\x94\x8c\x13FixedOffsetTimezone\x94\x93\x94KxN\x86\x94R\x94}\x94\x8c\x07_offset\x94\x8c\x08datetime\x94\x8c\ttimedelta\x94\x93\x94K\x00M \x1cK\x00\x87\x94R\x94sb\x86\x94R\x94ua\x85\x94}\x94t\x94.' description results.tasks.update_prices([{'arr': 'DE-CGN-00-0', 'dep': 'DE-BER-02-0', 'rtn': 0, 'seller': 'KIWI', 'dep_dt': datetime.datetime(2017, 7, 24, 0, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=120, name=None))}]) enqueued_at 2017-06-26T13:13:33Z status queued) of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'.
[26/Jun/2017 13:13:33] "GET /en/results/?From=Berlin&To=Cologne&OutboundDate=24-07-2017&ReturnDate= HTTP/1.1" 500 174583
这就是我完成任务的方式
#views.py
import django_rq
from .tasks import update_prices
...
queue = django_rq.get_queue('default')
queue.enqueue(update_prices, queries)
和
#tasks.py
from django_rq import job
@job
def update_prices(queries):
...
return queries_to_background(queries)
之前遇到过类似的问题。我的 Flask 应用程序部署到 Heroku 上，我在 Heroku 上使用 redistogo，免费计划的内存非常少，这导致我出现超出最大内存的错误。然后换成heroku-redis，问题解决