"Can't open a new thread" Heroku 上的错误

Question

我正在编写一个 Python 用于网络抓取的脚本，并尝试实施多线程方法以更快地完成工作。我正在使用 ThreadPoolExecutor 来设置最大线程数。

from threading import current_thread
from concurrent.futures import ThreadPoolExecutor, as_completed

MAX_THREADS = 100

def process_user(user):
    # makes one or more network requests and returns the results
    return {"count": 123}

users = get_users_from_database() # gets user records from the database

with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    futures = [executor.submit(process_user, row) for row in users]
    batch = []
    for index, future in enumerate(as_completed(futures)):
        result = future.result()
        batch.append(result)

        # store full batch or when there are no more left to store...
        if (len(batch) >= BATCH_SIZE) or (index + 1 >= len(futures)):
            insert_users(batch) # stores in the database
            batch = []

Heroku says 他们的免费层 dyno 可以运行最多 256 个线程。

然而，当我运行免费层级 Heroku 服务器上的脚本时，似乎运行最多 10 个线程没问题，但是当尝试运行更多线程时，该脚本仅运行出现 "RuntimeError: can't start new thread" 个错误。

有没有办法使用10个以上的线程？我需要升级等级吗？感谢您的建议。

Answer 1

似乎是内存问题。扩大测功机允许使用更多线程。

"Can't open a new thread" Heroku 上的错误

"Can't open a new thread" errors on Heroku

python

multithreading

heroku

threadpool

python-multithreading