Python multiprocessing TypeError: join() takes exactly 1 argument (2 given)
Python multiprocessing TypeError: join() takes exactly 1 argument (2 given)
我对多处理做了很多研究!
基本上我是从 API 下载数据并插入数据库。
我创建了一个池并使用 pool.imap 访问下载功能,用结果创建一个元组并一次性插入数据库中。
我反复访问这个函数,有时我的进程挂了!
我已尝试关注 https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map 并访问超时连接。
但是pool.join(超时)returns"TypeError: join() takes exactly 1 argument (2 given)"。我想一个参数是默认的 "self"?
一小段代码:
timeout = 10
pool = Pool(10)
in_tuple = [x for x in pool.imap(multi_details,items) if x is not None]
pool.close()
pool.join(timeout) # from the documentation I should be able to put the timeout in join
writing_to_database(in_tuple)
# function that generate the content for DB
def multi_details(item):
tuple = get_details(item)
return tuple
我看到了创建进程和生成 terminate() 或 join(timeout) 的不同方法,但都没有使用 imap/map - 在我的案例中,这更简单!
与 Process
class 不同,Pool
class 在其 join
方法中不接受 timeout
参数:
https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.join
这就是解决方案!
我没能使用 "next(timeout)" 因为它只解析了几个项目而不是在 运行 整个列表之前停止!
我开始使用apply_async。唯一就是我有种奇怪的感觉,比imap.
慢
功能代码为:
timeout = 1
pool = Pool(10)
for x in items:
try:
res = pool.apply_async(multi_details,(x,)).get(timeout)
except Exception as e:
pass # you can put anything you want but my scope was to skip the things that took too much!
else:
if res is not None: # now this could be a better pythonic way to write this. Any help will be highly appreciated!
in_tuple.append(res)
pool.close()
pool.join()
谢谢,希望有用!
我对多处理做了很多研究! 基本上我是从 API 下载数据并插入数据库。
我创建了一个池并使用 pool.imap 访问下载功能,用结果创建一个元组并一次性插入数据库中。
我反复访问这个函数,有时我的进程挂了! 我已尝试关注 https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map 并访问超时连接。
但是pool.join(超时)returns"TypeError: join() takes exactly 1 argument (2 given)"。我想一个参数是默认的 "self"?
一小段代码:
timeout = 10
pool = Pool(10)
in_tuple = [x for x in pool.imap(multi_details,items) if x is not None]
pool.close()
pool.join(timeout) # from the documentation I should be able to put the timeout in join
writing_to_database(in_tuple)
# function that generate the content for DB
def multi_details(item):
tuple = get_details(item)
return tuple
我看到了创建进程和生成 terminate() 或 join(timeout) 的不同方法,但都没有使用 imap/map - 在我的案例中,这更简单!
与 Process
class 不同,Pool
class 在其 join
方法中不接受 timeout
参数:
https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.join
这就是解决方案!
我没能使用 "next(timeout)" 因为它只解析了几个项目而不是在 运行 整个列表之前停止!
我开始使用apply_async。唯一就是我有种奇怪的感觉,比imap.
慢功能代码为:
timeout = 1
pool = Pool(10)
for x in items:
try:
res = pool.apply_async(multi_details,(x,)).get(timeout)
except Exception as e:
pass # you can put anything you want but my scope was to skip the things that took too much!
else:
if res is not None: # now this could be a better pythonic way to write this. Any help will be highly appreciated!
in_tuple.append(res)
pool.close()
pool.join()
谢谢,希望有用!