即使存在死锁，如何强制关闭 ProcessPoolExecutor

Question

我正在尝试使用单独的进程通过并发期货流式传输数据。然而，另一方面，有时对方会停止数据馈送。但是只要我重新启动这个 threadable 然后它就会再次工作。所以我设计了这样的东西，能够在没有干预的情况下保持流数据。

executor = concurrent.futures.ProcessPoolExecutor()
job2 = executor.submit(threadable,list_tmp_replace)
time.sleep(3600)
executor_tmp = executor
executor = concurrent.futures.ProcessPoolExecutor(1)
job2 = executor.submit(threadable, list_tmp_replace_2)
time.sleep(20). #warm up the new process 
executor_tmp.shutdown() #to avoid infinite number of pools in the long run, also threadable itself involves writing data to database. best to avoid duplicate tasks.

但是，我遇到了这个错误

File "/home/ubuntu/anaconda3/lib/python3.8/asyncio/tasks.py", line 280, in __step
  result = coro.send(None)
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/cryptofeed/backends/postgres.py", line 61, in writer
  await self.write_batch(updates)
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/cryptofeed/backends/postgres.py", line 75, in write_batch
  await self.conn.execute(f"INSERT INTO {self.table} VALUES {args_str}")
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/asyncpg/connection.py", line 315, in execute
  return await self._protocol.query(query, timeout)
File "asyncpg/protocol/protocol.pyx", line 338, in query
File "/home/ubuntu/anaconda3/lib/python3.8/asyncio/futures.py", line 260, in __await__
  yield self  # This tells Task to wait for completion.
File "/home/ubuntu/anaconda3/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
  future.result()
File "/home/ubuntu/anaconda3/lib/python3.8/asyncio/futures.py", line 178, in result
  raise self._exception
asyncpg.exceptions.DeadlockDetectedError: deadlock detected
DETAIL:  Process 2576028 waits for ShareLock on transaction 159343645; blocked by process 2545736.
Process 2545736 waits for ShareLock on transaction 159343644; blocked by process 2576028.
HINT:  See server log for query details.

之前，我手动关闭 Python 程序 (ctrl C) 并从终端重新启动它（使用屏幕）。但我希望这样的过程是自动的，由代码本身控制以自动重新连接到数据馈送。无论如何，我是否可以在同一个 python 程序中强制关闭死锁？

Answer 1

您的代码似乎表明可以同时拥有两个 threadable 运行ning 实例，至少在某些重叠期间是这样，并且您无条件地想要运行 a a threadable 的新实例在 3600 秒后到期。这就是我所能继续的，基于此我唯一的建议是你可以考虑切换到使用 multiprocessing.pool.Pool class 作为多处理池，它的优点是 (1) 它是一个不同的 class 与您一直使用的方法相比，没有其他原因可能会产生不同的结果，并且 (2) 与 ProcessPoolExecutor.shutdown 方法不同，Pool.terminate 方法实际上将终止运行立即 ning 作业（ProcessPoolExecutor.shutdown 方法将等待已经开始的作业，即未决的期货，即使您指定了 shutdown(wait=False)，但您没有指定）。

利用 multiprocessing.pool.Pool 的等效代码为：

from multiprocessing import Pool
...

# Only need a pool size of 1:
pool = Pool(1)
job2 = pool.apply_async(threadable, args=(list_tmp_replace,))
time.sleep(3600)
pool_tmp = pool
pool = Pool(1)
job2 = pool.apply_async(threadable, args=(list_tmp_replace_2,))
time.sleep(20) #warm up the new process 
pool_tmp.terminate()
pool_tmp.join()

但是你为什么还要使用池来运行单个进程？考虑使用 multiprocessing.Process 个实例：

from multiprocessing import Process
...

# Only need a pool size of 1:
job2 = Process(targeet=threadable, args=(list_tmp_replace,))
job2.start()
time.sleep(3600)
job2_tmp = job2
job2 = Process(targeet=threadable, args=(list_tmp_replace_2,))
job2.start()
time.sleep(20) #warm up the new process 
job2_tmp.terminate()

即使存在死锁，如何强制关闭 ProcessPoolExecutor

How to force close ProcessPoolExecutor even when there is deadlock

python

concurrent.futures