将多处理模块与 BaseHTTPRequestHandler 一起使用

Question

我正在尝试使用 python 多处理模块运行另一个使用 http.server.BaseHTTPRequestHandler 模块的线程中的服务器。我被卡住了，运行正在陷入“_thread.lock”问题。

我不想使用线程模块，因为我宁愿使用真正的多线程和多处理模块。

如果有人知道我做错了什么，或者可以指出一个好的库供我使用，那就太棒了。

import multiprocessing
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

if __name__ == '__main__':
  httpd = ThreadingHTTPServer(('localhost', 4433), BaseHTTPRequestHandler)

  manager = multiprocessing.Manager()
  manager.http_server = httpd
  running_server = multiprocessing.Process(target=manager.http_server.serve_forever)
  running_server.start()

堆栈跟踪：

File "/Users/redacted/python/test2/test1.py", line 10, in <module>
    running_server.start()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

Answer 1

Python 在使用多进程模块时使用 pickle 将对象传递给另一个进程。在您的情况下，httpserver 中使用的线程锁不可腌制。所以它报告错误。您可以做的是完全像这样在另一个进程中启动 http 服务器：

import multiprocessing
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

def startServer():
    httpd = ThreadingHTTPServer(('localhost', 4433), BaseHTTPRequestHandler)
    httpd.serve_forever()
    
if __name__ == '__main__':
  manager = multiprocessing.Manager()
  running_server = multiprocessing.Process(target=startServer)
  running_server.start()

此外，您可能想尝试 4433 以外的其他端口。我无法连接到我的 windows 机器上的这个端口。但是如果我使用 8000 一切正常。

将多处理模块与 BaseHTTPRequestHandler 一起使用

Using multiprocessing module with BaseHTTPRequestHandler

python

python-multiprocessing

python-multithreading

httpserver