关闭基于 Python 的 socketserver 3 服务器挂起
Shutdown for socketserver based Python 3 server hangs
我正在 "simple" 服务器上工作,在 Python 3 中使用线程 SocketServer。
为此,我在实施 shutdown 时遇到了很多麻烦。我在 Internet 上找到的下面的代码和关机最初工作,但在通过 telnet 从客户端发送一些命令后停止工作。一些调查告诉我它挂在 threading._shutdown...threading._wait_for_tstate_lock 但到目前为止这还没有响起。
我的研究告诉我,有大约 42 种不同的解决方案、框架等可以在不同的 python 版本中实现这一点。到目前为止,我找不到 python3 的工作方法。例如。我喜欢 telnetsrv
(https://pypi.python.org/pypi/telnetsrv/0.4) 对于 python 2.7(它使用来自 gevent 的 greenlets)但是这个不适用于 python 3。所以如果有更多的 pythonic,std lib方法或可靠的方法,我很想听听!
我目前的赌注是 socketserver,但我还不知道如何处理挂起的服务器。我删除了所有日志语句和大部分功能,因此我可以 post 这个暴露问题的最小服务器:
# -*- coding: utf-8 -*-
import socketserver
import threading
SERVER = None
def shutdown_cmd(request):
global SERVER
request.send(bytes('server shutdown requested\n', 'utf-8'))
request.close()
SERVER.shutdown()
print('after shutdown!!')
#SERVER.server_close()
class service(socketserver.BaseRequestHandler):
def handle(self):
while True:
try:
msg = str(self.request.recv(1024).strip(), 'utf-8')
if msg == 'shutdown':
shutdown_cmd(msg, self.request)
else:
self.request.send(bytes("You said '{}'\n".format(msg), "utf-8"))
except Exception as e:
pass
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def run():
global SERVER
SERVER = ThreadedTCPServer(('', 1520), service)
server_thread = threading.Thread(target=SERVER.serve_forever)
server_thread.daemon = True
server_thread.start()
input("Press enter to shutdown")
SERVER.shutdown()
if __name__ == '__main__':
run()
如果能够从处理程序中停止服务器也很棒(参见 shutdown_cmd)
经过更多研究,我发现了一个使用 asyncio:
的示例
# -*- coding: utf-8 -*-
import asyncio
# after further research I found this relevant europython talk:
# https://www.youtube.com/watch?v=pi49aiLBas8
# * protocols and transport are useful if you do not have tons of socket based code
# * event loop pushes data in
# * transport used to push data back to the client
# found decent sample in book by wrox "professional python"
class ServerProtocol(asyncio.Protocol):
def connection_made(self, transport):
self.transport = transport
self.write('Welcome')
def connection_lost(self, exc):
self.transport = None
def data_received(self, data):
if not data or data == '':
return
message = data.decode('ascii')
command = message.strip().split(' ')[0].lower()
args = message.strip().split(' ')[1:]
#sanity check
if not hasattr(self, 'command_%s' % command):
self.write('Invalid command: %s' % command)
return
# run command
try:
return getattr(self, 'command_%s' % command)(*args)
except Exception as ex:
self.write('Error: %s' % str(ex))
def write(self, msg):
self.transport.write((msg + '\n').encode('ascii', 'ignore'))
def command_shutdown(self):
self.write('Okay. shutting down')
raise KeyboardInterrupt
def command_bye(self):
self.write('bye then!')
self.transport.close()
self.transport = None
if __name__ == '__main__':
loop = asyncio.get_event_loop()
coro = loop.create_server(ServerProtocol, '127.0.0.1', 8023)
asyncio.async(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
据我所知,这是进行此类网络编程最有用的方法。如有必要,可以使用与 uvloop (https://magic.io/blog/uvloop-blazing-fast-python-networking/).
相同的代码来提高性能
shutdown()
按预期工作,服务器已停止接受新连接,但 python 仍在等待活动线程终止。
默认情况下,socketserver.ThreadingMixIn
将创建新线程来处理传入连接,默认情况下,这些线程是非守护线程,因此 python 将等待所有活动的非守护线程终止。
当然,你可以让服务器生成守护线程,这样python就不会等待了:
The ThreadingMixIn class defines an attribute daemon_threads, which indicates whether or not the server should wait for thread termination. You should set the flag explicitly if you would like threads to behave autonomously; the default is False, meaning that Python will not exit until all threads created by ThreadingMixIn have exited.
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
daemon_threads = True
但这不是理想的解决方案,您应该检查为什么线程永远不会终止,通常,服务器应该在没有新数据可用或客户端关闭连接时停止处理连接:
import socketserver
import threading
shutdown_evt = threading.Event()
class service(socketserver.BaseRequestHandler):
def handle(self):
self.request.setblocking(False)
while True:
try:
msg = self.request.recv(1024)
if msg == b'shutdown':
shutdown_evt.set()
break
elif msg:
self.request.send(b'you said: ' + msg)
if shutdown_evt.wait(0.1):
break
except Exception as e:
break
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def run():
SERVER = ThreadedTCPServer(('127.0.0.1', 10000), service)
server_thread = threading.Thread(target=SERVER.serve_forever)
server_thread.daemon = True
server_thread.start()
input("Press enter to shutdown")
shutdown_evt.set()
SERVER.shutdown()
if __name__ == '__main__':
run()
我尝试了两种解决方案来实现在 Python 3 上同时在 Linux 和 Windows 上运行的 tcp 服务器(我尝试了 Windows 7):
- 使用 socketserver(我的问题)- 关机不起作用
- 使用 asyncio(post 编辑了一个答案)- 不适用于 Windows
这两种解决方案都基于网络上的搜索结果。最后我不得不放弃寻找经过验证的解决方案的想法,因为我找不到。因此我实现了自己的解决方案(基于 gevent)。我 post 把它放在这里是因为我希望它能对其他人有所帮助,避免像我那样犯错误。
# -*- coding: utf-8 -*-
from gevent.server import StreamServer
from gevent.pool import Pool
class EchoServer(StreamServer):
def __init__(self, listener, handle=None, spawn='default'):
StreamServer.__init__(self, listener, handle=handle, spawn=spawn)
def handle(self, socket, address):
print('New connection from %s:%s' % address[:2])
socket.sendall(b'Welcome to the echo server! Type quit to exit.\r\n')
# using a makefile because we want to use readline()
rfileobj = socket.makefile(mode='rb')
while True:
line = rfileobj.readline()
if not line:
print("client disconnected")
break
if line.strip().lower() == b'quit':
print("client quit")
break
if line.strip().lower() == b'shutdown':
print("client initiated server shutdown")
self.stop()
break
socket.sendall(line)
print("echoed %r" % line.decode().strip())
rfileobj.close()
srv = EchoServer(('', 1520), spawn=Pool(20))
srv.serve_forever()
另一种关闭服务器的方法是为 serve_forever 调用创建一个 process/thread。
启动server_forever后,只需等待自定义标志触发并在服务器上使用server_close,然后终止进程。
streaming_server = StreamingServer(('', 8000), StreamingHandler)
FLAG_KEEP_ALIVE.value = True
process_serve_forever = Process(target=streaming_server.serve_forever)
process_serve_forever.start()
while FLAG_KEEP_ALIVE.value:
pass
streaming_server.server_close()
process_serve_forever.terminate()
我正在 "simple" 服务器上工作,在 Python 3 中使用线程 SocketServer。
为此,我在实施 shutdown 时遇到了很多麻烦。我在 Internet 上找到的下面的代码和关机最初工作,但在通过 telnet 从客户端发送一些命令后停止工作。一些调查告诉我它挂在 threading._shutdown...threading._wait_for_tstate_lock 但到目前为止这还没有响起。
我的研究告诉我,有大约 42 种不同的解决方案、框架等可以在不同的 python 版本中实现这一点。到目前为止,我找不到 python3 的工作方法。例如。我喜欢 telnetsrv (https://pypi.python.org/pypi/telnetsrv/0.4) 对于 python 2.7(它使用来自 gevent 的 greenlets)但是这个不适用于 python 3。所以如果有更多的 pythonic,std lib方法或可靠的方法,我很想听听!
我目前的赌注是 socketserver,但我还不知道如何处理挂起的服务器。我删除了所有日志语句和大部分功能,因此我可以 post 这个暴露问题的最小服务器:
# -*- coding: utf-8 -*-
import socketserver
import threading
SERVER = None
def shutdown_cmd(request):
global SERVER
request.send(bytes('server shutdown requested\n', 'utf-8'))
request.close()
SERVER.shutdown()
print('after shutdown!!')
#SERVER.server_close()
class service(socketserver.BaseRequestHandler):
def handle(self):
while True:
try:
msg = str(self.request.recv(1024).strip(), 'utf-8')
if msg == 'shutdown':
shutdown_cmd(msg, self.request)
else:
self.request.send(bytes("You said '{}'\n".format(msg), "utf-8"))
except Exception as e:
pass
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def run():
global SERVER
SERVER = ThreadedTCPServer(('', 1520), service)
server_thread = threading.Thread(target=SERVER.serve_forever)
server_thread.daemon = True
server_thread.start()
input("Press enter to shutdown")
SERVER.shutdown()
if __name__ == '__main__':
run()
如果能够从处理程序中停止服务器也很棒(参见 shutdown_cmd)
经过更多研究,我发现了一个使用 asyncio:
的示例# -*- coding: utf-8 -*-
import asyncio
# after further research I found this relevant europython talk:
# https://www.youtube.com/watch?v=pi49aiLBas8
# * protocols and transport are useful if you do not have tons of socket based code
# * event loop pushes data in
# * transport used to push data back to the client
# found decent sample in book by wrox "professional python"
class ServerProtocol(asyncio.Protocol):
def connection_made(self, transport):
self.transport = transport
self.write('Welcome')
def connection_lost(self, exc):
self.transport = None
def data_received(self, data):
if not data or data == '':
return
message = data.decode('ascii')
command = message.strip().split(' ')[0].lower()
args = message.strip().split(' ')[1:]
#sanity check
if not hasattr(self, 'command_%s' % command):
self.write('Invalid command: %s' % command)
return
# run command
try:
return getattr(self, 'command_%s' % command)(*args)
except Exception as ex:
self.write('Error: %s' % str(ex))
def write(self, msg):
self.transport.write((msg + '\n').encode('ascii', 'ignore'))
def command_shutdown(self):
self.write('Okay. shutting down')
raise KeyboardInterrupt
def command_bye(self):
self.write('bye then!')
self.transport.close()
self.transport = None
if __name__ == '__main__':
loop = asyncio.get_event_loop()
coro = loop.create_server(ServerProtocol, '127.0.0.1', 8023)
asyncio.async(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
据我所知,这是进行此类网络编程最有用的方法。如有必要,可以使用与 uvloop (https://magic.io/blog/uvloop-blazing-fast-python-networking/).
相同的代码来提高性能shutdown()
按预期工作,服务器已停止接受新连接,但 python 仍在等待活动线程终止。
默认情况下,socketserver.ThreadingMixIn
将创建新线程来处理传入连接,默认情况下,这些线程是非守护线程,因此 python 将等待所有活动的非守护线程终止。
当然,你可以让服务器生成守护线程,这样python就不会等待了:
The ThreadingMixIn class defines an attribute daemon_threads, which indicates whether or not the server should wait for thread termination. You should set the flag explicitly if you would like threads to behave autonomously; the default is False, meaning that Python will not exit until all threads created by ThreadingMixIn have exited.
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
daemon_threads = True
但这不是理想的解决方案,您应该检查为什么线程永远不会终止,通常,服务器应该在没有新数据可用或客户端关闭连接时停止处理连接:
import socketserver
import threading
shutdown_evt = threading.Event()
class service(socketserver.BaseRequestHandler):
def handle(self):
self.request.setblocking(False)
while True:
try:
msg = self.request.recv(1024)
if msg == b'shutdown':
shutdown_evt.set()
break
elif msg:
self.request.send(b'you said: ' + msg)
if shutdown_evt.wait(0.1):
break
except Exception as e:
break
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def run():
SERVER = ThreadedTCPServer(('127.0.0.1', 10000), service)
server_thread = threading.Thread(target=SERVER.serve_forever)
server_thread.daemon = True
server_thread.start()
input("Press enter to shutdown")
shutdown_evt.set()
SERVER.shutdown()
if __name__ == '__main__':
run()
我尝试了两种解决方案来实现在 Python 3 上同时在 Linux 和 Windows 上运行的 tcp 服务器(我尝试了 Windows 7):
- 使用 socketserver(我的问题)- 关机不起作用
- 使用 asyncio(post 编辑了一个答案)- 不适用于 Windows
这两种解决方案都基于网络上的搜索结果。最后我不得不放弃寻找经过验证的解决方案的想法,因为我找不到。因此我实现了自己的解决方案(基于 gevent)。我 post 把它放在这里是因为我希望它能对其他人有所帮助,避免像我那样犯错误。
# -*- coding: utf-8 -*-
from gevent.server import StreamServer
from gevent.pool import Pool
class EchoServer(StreamServer):
def __init__(self, listener, handle=None, spawn='default'):
StreamServer.__init__(self, listener, handle=handle, spawn=spawn)
def handle(self, socket, address):
print('New connection from %s:%s' % address[:2])
socket.sendall(b'Welcome to the echo server! Type quit to exit.\r\n')
# using a makefile because we want to use readline()
rfileobj = socket.makefile(mode='rb')
while True:
line = rfileobj.readline()
if not line:
print("client disconnected")
break
if line.strip().lower() == b'quit':
print("client quit")
break
if line.strip().lower() == b'shutdown':
print("client initiated server shutdown")
self.stop()
break
socket.sendall(line)
print("echoed %r" % line.decode().strip())
rfileobj.close()
srv = EchoServer(('', 1520), spawn=Pool(20))
srv.serve_forever()
另一种关闭服务器的方法是为 serve_forever 调用创建一个 process/thread。
启动server_forever后,只需等待自定义标志触发并在服务器上使用server_close,然后终止进程。
streaming_server = StreamingServer(('', 8000), StreamingHandler)
FLAG_KEEP_ALIVE.value = True
process_serve_forever = Process(target=streaming_server.serve_forever)
process_serve_forever.start()
while FLAG_KEEP_ALIVE.value:
pass
streaming_server.server_close()
process_serve_forever.terminate()