TimeoutError QueuePool limit of size x overflow xy reached，连接超时，超时 30
TimeoutError QueuePool limit of size x overflow xy reached, connection timed out, timeout 30
这是我得到的错误：
RemoteError：远程错误：达到 TimeoutError QueuePool limit of size x overflow xy，连接超时，超时 30（此错误的背景位于：http://sqlalche.me/e/3o7r）
元数据代理存在问题。一个计算节点承载多个虚拟机，虚拟机无法获取 ip。我尝试在 dhcp 元数据服务器上重新启动这两个服务，也尝试重新启动我们的 rabbitmq 服务器。还重新启动了 neutron 服务器，但问题仍然存在并且虚拟机已关闭。
像这样的错误：
2019-10-24 10:34:47.794 86154 ERROR neutron.agent.metadata.agent
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [-] Unexpected error.: RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 89, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 116, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.func(target_self, *args, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ip_address=remote_address)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.plugin_rpc.get_ports(self.context, filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 72, in get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return cctxt.call(context, 'get_ports', filters=filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._original_context.call(ctxt, method, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent retry=self.retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent retry=retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent call_monitor_timeout, retry=retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent raise result
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/metadata_rpc.py", line 43, in get_ports\n return self.plugin.get_ports(context, filters=filters)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped\n return method(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1435, in get_ports\n items = [self._make_port_dict(c, fields) for c in query]\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, in _execute_and_instances\n close_with_result=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2955, in _get_bind_args\n **kw\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2937, in _connection_from_session\n conn = self.session.connection(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection\n execution_options=execution_options)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind\n conn = engine.contextual_connect(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect\n self._wrap_pool_connect(self.pool.connect, None),\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect\n return fn()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect\n return _ConnectionFairy._checkout(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get\n (self.size(), self.overflow(), self._timeout), code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)\n'].
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent
2019-10-24 10:35:04.436 86151 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
2019-10-24 10:35:04.637 86156 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
2019-10-24 10:35:05.339 86149 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
我应该怎么做才能解决它:( ?
或者我该如何微调这个 sqlalchemy？老实说，我不知道这是什么。
解决方案是添加更多的 neutron 服务器。
内存和 cpu 增加没有帮助。
这是我得到的错误： RemoteError：远程错误：达到 TimeoutError QueuePool limit of size x overflow xy，连接超时，超时 30（此错误的背景位于：http://sqlalche.me/e/3o7r）
元数据代理存在问题。一个计算节点承载多个虚拟机，虚拟机无法获取 ip。我尝试在 dhcp 元数据服务器上重新启动这两个服务，也尝试重新启动我们的 rabbitmq 服务器。还重新启动了 neutron 服务器，但问题仍然存在并且虚拟机已关闭。
像这样的错误：
2019-10-24 10:34:47.794 86154 ERROR neutron.agent.metadata.agent
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [-] Unexpected error.: RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 89, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 116, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.func(target_self, *args, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ip_address=remote_address)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.plugin_rpc.get_ports(self.context, filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 72, in get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return cctxt.call(context, 'get_ports', filters=filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._original_context.call(ctxt, method, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent retry=self.retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent retry=retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent call_monitor_timeout, retry=retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent raise result
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/metadata_rpc.py", line 43, in get_ports\n return self.plugin.get_ports(context, filters=filters)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped\n return method(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1435, in get_ports\n items = [self._make_port_dict(c, fields) for c in query]\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, in _execute_and_instances\n close_with_result=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2955, in _get_bind_args\n **kw\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2937, in _connection_from_session\n conn = self.session.connection(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection\n execution_options=execution_options)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind\n conn = engine.contextual_connect(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect\n self._wrap_pool_connect(self.pool.connect, None),\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect\n return fn()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect\n return _ConnectionFairy._checkout(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get\n (self.size(), self.overflow(), self._timeout), code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)\n'].
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent
2019-10-24 10:35:04.436 86151 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
2019-10-24 10:35:04.637 86156 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
2019-10-24 10:35:05.339 86149 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
我应该怎么做才能解决它:( ?
或者我该如何微调这个 sqlalchemy？老实说，我不知道这是什么。
解决方案是添加更多的 neutron 服务器。 内存和 cpu 增加没有帮助。