具有延迟的 HTTP 客户端异步调用

Question

我正在使用 httpclient.HTTPRequest 库发送异步请求，但需要在请求之间添加延迟。这意味着假设我配置 RPS（每秒请求数）= 5。然后我每 0.2 发送一个请求，但是是异步的。如何在不等待每个请求响应的情况下异步发送请求。

这是我的代码：

def process_campaign(self, campaign_instance):
  ioloop.IOLoop.current().run_sync(lambda: start_campaign(campaign_instance))

@gen.coroutine
def start_campaign(campaign_instance):
      ...
      while True:
            try:
                log.info("start_campaign() Requests in Queue: {}".format(len(web_requests)))
                web_request = web_requests.pop()
                time.sleep(delay)
                headers = {'Content-Type': 'application/json'}
                request = httpclient.HTTPRequest(auth_username=settings.api_account,
                                                 auth_password=settings.api_password,
                                                 url=settings.api_url,
                                                 body=json.dumps(web_request),
                                                 headers=headers,
                                                 request_timeout=15,
                                                 method="POST")
                response = yield http_client.fetch(request)

            except httpclient.HTTPError, e:
                log.exception("start_campaign() " + str(e))

            except IndexError:
                log.info('start_campaign() Campaign web requests completed. Errors {}'.format(api_errors))
                break

但似乎要等待 HTTP 响应才能继续。

Answer 1

你可以试试：

class WebRequest(RequestHandler):
    def __init__(self, web_request):
        self.delay = 0
        self.web_request = web_request

    @asynchronous
    def post(self):
        IOLoop.instance().add_timeout(self.delay, self._process)

    @gen.coroutine
    def _process(self):
        try:
            http_client = httpclient.AsyncHTTPClient()
            log.info("start_campaign() Web request: {}".format(self.web_request))
            headers = {'Content-Type': 'application/json'}
            request = httpclient.HTTPRequest(auth_username=settings.api_account,
                                             auth_password=settings.api_password,
                                             url=settings.api_url,
                                             body=json.dumps(self.web_request),
                                             headers=headers,
                                             request_timeout=15,
                                             method="POST")
            response = yield http_client.fetch(request)
        except Exception, exception:
            log.exception(exception)

重新使用你的 while 循环：

while True:
            try:
                web_request = web_requests.pop()
                time.sleep(delay)
                client = WebRequest(web_request)
                client.post()

            except IndexError:               
                break

具有延迟的 HTTP 客户端异步调用

HTTP client asynch calls with delay

python

client

multithreading

asynchronous

http