Python + BigQuery + ResponseNotReady()

Question

我正在使用 "bigquery_service = build('bigquery', 'v2', http=http)"，它在我的笔记本电脑上正确执行，但是当我在服务器上执行此查询时，出现以下错误。

File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "QueryAPI.py", line 117, in localMain
    exportDataToGCS(canonicalDate);
  File "QueryAPI.py", line 177, in exportDataToGCS
    bigquery_service = build('bigquery', 'v2', http=http)
  File "/homeBigQuery/src/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/homeBigQuery/src/googleapiclient/discovery.py", line 198, in build
    resp, content = http.request(requested_url)
  File "/homeBigQuery/src/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/homeBigQuery/src/oauth2client/client.py", line 538, in new_request
    redirections, connection_type)
  File "/homeBigQuery/src/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/homeBigQuery/src/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/homeBigQuery/src/httplib2/__init__.py", line 1286, in _conn_request
    response = conn.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 980, in getresponse
    raise ResponseNotReady()

Answer 1

您是否尝试删除 "http=http" 参数？

这里有一个使用BQ的例子API:

import httplib2

from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials

# REPLACE WITH YOUR Project ID
PROJECT_NUMBER = 'XXXXXXXXXXX'
# REPLACE WITH THE SERVICE ACCOUNT EMAIL FROM GOOGLE DEV CONSOLE
SERVICE_ACCOUNT_EMAIL = 'XXXXX@developer.gserviceaccount.com'

# OBTAIN THE KEY FROM THE GOOGLE APIs CONSOLE
# More instructions here: http://goo.gl/w0YA0
f = file('key.p12', 'rb')
key = f.read()
f.close()

credentials = SignedJwtAssertionCredentials(
    SERVICE_ACCOUNT_EMAIL,
    key,
    scope='https://www.googleapis.com/auth/bigquery')

http = httplib2.Http()
http = credentials.authorize(http)

service = build('bigquery', 'v2')
datasets = service.datasets()
response = datasets.list(projectId=PROJECT_NUMBER).execute(http)

print 'Dataset list:'
for dataset in response['datasets']:
  print '%s' % dataset['datasetReference']['datasetId']

Answer 2

你可以试试this 3rd party library for BigQuery。它是 Google BigQuery API.

的薄包装

登录 BigQuery：

import bigquery as bq

with open('my_key.pem', 'r') as key:
    key_content = key.read()

bq_client = bq.get_client(project_id='foo',
                          service_account='bar',
                          private_key=key_content)

提交加载作业：

schema = [{'name':'foo', 'type':'STRING'}]
job = bq_client.import_data_from_uris(source_uris=['gs://foo/bar'],
                                        dataset='foo',
                                        table='bar',
                                        schema=schema,
                                        source_format=bq.client.JOB_FORMAT_CSV,
                                        write_disposition=bq.client.JOB_WRITE_TRUNCATE,
                                        field_delimiter=',',
                                        skip_leading_rows=1)


try:
    job_resource = bq_client.wait_for_job(job, timeout=60)        
except bq.errors.BigQueryTimeoutException:
    logging.critical('BigQuery loading timeout.')

Answer 3

通过为我运行所在的服务器指定代理服务器主机和端口解决了这个问题。在凭据的帮助下启动授权的http服务对象时，我将代码修改如下：

之前：

http = httplib2.Http()
http = credentials.authorize(http)
service = build('bigquery', 'v2', http=http)

修复：

http = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP, 'your-proxy-host-name', your-proxy-port))
http = credentials.authorize(http)
service = build('bigquery', 'v2', http=http)

Python + BigQuery + ResponseNotReady()

Python + BigQuery + ResponseNotReady()

python

google-bigquery