Python Django 截断响应

Question

我创建了一个小的 Django 应用程序并将其部署到 Ubuntu， views.py

from django.shortcuts import render
import json
import psycopg2
import csv
from django.http import HttpResponse
from ratelimit.decorators import ratelimit
def create_connection():
    return psycopg2.connect(database="data",user="ag",host="localhost",port=5432,password='pass')

def execute_query(query, parameters):
    with create_connection() as connection:
        result = []
        cursor = connection.cursor()
        cursor.execute(query, parameters)
        for row in cursor:
            result.append(row)
        for i in range(len(result)):
            result[i] = list(result[i])
            for j in range(len(result[i])):
                result[i][j] = result[i][j] if result[i][j] else ""
        return result

@ratelimit(key='ip', rate='10/m')
def to_csv(request):
    j_body = json.loads(request.GET.get('data', ''))
    query = j_body['query']
    print(query)
    print("****************************************************************")
    parameters = j_body['params']
    headers = j_body['headers']
    rows = execute_query(query, parameters)
    response = HttpResponse(content_type='text_csv')
    response['Content-Disposition'] = 'attachment; filename="result.csv"'
    writer = csv.writer(response)
    writer.writerow(headers)
    writer.writerows(rows)
    return response

此应用程序的主要目标 - 获取 sql 查询和 return csv 响应文件，其大小可以超过 200 mb。我有一个问题 - 我的响应总是被截断为 696 kb（无论 cursor.execute 结果的行数如何）。有例子。您可以看到最后截断的行。

服务器是运行 'python3 manage.py runserver 0.0.0.0:8000' 命令。请帮我解决我的问题。

已更新 我用 Nginx 和 gunicorn 重新部署了它，并尝试了这里建议的解决方案尽管如此，我仍然遇到同样的错误

我按照这篇文章修正了我的代码 - https://docs.djangoproject.com/en/1.10/howto/outputting-csv/#streaming-large-csv-files

#Also changed execute_query method, so it doesn't create redurant copy of response from database
def execute_query(query, parameters):
    with create_connection() as connection:
        result = []
        cursor = connection.cursor()
        cursor.execute(query, parameters)
        return cursor

class Echo(object):
    """An object that implements just the write method of the file-like
    interface.
    """
    def write(self, value):
        """Write the value by returning it, instead of storing in a buffer."""
        return value

@ratelimit(key='ip', rate='10/m')
def to_csv(request):
    try:
        j_body = json.loads(request.GET.get('data', ''))
        query = j_body['query']
        print("****************************************************************")
        parameters = j_body['params']
        headers = j_body['headers']
        cursor = execute_query(query, parameters)
        pseudo_buffer = Echo()
        writer = csv.writer(pseudo_buffer)
        writer.writerow(headers)
        print('Writing began')
        response = StreamingHttpResponse((writer.writerow(row) for row in cursor),
                                     content_type="text/csv")
        response['Content-Disposition'] = 'attachment; filename="result.csv"'
        print('Sending began')
        return response
    except:
        return HttpResponse('Error')

现在一切正常。

Answer 1

听起来您可能需要 StreamingHttpResponse。你试过实施这个吗？您似乎遇到了某种 buffered/chunked 响应的问题。可以使用 gunicorn 或 nginx 之类的东西来增加此缓冲区大小，但这不是真正的解决方案。

Python Django 截断响应

Python Django truncates the response

python

django

wsgi