关于从返回错误的大型查询返回结果的建议 - 耗尽了 734003200 字节的允许内存大小

Question

我正在尝试 return 将 3 个表连接在一起供用户下载为 CSV 格式的结果，这引发了错误：

Allowed memory size of 734003200 bytes exhausted

这是查询运行:

SELECT *
FROM `tblProgram`
JOIN `tblPlots` ON `tblPlots`.`programID`=`tblProgram`.`pkProgramID`
JOIN `tblTrees` ON `tblTrees`.`treePlotID`=`tblPlots`.`id`

导致错误的代码行是这样的：

$resultsALL=$this->db->query($fullQry);

其中 $fullQry 是上面显示的查询。当我注释掉那一行时，所有运行都没有错误。所以我确定它不是我遗漏的无限循环。

我想知道如何分解查询，以便在不出错的情况下获得结果？这些表目前只有相对少量的数据，最终会更大，所以我认为增加内存大小不是一个好的选择。

我正在使用 CodeIgniter/php/mysql。如果需要，我可以提供更多代码...

感谢您的指导！

Answer 1

Use getUnbufferedRow() for processing large result sets.

getUnbufferedRow()

This method returns a single result row without prefetching the whole result in memory as row() does. If your query has more than one row, it returns the current row and moves the internal data pointer ahead.

$query = $db->query("YOUR QUERY");

while ($row = $query->getUnbufferedRow()) {
    echo $row->title;
    echo $row->name;
    echo $row->body;
}

For use with MySQLi you may set MySQLi’s result mode to MYSQLI_USE_RESULT for maximum memory savings. Use of this is not generally recommended but it can be beneficial in some circumstances such as writing large queries to csv. If you change the result mode be aware of the tradeoffs associated with it.

$db->resultMode = MYSQLI_USE_RESULT; // for unbuffered results

$query = $db->query("YOUR QUERY");

$file = new \CodeIgniter\Files\File(WRITEPATH.'data.csv');

$csv = $file->openFile('w');

while ($row = $query->getUnbufferedRow('array'))
{
    $csv->fputcsv($row);
}

$db->resultMode = MYSQLI_STORE_RESULT; // return to default mode

注：

When using MYSQLI_USE_RESULT all subsequent calls on the same connection will result in error until all records have been fetched or a freeResult() call has been made. The getNumRows() method will only return the number of rows based on the current position of the data pointer. MyISAM tables will remain locked until all the records have been fetched or a freeResult() call has been made.

You can optionally pass ‘object’ (default) or ‘array’ in order to specify the returned value’s type:

$query->getUnbufferedRow();         // object
$query->getUnbufferedRow('object'); // object
$query->getUnbufferedRow('array');  // associative array

freeResult()

It frees the memory associated with the result and deletes the result resource ID. Normally PHP frees its memory automatically at the end of script execution. However, if you are running a lot of queries in a particular script you might want to free the result after each query result has been generated in order to cut down on memory consumption.

$query = $thisdb->query('SELECT title FROM my_table');

foreach ($query->getResult() as $row) {
    echo $row->title;
}

$query->freeResult(); // The $query result object will no longer be available

$query2 = $db->query('SELECT name FROM some_table');

$row = $query2->getRow();
echo $row->name;
$query2->freeResult(); // The $query2 result object will no longer be available

Answer 2

基于：

您也可以尝试使用 LIMIT 子句以块的形式检索数据。

由于您使用的是 CodeIgniter 3，下面是您可以使用的方法。

您可能需要将不同的 $orderBy argument#6 传递给 getChunk(...) 方法，以防万一您的联接表有冲突 id 列名。

即：$this->getChunk(..., ..., ..., 0, 2000, "tblProgram.id");

解决方案：

<?php

class Csv_model extends CI_Model
{
    public function __construct()
    {
        parent::__construct();
        $this->load->database();
    }

    public function index()
    {
        $sql = <<< END
SELECT *
FROM `tblProgram`
JOIN `tblPlots` ON `tblPlots`.`programID`=`tblProgram`.`pkProgramID`
JOIN `tblTrees` ON `tblTrees`.`treePlotID`=`tblPlots`.`id`
END;

        $this->getChunk(function (array $chunk) {
            /*
             * Do something with each chunk here;
             * Do something with each chunk here;
             * log_message('error', json_encode($chunk));
             * */
        }, $this->db, $sql);
    }

    /*
     * Processes a raw SQL query result in chunks sending each chunk to the provided callback function.
     * */
    function getChunk(callable $callback, $DBContext, string $rawSQL = "SELECT 1", int $initialRowOffset = 0, int $maxRows = 2000, string $orderBy = "id")
    {
        $DBContext->query('DROP TEMPORARY TABLE IF EXISTS chunkable');
        $DBContext->query("CREATE TEMPORARY TABLE chunkable AS ( $rawSQL ORDER BY `$orderBy` )");

        do {

            $constrainedSQL = sprintf("SELECT * FROM chunkable ORDER BY `$orderBy` LIMIT %d, %d", $initialRowOffset, $maxRows);
            $queryBuilder = $DBContext->query($constrainedSQL);
            $callback($queryBuilder->result_array());
            $initialRowOffset = $initialRowOffset + $maxRows;

        } while ($queryBuilder->num_rows() === $maxRows);
    }

}

关于从返回错误的大型查询返回结果的建议 - 耗尽了 734003200 字节的允许内存大小

Advice on returning results from a large query returning error- Allowed memory size of 734003200 bytes exhausted

php

sql

codeigniter

codeigniter-3

注：

解决方案：