php 从 json 向 mysql 插入数据运行速度太慢

Question

我有以下代码来读取 JSON 并将结果存储在 DDBB 中。
代码有效，但仅插入 400 条记录就需要一分多钟。
如果我打开 json，它加载速度非常快。
我做错了什么？

    $db = new PDO('', '', '');
    $db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
    if(tableExists($db, 'locations') == 1)
    {
        $sql=$db->prepare("DROP TABLE locations");
        $sql->execute();
    }
    $sql ="CREATE TABLE `locations` (id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, evid INT(6) NOT NULL, place VARCHAR(150), country VARCHAR(150), reg_date TIMESTAMP)" ;
    $db->exec($sql);
    $json = file_get_contents('thejson.php');
    $data = array();
    $data = json_decode($json); 
    foreach ($data as $key => $object) 
    {
        if(is_object($object))
        {
            $id = $object->id;
            $place = $object->name;
            $country = substr(strrchr($object->name, "-"), 2);
            $stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
            $stmt->bindValue(':evid', $id, PDO::PARAM_INT);
            $stmt->bindValue(':places', $place, PDO::PARAM_STR);  
            $stmt->bindValue(':country', $country, PDO::PARAM_STR);      
            $stmt->execute();
        }
    }

Answer 1

尝试用毫秒回显时间戳，看看运行慢的是什么。可能执行 400 个插入查询（包括 opening/closing 个连接）。

Answer 2

影响数据库性能的因素很多，请提供数据库系统、PHP版本及相关硬件的详细信息。

瓶颈可能在：

file_get_contents('thejson.php')

如果 JSON 内容是从远程主机获取的，即 DB 正常运行，则网络速度较慢。

您可能还想考虑搬家：

$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");

退出 foreach 循环。

Answer 3

所以我首先要尝试的两件事是将 prepare 移出循环，并将其包装在事务中：

try {
    $db->beginTransaction();
    $stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");

    foreach ($data as $key => $object) 
    {
        if(is_object($object))
        {
            $id = $object->id;
            $place = $object->name;
            $country = substr(strrchr($object->name, "-"), 2);

            $stmt->bindValue(':evid', $id, PDO::PARAM_INT);
            $stmt->bindValue(':places', $place, PDO::PARAM_STR);  
            $stmt->bindValue(':country', $country, PDO::PARAM_STR);      
            $stmt->execute();
        }
    }

    $db->commit();

} catch (Exception $e) {
    $db->rollBack();
    throw $e;
}

您可以做的另一件事是尝试使用 bindParam 通过引用绑定变量 - 这样您只需要在开始时对每个变量名称调用一次 bindParam ，然后只需覆盖每次迭代时这些变量的值并调用执行。

try {
    $db->beginTransaction();
    $stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
     $id = 0;
     $place = '';
     $country = '';

     $stmt->bindParam(':evid', $id, PDO::PARAM_INT);
     $stmt->bindParam(':places', $place, PDO::PARAM_STR);  
     $stmt->bindParam(':country', $country, PDO::PARAM_STR); 

    foreach ($data as $key => $object) 
    {
        if(is_object($object))
        {
            $id = $object->id;
            $place = $object->name;
            $country = substr(strrchr($object->name, "-"), 2);
            $stmt->execute();
        }
    }

    $db->commit();

} catch (Exception $e) {
    $db->rollBack();
    throw $e;
}

与此类似，您可以通过 execute:

传递值而不是调用 bind*

try {
    $db->beginTransaction();
    $stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");

    foreach ($data as $key => $object) 
    {
        if(is_object($object))
        {
            $params = array(
                ':id' => $object->id,
                ':places' => $object->name,
                ':country' => substr(strrchr($object->name, "-"), 2)
            );

            $stmt->execute($params);
        }
    }

    $db->commit();

} catch (Exception $e) {
    $db->rollBack();
    throw $e;
}

我怀疑使用事务会让你获得性能提升，但我不知道切换绑定方法会有很大的不同。

您最好的选择可能是按照@PavanJiwnani 的建议在单个查询中插入所有记录：

// first we need to compile a structure of only items 
// we will insert with the values properly transformed

$insertData = array_map(function ($object) {
     if (is_object($object)) {
        return array(
            $object->id,
            $object->name,
            substr(strrchr($object->name, "-"), 2)
        );
     } else {
       return false;
     }
}, $data);

// filter out the FALSE values
$insertData = array_filter($insertData);

// get the number of records we have to insert
$nbRecords = count($insertData);

// $records is an array containing a (?,?,?) 
// for each item we want to insert
$records = array_fill(0, $nbRecords, '(?,?,?)');

// now now use sprintf and implode to generate the SQL like:
// INSERT INTO `locations` (evid, place, country) VALUES (?,?,?),(?,?,?),(?,?,?),(?,?,?)
$sql = sprintf(
    'INSERT INTO `locations` (evid, place, country) VALUES %s',   
    implode(',', $records)
);

$stmt = $db->prepare($sql);

// Now we need to flatten our array of insert values as that is what 
// will be expected by execute()
$params = array();
foreach ($insertData as $datum) {
   $params = array_merge($params, $datum);
}

// and finally we attempt to execute
$stmt->execute($params);

php 从 json 向 mysql 插入数据运行速度太慢

php inserting data in mysql from json runs too slowly

php

mysql

json

pdo