脚本将部分内容写入 csv 文件

Question

我在 php 中编写了一个脚本，用于从网页中抓取 titles 及其 links 并将它们相应地写入 csv 文件。当我处理一个分页网站时，只有最后一页的内容保留在 csv 文件中，其余的都被覆盖了。我尝试使用写作模式 w。但是，当我使用 append a 执行相同操作时，我会在该 csv 文件中找到所有数据。

由于 appending 和 writing 数据使 csv 文件多次打开和关闭（因为我可能错误地应用了循环），脚本变得效率低下且耗时。

我怎样才能有效地做同样的事情，当然还要使用（写作）w 模式？

这是我到目前为止写的：

<?php
include "simple_html_dom.php";
$link = "https://whosebug.com/questions/tagged/web-scraping?page="; 

function get_content($url)
    {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        $htmlContent = curl_exec($ch);
        curl_close($ch);
        $dom = new simple_html_dom();
        $dom->load($htmlContent);
        $infile = fopen("itemfile.csv","a");
        foreach($dom->find('.question-summary') as $file){
            $itemTitle = $file->find('.question-hyperlink', 0)->innertext;
            $itemLink = $file->find('.question-hyperlink', 0)->href;
            echo "{$itemTitle},{$itemLink}<br>";
            fputcsv($infile,[$itemTitle,$itemLink]);
        }
        fclose($infile);
    }
for($i = 1; $i<10; $i++){
        get_content($link.$i);
    }
?>

Answer 1

如果您不想多次打开和关闭文件，请将打开脚本移到 for 循环之前，然后将其关闭：

function get_content($url, $inifile)
{
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $htmlContent = curl_exec($ch);
    curl_close($ch);
    $dom = new simple_html_dom();
    $dom->load($htmlContent);
    foreach($dom->find('.question-summary') as $file){
        $itemTitle = $file->find('.question-hyperlink', 0)->innertext;
        $itemLink = $file->find('.question-hyperlink', 0)->href;
        echo "{$itemTitle},{$itemLink}<br>";
        fputcsv($infile,[$itemTitle,$itemLink]);
    }
}

$infile = fopen("itemfile.csv","w");

for($i = 1; $i<10; $i++) {
    get_content($link.$i, $inifile);
}

fclose($infile);
?>

Answer 2

我会考虑不将结果回显或写入 get_content 函数中的文件。我会重写它，让它只 get 内容，这样我就可以按我喜欢的方式处理提取的数据。像这样（请阅读代码注释）：

<?php
include "simple_html_dom.php";
$link = "https://whosebug.com/questions/tagged/web-scraping?page="; 

// This function does not write data to a file or print it. It only extracts data
// and returns it as an array.
function get_content($url)
    {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        $htmlContent = curl_exec($ch);
        curl_close($ch);
        $dom = new simple_html_dom();
        $dom->load($htmlContent);
        // We don't need the following line anymore
        // $infile = fopen("itemfile.csv","a");
        // We will collect extracted data in an array
        $result = [];
        foreach($dom->find('.question-summary') as $file){
            $itemTitle = $file->find('.question-hyperlink', 0)->innertext;
            $itemLink = $file->find('.question-hyperlink', 0)->href;
            $result []= [$itemTitle, $itemLink];
            // echo "{$itemTitle},{$itemLink}<br>";
            // No need to write to file, so we don't need the following as well
            // fputcsv($infile,[$itemTitle,$itemLink]);
        }
        // No files opened, so the following line is no more required
        // fclose($infile);
        // Return extracted data from this specific URL
        return $result;
    }
// Merge all results (result for each url with different page parameter
// With a little refactoring, get_content() can handle this as well
$result = [];
for($page = 1; $page < 10; $page++){
    $result = array_merge($result, get_content($link.$page));
}
// Now do whatever you want with $result. Like writing its values to a file, or print it, etc.
// You might want to write a function for this
$outputFile = fopen("itemfile.csv","a");
foreach ($result as $row) {
    fputcsv($outputFile, $row);
}
fclose($outputFile);

?>

脚本将部分内容写入 csv 文件

Script writes partial content to a csv file

php

csv

curl

simple-html-dom

web-scraping