让 php 抓取工具跳过特定网址

Question

我有这个 GenerateSitemap.php 文件，我可以在其中配置抓取工具，但我不明白如何让抓取工具跳过某些特定的 URL，例如 (https://example.com/noindex-url). I have read this but I can't get my head around it. https://github.com/spatie/laravel-sitemap

namespace Example\Console\Commands;

use Illuminate\Console\Command;
use Spatie\Sitemap\SitemapGenerator;
use Spatie\Sitemap\Tags\Url;
use Spatie\Crawler\Crawler;

class GenerateSitemap extends Command
{
    /**
     * The console command name.
     *
     * @var string
     */
    protected $signature = 'sitemap:generate';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Generate the sitemap.';

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {   
        $siteURL = 'https://example.com';
        SitemapGenerator::create($siteURL)
            ->configureCrawler(function (Crawler $crawler) {
                $crawler->ignoreRobots();
            })
            ->hasCrawled(function (Url $url) {
                if ((string)$url->path() === '/') {
                    return;
                }

                $this->output->writeln('Crawled: ' . (string)$url->path());

                return $url;
            })
            ->writeToFile(public_path('sitemap.xml'));

Answer 1

if ((string)$url->path() === '/noindex-url') {
    return;
}

做到了！

所以这是现在的样子：

namespace Example\Console\Commands;

use Illuminate\Console\Command;
use Spatie\Sitemap\SitemapGenerator;
use Spatie\Sitemap\Tags\Url;
use Spatie\Crawler\Crawler;

class GenerateSitemap extends Command
{
    /**
     * The console command name.
     *
     * @var string
     */
    protected $signature = 'sitemap:generate';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Generate the sitemap.';

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {   
        $siteURL = 'https://example.com';
        SitemapGenerator::create($siteURL)
            ->configureCrawler(function (Crawler $crawler) {
                $crawler->ignoreRobots();
            })
            ->hasCrawled(function (Url $url) {
                if ((string)$url->path() === '/') {
                    return;
                }

                if ((string)$url->path() === '/noindex-url') {
                    return;
                }
                $this->output->writeln('Crawled: ' . (string)$url->path());

                return $url;
            })
            ->writeToFile(public_path('sitemap.xml'));

让 php 抓取工具跳过特定网址

Getting php crawler to skip specific urls

php

laravel

web-crawler

sitemap