Jsoup reddit 刮板 429 错误

Question

所以我尝试使用 jsoup 来抓取 Reddit 上的图像，但是当我抓取某些 subreddits 时，例如 /r/wallpaper，我收到 429 错误并且想知道如何解决这个问题。完全理解这段代码很糟糕，这是一个非常菜鸟的问题，但我对此完全陌生。不管怎样：

import java.io.IOException;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;

import java.io.*;
import java.net.URL;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.io.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.jsoup.Jsoup;

import org.jsoup.nodes.Attributes;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import java.net.URL;
import java.util.Scanner;

public class javascraper{

public static void main (String[]args) throws MalformedURLException
{
    Scanner scan = new Scanner (System.in);
    System.out.println("Where do you want to store the files?");
    String folderpath = scan.next();
    System.out.println("What subreddit do you want to scrape?");
    String subreddit = scan.next();
    subreddit = ("http://reddit.com/r/" + subreddit);
    new File(folderpath + "/" + subreddit).mkdir();

    //test

    try{
        //gets http protocol
        Document doc = Jsoup.connect(subreddit).timeout(0).get();

        //get page title
        String title = doc.title();
        System.out.println("title : " + title);

        //get all links
        Elements links = doc.select("a[href]");

        for(Element link : links){

            //get value from href attribute
            String checkLink = link.attr("href");
            Elements images = doc.select("img[src~=(?i)\.(png|jpe?g|gif)]");
            if (imgCheck(checkLink)){ // checks to see if img link j
                System.out.println("link : " + link.attr("href"));
                downloadImages(checkLink, folderpath);
            }
        }
    }
    catch (IOException e){
        e.printStackTrace();
    }
}

public static boolean imgCheck(String http){
    String png = ".png";
    String jpg = ".jpg";
    String jpeg = "jpeg"; // no period so checker will only check last four characaters
    String gif = ".gif";
    int length = http.length();

    if (http.contains(png)|| http.contains("gfycat") || http.contains(jpg)|| http.contains(jpeg) || http.contains(gif)){
        return true;
    }
    else{
        return false;
    }
}

private static void downloadImages(String src, String folderpath) throws IOException{
    String folder = null;

    //Exctract the name of the image from the src attribute

    int indexname = src.lastIndexOf("/");

    if (indexname == src.length()) {
        src = src.substring(1, indexname);
    }
    indexname = src.lastIndexOf("/");

    String name = src.substring(indexname, src.length());

    System.out.println(name);

    //Open a URL Stream

    URL url = new URL(src);

    InputStream in = url.openStream();

    OutputStream out = new BufferedOutputStream(new FileOutputStream( folderpath+ name));

    for (int b; (b = in.read()) != -1;) {

        out.write(b);

    }

    out.close();

    in.close();
}

}

Answer 1

您可以查看 Wikipedia 429 状态代码告诉您您有太多请求：

The user has sent too many requests in a given amount of time. Intended for use with rate limiting schemes.

一个解决方案是放慢你的抓取速度。有一些方法可以做到这一点，一种是 use sleep.

Answer 2

您的问题是由于您的抓取工具违反了 reddit's API rules。错误 429 表示 "Too many requests" – 您请求的页面过多，速度过快。

你可以每2秒发出一个请求，你还需要设置一个合适的user agent（他们推荐的格式是<platform>:<app ID>:<version string> (by /u/<reddit username>)）。目前看来，您的代码运行太快了并且没有指定，因此它将受到严格的速率限制。

要修复它，首先，将其添加到 class 的开头，在主要方法之前：

public static final String USER_AGENT = "<PUT YOUR USER AGENT HERE>";

（确保指定一个实际的用户代理）。

然后，改变这个（在downloadImages）

URL url = new URL(src);
InputStream in = url.openStream();

对此：

URLConnection connection = (new URL(src)).openConnection();

Thread.sleep(2000); //Delay to comply with rate limiting
connection.setRequestProperty("User-Agent", USER_AGENT);

InputStream in = connection.getInputStream();

您还需要更改此设置（在 main 中）

Document doc = Jsoup.connect(subreddit).timeout(0).get();

对此：

Document doc = Jsoup.connect(subreddit).userAgent(USER_AGENT).timeout(0).get();

然后你的代码应该停止运行进入那个错误。

请注意，使用 reddit's API（IE，/r/subreddit.json 而不是 /r/subreddit）可能会使该项目更容易，但这不是必需的，您的当前代码将起作用。

Jsoup reddit 刮板 429 错误

Jsoup reddit scraper 429 error

java

web-scraping

jsoup

reddit