Ruby 机械化抓取 ResponseCodeError

Question

我正在尝试抓取网站的所有结果页面，它可以工作，但有时脚本会停止并显示此错误：

502 => Net::HTTPBadGateway for https://website.com/id/12/ -- unhandled response (Mechanize::ResponseCodeError)

我想继续脚本，即使它发现错误。

我的脚本：

require 'mechanize'
require 'csv'

a = Mechanize.new
CSV.open('datas.csv', "wb") do |csv|
    page = a.get("https://website.com/?page=1-200") #498
    number = 0
    page.links_with(:class => "btn btn-default").each do |link|
        post_link = link.href
        inside_page = a.get("https://website.com#{post_link}")
        title = inside_page.at("h1.serviceTitle").text.strip
        author = inside_page.at(".name").text.strip
        number+=1
        csv << [title, author]
    end
end

有什么想法吗？

Answer 1

这很容易通过适当的异常处理来解决。你可以check this page for a better explanation

至于你的代码，你可以像这样处理异常

CSV.open('datas.csv', "wb") do |csv|
  begin
    a = Mechanize.new
    page = a.get("https://website.com/?page=1-200") #498
    number = 0
    page.links_with(:class => "btn btn-default").each do |link|
        post_link = link.href
        inside_page = a.get("https://website.com#{post_link}")
        title = inside_page.at("h1.serviceTitle").text.strip
        author = inside_page.at(".name").text.strip
        number+=1
        csv << [title, author]
    end
  rescue => e
    // do nothing and move on to the next line
  end
end

Ruby 机械化抓取 ResponseCodeError

Ruby Mechanize scraping ResponseCodeError

ruby

mechanize

mechanize-ruby