数据抓取多个页面点击循环

Question

试图找出一种方法来使用一种机制从 UCAS 网站上抓取我们想要的所有数据并将其添加到数组中。目前，我们正在为 link 机械化点击编码而苦苦挣扎。想知道是否有人可以提供帮助，在循环中连续点击 3 次 link 以浏览所有搜索结果页面。显示所有大学课程的第一个 link 在 div class morecourseslink

内

第二个 link 显示课程名称、持续时间和质量在 div class coursenamearea

第三个 link 在 div coursedetailsshowable 中，a id 是 coursedetailtab_entryreqs

目前我们正在使用以下内容抓取 uninames：

class PagesController < ApplicationController
  def home


require 'mechanize'
mechanize = Mechanize.new

@uninames_array = []

   page = mechanize.get('http://search.ucas.com/search/providers?CountryCode=3&RegionCode=&Lat=&Lng=&Feather=&Vac=2&Query=&ProviderQuery=&AcpId=&Location=scotland&IsFeatherProcessed=True&SubjectCode=&AvailableIn=2016')


page.search('li.result h3').each do |h3|
  name = h3.text
  @uninames_array.push(name)
end

while next_page_link = page.at('.pager a[text()=">"]')
  page = mechanize.get(next_page_link['href'])

  page.search('li.result h3').each do |h3|
    name = h3.text
    @uninames_array.push(name)
  end
end

puts @uninames_array.to_s
  end
end

课程名称持续时间和资格如下：

require 'mechanize'


mechanize = Mechanize.new
@duration_array = []
@qual_array = []
@courses_array = []

page = mechanize.get('http://search.ucas.com/search/results?Vac=2&AvailableIn=2016&IsFeatherProcessed=True&page=1&providerids=41')


page.search('div.courseinfoduration').each do |x|
puts x.text.strip
page.search('div.courseinfooutcome').each do |y|
puts y.text.strip

end

while next_page_link = page.at('.pager a[text()=">"]')
  page = mechanize.get(next_page_link['href'])

page.search('div.courseinfoduration').each do |x|
    name = x
    @duration_array.push(name)
    puts x.text.strip
  end
end
while next_page_link = page.at('.pager a[text()=">"]')
  page = mechanize.get(next_page_link['href'])

page.search('div.courseinfooutcome').each do |y|
    name = y
    @qual_array.push(name)
    puts y.text.strip
  end
end
page.search('div.coursenamearea h4').each do |h4|
puts h4.text.strip

end

while next_page_link = page.at('.pager a[text()=">"]')
  page = mechanize.get(next_page_link['href'])

page.search('div.coursenamearea h4').each do |h4|
    name = h4.text
    @courses_array.push(name)
    puts h4.text.strip
  end
end
end

Answer 1

如果您想对一个 Mechanize 实例执行此操作，为什么不将它们全部串在一起并将需要跳转的页面存储在变量中？

如果您的所有代码都能正常工作，那么您可以简单地将它们串在一起成为一个方法调用：

def home


  require 'mechanize'
  mechanize = Mechanize.new

  @uninames_array = []

  page = mechanize.get('http://search.ucas.com/search/providers?CountryCode=3&RegionCode=&Lat=&Lng=&Feather=&Vac=2&Query=&ProviderQuery=&AcpId=&Location=scotland&IsFeatherProcessed=True&SubjectCode=&AvailableIn=2016')


  page.search('li.result h3').each do |h3|
    name = h3.text
    @uninames_array.push(name)
  end

  while next_page_link = page.at('.pager a[text()=">"]')
    page = mechanize.get(next_page_link['href'])

    page.search('li.result h3').each do |h3|
      name = h3.text
      @uninames_array.push(name)
    end
  end


@duration_array = []
@qual_array = []
@courses_array = []

page = mechanize.get('http://search.ucas.com/search/results?Vac=2&AvailableIn=2016&IsFeatherProcessed=True&page=1&providerids=41')


page.search('div.courseinfoduration').each do |x|
puts x.text.strip
page.search('div.courseinfooutcome').each do |y|
puts y.text.strip

end

while next_page_link = page.at('.pager a[text()=">"]')
  page = mechanize.get(next_page_link['href'])

page.search('div.courseinfoduration').each do |x|
    name = x
    @duration_array.push(name)
    puts x.text.strip
  end
end
while next_page_link = page.at('.pager a[text()=">"]')
  page = mechanize.get(next_page_link['href'])

page.search('div.courseinfooutcome').each do |y|
    name = y
    @qual_array.push(name)
    puts y.text.strip
  end
end
page.search('div.coursenamearea h4').each do |h4|
puts h4.text.strip

end

while next_page_link = page.at('.pager a[text()=">"]')
  page = mechanize.get(next_page_link['href'])

page.search('div.coursenamearea h4').each do |h4|
    name = h4.text
    @courses_array.push(name)
    puts h4.text.strip
  end
end

数据抓取多个页面点击循环

Data scraping multiple page clicks loops

ruby

ruby-on-rails

mechanize

nokogiri

web-scraping