如何使用 mechanize 和 nokogiri 获取链接 ruby

Question

鉴于下面的示例，任何人都可以告诉我如何使用 Nokogiri 和 Mechanize 获取单独组中每个 <h4> 标签下的所有链接，I.E.以下所有链接：

"some text"
"some more text"
"some additional text"

<div id="right_holder">
    <h3><a href="#"><img src="http://example.com" width="11" height="11"></a></h3>
    <br />
    <br />
    <h4><a href="#">Some text</a></h4>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <br />
    <br />
    <h4><a href="#">Some more text</a></h4>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <br />
    <br />
    <h4><a href="#">Some additional text</a></h4>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
</div>

Answer 1

您可以像“How to split a HTML document using Nokogiri?”一样遍历并分隔数据，但如果您知道标签是什么，您可以 split 它：

# html is the raw html string
html.split('<h4').map{|g| Nokogiri::HTML::DocumentFragment.parse(g).css('a') }

page = Nokogiri::HTML(html).css("#right_holder")
links = page.children.inject([]) do |link_hash, child|
  if child.name == 'h4'
    name = child.text
    link_hash << { :name => name, :content => ""}
  end

  next link_hash if link_hash.empty?
  link_hash.last[:content] << child.to_xhtml
  link_hash
end

grouped_hsh = links.inject({}) do |hsh, link|
  hsh[link[:name]] = Nokogiri::HTML::DocumentFragment.parse(link[:content]).css('a')
  hsh
end

# {"Some text"=>[#<Nokogiri::XML::Element:0x3ff4860d6c30,
#  "Some more text"=>[#<Nokogiri::XML::Element:0x3ff486096c20...,
#  "Some additional text"=>[#<Nokogiri::XML::Element:0x3ff486f2de78...}

Answer 2

一般来说你会这样做：

page.search('h4 a').each do |a|
  puts a[:href]
end

但我相信您已经注意到 none 这些链接实际上可以到达任何地方。

更新：

要对它们进行分组，一些节点集数学如何：

page.search('h4').each do |h4|
  puts h4.text
  (h4.search('~ a') - h4.search('~ h4 ~ a')).each do |a|
    puts a.text
  end
end

这意味着每个 a 都跟在 h4 之后并且不跟在另一个 h4

之后

如何使用 mechanize 和 nokogiri 获取链接 ruby

How to get links using mechanize and nokogiri ruby

ruby

web-scraping

nokogiri

mechanize