使用 Scrapy Selector 提取包含其他元素内容的段落文本

Question

使用Scrapy 0.24 Selectors，我想提取包含其他元素内容的段落内容（在下面的示例中，它是锚点<a>。我该如何实现？

代码

>>> from scrapy import Selector
>>> html = """
        <html>
            <head>
                <title>Test</title>
            </head>
            <body>
                <div>
                    <p>Hello, can I get this paragraph content without this <a href="http://google.com">Google link</a>?
                </div>
            </body>
        </html>
        """
>>> sel = Selector(text=html, type="html")
>>> sel.xpath('//p/text()').extract()
[u'Hello, can I get this paragraph content with this ', u'?']

输出

[u'Hello, can I get this paragraph content with this ', u'?']

预期输出

[u'Hello, can I get this paragraph content with this Google link?']

Answer 1

我会推荐 BeautifulSoup。 scrapy是一个完整的爬虫框架，而BS是一个强大的解析库(Difference between BeautifulSoup and Scrapy crawler?)。

文档：http://www.crummy.com/software/BeautifulSoup/bs4/doc/

安装：pip install beautifulsoup4

针对您的情况：

# 'html' is the one your provided
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
res = [p.get_text().strip() for p in soup.find_all('p')]

结果：

[u'Hello, can I get this paragraph content without this Google link?']

使用 Scrapy Selector 提取包含其他元素内容的段落文本

Extracting paragraph text including other element's content using Scrapy Selector

html

python

xpath

scrapy

代码

输出

预期输出