表单提交后Scrapy下载MIME类型文件

Question

在页面 http://comunicacion.movistarplus.es/guiaProgramacion/exportarProgramacion 中有一个表单，用户必须在其中键入一些字段才能下载最重要的西班牙广播公司的频道时间表。用户填写表格并单击 Exportar 按钮后，浏览器会自动开始下载扩展名为 formato 下拉列表的文件。

现在，从我的 Chrome 浏览器中获取文件非常容易。但是，我想使用 Scrapy 脚本将其自动化，您可以在下面找到我到目前为止所做的事情：

from scrapy.item import Item, Field
from scrapy.http import FormRequest
from scrapy.spiders import Spider
from scrapy.selector import Selector
from scrapy.http import Request

class ProgramacionSpider(Spider):
    name = "Programacion"
    allowed_domains = ["comunicacion.movistarplus.es"]
    start_urls = ["http://comunicacion.movistarplus.es/guiaProgramacion/exportarProgramacion"]

    def parse(self, response):
        yield FormRequest.from_response(response, formname='formExportar', formdata={"fechaInicio": "2017-01-05", "fechaFin": "2017-01-19", "genero" : "0", "selPredefinicion": "0", "formato": "csv", "cadena[]": "{INTECO, ETB}"}, callback=self.parse1)

    def parse1(self, response):
        print response

现在，在使用需要与表单一起发送的 POST 参数完成 FormRequest.from_response 之后，Scrapy 响应以下文本：

2017-01-05 13:19:11 [scrapy] INFO: Scrapy 1.1.2 started (bot: Programacion)
2017-01-05 13:19:11 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Programacion.spiders', 'FEED_URI': 'sss.csv', 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 'SPIDER_MODULES': ['Programacion.spiders'], 'BOT_NAME': 'Programacion', 'FEED_FORMAT': 'csv', 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage'}
2017-01-05 13:19:11 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2017-01-05 13:19:12 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-01-05 13:19:12 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_splash.SplashDeduplicateArgsMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-01-05 13:19:12 [scrapy] INFO: Enabled item pipelines:
[]
2017-01-05 13:19:12 [scrapy] INFO: Spider opened
2017-01-05 13:19:12 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-05 13:19:12 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-05 13:19:12 [scrapy] DEBUG: Crawled (200) <GET http://comunicacion.movistarplus.es/guiaProgramacion/exportarProgramacion> (referer: None)
2017-01-05 13:19:12 [scrapy] DEBUG: Crawled (200) <POST http://comunicacion.movistarplus.es/guiaProgramacion/exportar> (referer: http://comunicacion.movistarplus.es/guiaProgramacion/exportarProgramacion)
<200 http://comunicacion.movistarplus.es/guiaProgramacion/exportar>
2017-01-05 13:19:12 [scrapy] INFO: Closing spider (finished)
2017-01-05 13:19:12 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 822,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 1,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 6603,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 1, 5, 12, 19, 12, 783383),
 'log_count/DEBUG': 3,
 'log_count/INFO': 7,
 'request_depth_max': 1,
 'response_received_count': 2,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2017, 1, 5, 12, 19, 12, 204111)}
2017-01-05 13:19:12 [scrapy] INFO: Spider closed (finished)

显然，爬虫似乎工作正常，因为它没有响应 500 错误（即它接受来自表单的 POST 变量）。

问题

我想知道如何使用 Scrapy 直接下载文件，但完全不知道该怎么做。谢谢。

Answer 1

您正在谈论的文件在您收到的响应中，更准确地说，它在 response.body 属性中。

您可以简单地打开一个文件并将response.body的内容写入其中。像这样：

def parse(self, response):
    # check if response is valid
    if response.status != 200:
        print("failed to get schedule")
        return
    # save to file valid response
    with open('results.csv', 'w') as f:
        f.write(response.body)

编辑：我发现您的表单数据存在问题，这就是返回空文件的原因：

def parse(self, response):
    formdata={"fechaInicio": "2017-01-05", 
              "fechaFin": "2017-01-19", 
              "genero": "0",
              "selPredefinicion": "0", 
              "formato": "csv",
              # "cadena[]": "{INTECO, ETB}",  # incorrect
              "cadena[]": ["INTECO", "ETB"]}  # <---- this is correct
    yield FormRequest.from_response(response, formname='formExportar', 
                                    formdata=formdata,
                                    callback=self.parse1)

您在此处提供了不正确的字符串而不是列表，上面的示例是正确的并且 returning/saving 结果正确。

表单提交后Scrapy下载MIME类型文件

Scrapy download MIME type file after form submission

javascript

forms

mime

scrapy