Scrapy FormRequest 无法将复杂的字典作为表单数据处理

Question

我正在尝试向 scrapy.FormRequest 对象提供表单数据。表单数据是以下结构的字典：

{
  "param1": [
    {
      "paramA": "valueA",
      "paramB": "valueB"
    }
  ]
}

相当于下面的代码，运行 in scrapy shell:

from scrapy import FormRequest

url = 'www.example.com'
method_post = 'POST'
formdata = <the above dict>

fr = FormRequest(url=url, method=method_post, formdata=formdata)

fetch(fr)

作为回应，我收到以下错误：

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 31, in __init__
    querystr = _urlencode(items, self.encoding)
  File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 66, in _urlencode
    for k, vs in seq
  File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 67, in <listcomp>
    for v in (vs if is_listlike(vs) else [vs])]
  File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/utils/python.py", line 119, in to_bytes
    'object, got %s' % type(text).__name__)
TypeError: to_bytes must receive a unicode, str or bytes object, got dict

我尝试了多种解决方案，包括将整个内容作为字符串，使用各种转义字符，以及对 dict 进行变体以使其更令人满意，但是 none 解决方案消除了此错误为请求工作（我收到 400 响应）。

我知道表单数据和我所做的其他一切都是正确的，因为我已经在 curl 中成功复制了它（表单数据是通过 -d formdata.txt 提供的）。

有没有办法解决 FormRequest 无法处理复杂字典结构的问题？或者我错过了什么？

Answer 1

您可以尝试使用 body 参数代替 formdata。示例：

FormRequest(url=url, method=method_post, body=json.dumps(formdata))

Scrapy FormRequest 无法将复杂的字典作为表单数据处理

Scrapy FormRequest can't handle complex dicts as formdata

python

form-data

scrapy

web-scraping

scrapy-shell