通过从个人资料 URL 重定向到一般 URL 下载文件

Question

我遇到这样一种情况，即一个网站的下载 link 被推广到所有个人资料：https://www.ffiec.gov/npw/Institution/HistoryExportToCSV

这似乎是通过访问在访问配置文件 URL 之后实现的：https://www.ffiec.gov/npw/Institution/Profile/242?dt=20120101

我正在尝试使用请求库来处理这个问题，但是有没有办法在访问第二个之后依次访问第一个 URL？

我认为以下可能会做到，但我在 return 中没有得到任何东西：

import requests

headers = {"Referer": "https://www.ffiec.gov/npw/Institution/Profile/242?dt=20120101"}
url = "https://www.ffiec.gov/npw/Institution/HistoryExportToCSV"

req = requests.get(url, headers=headers)
csv = req.content

Answer 1

以 Json 格式（而不是 CSV）加载数据更容易：

import requests

history_url = "https://www.ffiec.gov/npw/Institution/LoadHistory"
payload = {"RssdID": "242"} # the number is from profile URL

data = requests.post(history_url, data=payload)
print(data.json())

打印：

[
    {
        "Event": "Address Changed",
        "EventDate": "3/31/2006",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "The address of FIRST COMMUNITY BANK XENIA-FLORA changed from FRONT STREET  to 260 FRONT STREET .",
    },
    {
        "Event": "Name Changed",
        "EventDate": "10/30/2001",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "FIRST NATIONAL BANK OF XENIA, THE changed to the new name FIRST COMMUNITY BANK XENIA-FLORA.",
    },
    {
        "Event": "Institution Type Changed",
        "EventDate": "10/30/2001",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "FIRST COMMUNITY BANK XENIA-FLORA changed from National Bank to State Member Bank.",
    },
    {
        "Event": "Institution Established",
        "EventDate": "1/1/1922",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "FIRST NATIONAL BANK OF XENIA, THE was established as a National Bank at FRONT STREET , XENIA, IL.",
    },
]

通过从个人资料 URL 重定向到一般 URL 下载文件

Download file by redirecting to general URL from profile URL

python

python-requests