将 SEC 10-K 年度报告文本保存到文件（解码有问题）

Question

我正在尝试从 10-K SEC Edgar 报告（不关心表格）中批量下载 "end-user" 可见的文本并将其保存在文本文件中。我在 Youtube 上找到了下面的代码，但是我面临 2 个挑战：

我不确定我是否捕获了所有文本，当我从下面打印 URL 时，我收到非常奇怪的输出（特殊字符，例如，在打印出来)
我似乎无法将文本保存在 txt 文件中，不确定这是否是编码问题（我是编程新手）。

import re
import requests
import unicodedata
from bs4 import BeautifulSoup

def restore_windows_1252_characters(restore_string):
    def to_windows_1252(match):
        try:
            return bytes([ord(match.group(0))]).decode('windows-1252')
        except UnicodeDecodeError:
            # No character at the corresponding code point: remove it.
            return ''

    return re.sub(r'[\u0080-\u0099]', to_windows_1252, restore_string)

# define the url to specific html_text file
new_html_text = r"https://www.sec.gov/Archives/edgar/data/796343/0000796343-14-000004.txt"

# grab the response
response = requests.get(new_html_text)
page_soup = BeautifulSoup(response.content,'html5lib')

page_text = page_soup.html.body.get_text(' ',strip = True)

# normalize the text, remove characters. Additionally, restore missing window characters.
page_text_norm = restore_windows_1252_characters(unicodedata.normalize('NFKD', page_text)) 

# print: this works however gives me weird special characters in the print (e.g., at the very end)
print(page_text_norm)

# save to file: this only gives me an empty text file
with open('testfile.txt','w') as file:
    file.write(page_text_norm)

Answer 1

试试这个。如果以你期望的数据为例，人们会更容易理解你的需求。

from simplified_scrapy import SimplifiedDoc,req,utils
url = 'https://www.sec.gov/Archives/edgar/data/796343/0000796343-14-000004.txt'
html = req.get(url)
doc = SimplifiedDoc(html)
# text = doc.body.text
text = doc.body.unescape() # Converting HTML entities
utils.saveFile("testfile.txt",text)

将 SEC 10-K 年度报告文本保存到文件（解码有问题）

Saving SEC 10-K annual report text to files (trouble with decoding)

parsing

edgar