无法使用 Request 和 BeautifulSoup 获取 lxml 站点的信息

Question

我正在尝试制作一个测试项目来抓取特定站点的信息，但没有成功。
我遵循了一些我发现的教程，甚至是 Whosebug 上的 post。 毕竟我卡住了！
帮助我的同父异母兄弟，我是一个炙手可热的新程序员python，我无法停止我的项目。

更多信息：这是一个彩票网站，我试图废弃它并进行一些分析以获得幸运号码。

我已经学习了这个教程：
https://towardsdatascience.com/how-to-collect-data-from-any-website-cb8fad9e9ec5

https://beautiful-soup-4.readthedocs.io/en/latest/

大家有我的感激之情！

from bs4 import BeautifulSoup as bs
import requests 
import html5lib
#import urllib3 # another attemp to make  another req in the url ------failed

url = '''https://loterias.caixa.gov.br/Paginas/Mega-Sena.aspx'''

#another try to take results in the <ul> but I have no qualified results  == None
def parse_ul(elem):#
    result = {}
    for sub in elem.find_all('li', recursive=False):
        if sub.li is None:
            continue
        data = {k: v for k, v in sub.attrs.items()}
        if sub.ul is not None:
            # recurse down
            data['children'] = parse_ul(sub.ul)
        result[sub.li.get_text(strip=True)] = data
    return result

page = requests.get(url)#taking info from website

print(page.encoding)# == UTF-8

soup = bs(page.content,features="lxml")#takes all info from the url and organizes it ==Beaultiful soup

numbers = soup.find(id='ulDezenas')#searcher in the content of this specific id// another try: soup.find('ul', {'class': ''})

result =  parse_ul(soup)#try to parse info, but none is found EVEN WITH THE ORIGINAL ONE

print(numbers)#The result is below:
'''<ul class="numbers megasena" id="ulDezenas">
<li ng-repeat="dezena in resultado.listaDezenas ">{{dezena.length &gt; 2 ? dezena.slice(1) : dezena}}</li>
</ul>'''
print(result)# == "{}" nothing found

#with open('''D:\Documents\python\_abretesesame.txt''', 'wb') as fd:
#    for chunk in page.iter_content(chunk_size=128):
#        fd.write(chunk)
# =======printing document(HTML) in file still no success in getting the numbers

Answer 1

主要问题是内容由 JavaScript 动态提供，但您可以通过另一个 url:

获取信息

jsonData = requests.get('https://servicebus2.caixa.gov.br/portaldeloterias/api/megasena/').json()

会给你以下JSON:

{'tipoJogo': 'MEGA_SENA', 'numero': 2468, 'nomeMunicipioUFSorteio': 'SÃO PAULO, SP', 'dataApuracao': '02/04/2022', 'valorArrecadado': 158184963.0, 'valorEstimadoProximoConcurso': 3000000.0, 'valorAcumuladoProximoConcurso': 0.0, 'valorAcumuladoConcursoEspecial': 36771176.89, 'valorAcumuladoConcurso_0_5': 33463457.98, 'acumulado': False, 'indicadorConcursoEspecial': 1, 'dezenasSorteadasOrdemSorteio': ['022', '041', '053', '042', '035', '057'], 'listaResultadoEquipeEsportiva': None, 'numeroJogo': 2, 'nomeTimeCoracaoMesSorte': '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'tipoPublicacao': 3, 'observacao': '', 'localSorteio': 'ESPAÇO DA SORTE', 'dataProximoConcurso': '06/04/2022', 'numeroConcursoAnterior': 2467, 'numeroConcursoProximo': 2469, 'valorTotalPremioFaixaUm': 0.0, 'numeroConcursoFinal_0_5': 2470, 'listaDezenas': ['022', '035', '041', '042', '053', '057'], 'listaDezenasSegundoSorteio': None, 'listaMunicipioUFGanhadores': [{'posicao': 1, 'ganhadores': 1, 'municipio': 'SANTOS', 'uf': 'SP', 'nomeFatansiaUL': '', 'serie': ''}], 'listaRateioPremio': [{'faixa': 1, 'numeroDeGanhadores': 1, 'valorPremio': 122627171.8, 'descricaoFaixa': '6 acertos'}, {'faixa': 2, 'numeroDeGanhadores': 267, 'valorPremio': 34158.18, 'descricaoFaixa': '5 acertos'}, {'faixa': 3, 'numeroDeGanhadores': 20734, 'valorPremio': 628.38, 'descricaoFaixa': '4 acertos'}], 'id': None, 'ultimoConcurso': True, 'exibirDetalhamentoPorCidade': True, 'premiacaoContingencia': None}

只需提取 dezenasSorteadasOrdemSorteio 并在 list comprehension 中处理：

[n if len(n) < 2 else n[1:] for n in jsonData['listaDezenas']]

结果将是：

['22', '35', '41', '42', '53', '57']

例子

import requests

jsonData = requests.get('https://servicebus2.caixa.gov.br/portaldeloterias/api/megasena/').json()
print([n if len(n) < 2 else n[1:] for n in jsonData['listaDezenas']])

无法使用 Request 和 BeautifulSoup 获取 lxml 站点的信息

Can't get info of a lxml site with Request and BeautifulSoup

python

beautifulsoup

web-scraping

python-requests

例子