您无权访问此服务器上的 "http://www.carrefour.pk/"。<p> 参考 #18.451d2017.1615456534.6b4445

Question

我正在尝试 通过 python 抓取 家乐福网站数据。我用过斗志旺盛的美丽汤和硒，但似乎没有任何效果。我收到您无权访问的错误消息。有什么办法可以抓取这个网站吗？代码附在下面，需要帮助！

from requests_html import HTMLSession


session = HTMLSession()
 
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"}

resp = session.get("https://www.carrefour.pk/",headers=headers)

resp.html.render()

a=resp.html.html

print(a)

Answer 1

认为你用错了headers。这些 headers 对我来说很好用。 headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'}

或已满：

import requests 
from bs4 import BeautifulSoup as bs


# Block cookies
from http import cookiejar  # Python 2: import cookielib as cookiejar
class BlockAll(cookiejar.CookiePolicy):
    return_ok = set_ok = domain_return_ok = path_return_ok = lambda self, *args, **kwargs: False
    netscape = True
    rfc2965 = hide_cookie2 = False
    
s = requests.Session()
s.cookies.set_policy(BlockAll())


#Get URL
url = "https://www.carrefour.pk"
        
headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'}


r = s.get(url, headers=headers)
soup = bs(r.text, 'html.parser')
print(soup)

您无权访问此服务器上的 "http://www.carrefour.pk/"。<p> 参考 #18.451d2017.1615456534.6b4445

You don't have permission to access "http://www.carrefour.pk/" on this server.<p> Reference #18.451d2017.1615456534.6b4445

html

python

request

web-scraping