使用 Python 和 BeautifulSoup 在 Javascript 中获取图像 url

Question

我正在尝试使用 Python 和 BeautifulSoup 从下面的页面获取产品图片。图片在 javascript 内。我正在使用 lxml。我创建了一个简化版本的代码，只关注图像。

我要的图片url是https://lapa.co.za/pub/media/catalog/product/cache/image/700x700/e9c3970ab036de70892d86c6d221abfe/l/e/learn_to_read_l3_b05_tippie_fish_cover.jpg

import json
from bs4 import BeautifulSoup
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148'
}
testlink = 'https://lapa.co.za/kinder-en-tienerboeke/leer-my-lees-vlak-1-grootboek-9-tippie-en-die-vis'

r = requests.get(testlink, headers=headers)
soup = BeautifulSoup(r.content, 'lxml')

title = soup.find('h1', class_='page-title').text.strip()
images = soup.find('div', class_='product-img-column')

# html_data = requests.get(testlink).text
# data = json.loads(re.search(r'window.INITIAL_REDUX_STATE=(\{.*?\});', html_data))

print(images)

Answer 1

json 在 <script> 标签中。只需要将其拉出即可。

import json
from bs4 import BeautifulSoup
import requests
import re

headers = {
    'User-Agent': 'Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148'
}
testlink = 'https://lapa.co.za/kinder-en-tienerboeke/leer-my-lees-vlak-1-grootboek-9-tippie-en-die-vis'

r = requests.get(testlink, headers=headers)
soup = BeautifulSoup(r.content, 'lxml')

title = soup.find('h1', class_='page-title').text.strip()
images = soup.find('div', class_='product-img-column')

script = images.find('script', {'type':'text/x-magento-init'})

jsonStr = re.search(r'<script type=\"text/x-magento-init\">(.*)</script>', str(script), re.IGNORECASE | re.DOTALL).group(1)
data = json.loads(jsonStr)

image_data = data['[data-gallery-role=gallery-placeholder]']['mage/gallery/gallery']['data'][0]
image_url = image_data['full']

# OR 
#image_url = image_data['img']

print(image_url)

输出：

print(image_url)
https://lapa.co.za/pub/media/catalog/product/cache/image/e9c3970ab036de70892d86c6d221abfe/9/7/9780799377347_1.jpg

使用 Python 和 BeautifulSoup 在 Javascript 中获取图像 url

Get the image url inside Javascript with Python and BeautifulSoup

python

web-scraping

beautifulsoup

lxml

screen-scraping