urllib2 / requests 不显示网页的 iframe
urllib2 / requests does not display iframe of the webpage
我正在尝试从 www.amazon.in
中删除一些图书数据
我需要位于 iframe 中的那本书的摘要，但问题是当我尝试使用 'requests' 打开 url 它不包含 iframe。
例如，当我这样做时
bookPage = requests.get(bookURL).text
bookSoup = BeautifulSoup(bookPage, "lxml")
bookPage中没有iframe，但实际页面有。
我也用 urllib2 试过，但似乎不起作用。
怎么了？
您可以从 div 元素中的 noscript 标签获取图书摘要，其中包含 id="bookDescription_feature_div":
>>> from bs4 import BeautifulSoup
>>> import requests
>>>
>>> response = requests.get('http://www.amazon.in/Life-What-Make-Preeti-Shenoy/dp/9380349300/ref=sr_1_6?s=books&ie=UTF8&qid=1424652069&sr=1-6',
... headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'})
>>>
>>> soup = BeautifulSoup(response.content)
>>> print soup.select('div#bookDescription_feature_div noscript')[0].get_text(strip=True)
Ankita Sharma has the world in her palms. She is young, smart and heads turn at every corner she walks by. Born into a conservative middle class household - this defines the chronicle of her life. Set in a time when Doordarshan was the prime source of entertainment and writing love letters was the general fad, every youngster dreams of the thrills of college life. And so, her admission into an MBA institute in Mumbai follows. Ankita's story begins here, from her life as a college student. Life seems all sunshine and flowers until a drastic turn leaves her staring at a disturbing path, only because of her own misdoing. Jump to six months later. The sun glistens on a sombre building. Magnetized in view, the words - “Mental Institute”. Who is the face staring out of the window?What if destiny twisted your journey? What if it dragged you to a place that houses your worst fears? Would you stand and fight or would you run? Set in the late eighties, across two cities, Life is What You Make It is a compelling account of growing up, determination, faith and how an unconquerable spirit can overcome the punches destiny throws at you. At its core, it is a love story that makes us question our identity and the concept of sanity.
我正在尝试从 www.amazon.in
中删除一些图书数据我需要位于 iframe 中的那本书的摘要，但问题是当我尝试使用 'requests' 打开 url 它不包含 iframe。
例如，当我这样做时
bookPage = requests.get(bookURL).text
bookSoup = BeautifulSoup(bookPage, "lxml")
bookPage中没有iframe，但实际页面有。
我也用 urllib2 试过，但似乎不起作用。
怎么了？
您可以从 div 元素中的 noscript 标签获取图书摘要，其中包含 id="bookDescription_feature_div":
>>> from bs4 import BeautifulSoup
>>> import requests
>>>
>>> response = requests.get('http://www.amazon.in/Life-What-Make-Preeti-Shenoy/dp/9380349300/ref=sr_1_6?s=books&ie=UTF8&qid=1424652069&sr=1-6',
... headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'})
>>>
>>> soup = BeautifulSoup(response.content)
>>> print soup.select('div#bookDescription_feature_div noscript')[0].get_text(strip=True)
Ankita Sharma has the world in her palms. She is young, smart and heads turn at every corner she walks by. Born into a conservative middle class household - this defines the chronicle of her life. Set in a time when Doordarshan was the prime source of entertainment and writing love letters was the general fad, every youngster dreams of the thrills of college life. And so, her admission into an MBA institute in Mumbai follows. Ankita's story begins here, from her life as a college student. Life seems all sunshine and flowers until a drastic turn leaves her staring at a disturbing path, only because of her own misdoing. Jump to six months later. The sun glistens on a sombre building. Magnetized in view, the words - “Mental Institute”. Who is the face staring out of the window?What if destiny twisted your journey? What if it dragged you to a place that houses your worst fears? Would you stand and fight or would you run? Set in the late eighties, across two cities, Life is What You Make It is a compelling account of growing up, determination, faith and how an unconquerable spirit can overcome the punches destiny throws at you. At its core, it is a love story that makes us question our identity and the concept of sanity.