如何抓取网站投资以获得股票的每项技术分析

Question

我正在尝试抓取网站 https://investing.com/ 以获取任何股票的技术数据。我想获得“移动平均线：”和“技术指标：”不同时期的买入量和卖出量：

5 小时
每天
每周

这是一张图片，可以查看我想要获取的数据： https://i.ibb.co/mHpM0Yw/Capture-d-e-cran-2019-08-14-a-00-15-45.png

url 是 https://investing.com/equities/credit-agricole-technical

当您导航到浏览器时，句点设置为“hourly”，您必须单击另一个句点才能获得正确的数据。 DOM 在 XML 请求后更新。

我想在 DOM 更新后抓取页面。

机械化

我尝试使用 Mechanize 进行抓取并单击“每周”并获取 DOM 进行抓取，但出现错误

这是我的代码：

    def mechanize_scraper(url)
      agent = Mechanize.new
      puts agent.user_agent_alias = 'Mac Safari'
      page = agent.get(url)
      link = page.link_with(text: 'Weekly')
      new_page = link.click
    end


    url = "https://investing.com/equities/credit-agricole-technical"
    mechanize_scraper(url)

这是错误：

Mechanize::UnsupportedSchemeError (Mechanize::UnsupportedSchemeError)

当我们检查 DOM 时，link 有一个属性 "href" = javascript(void);

    <li pairid="407" data-period="week" class="">
      <a href="javascript:void(0);">Weekly</a>
    </li>

所以在尝试和大量 google 搜索之后，我继续“Watir”以尝试抓取。

瓦提尔

这是我的代码：

    def watir_scraper(url)
      Watir.default_timeout = 10
      browser = Watir::Browser.new
      browser.goto(url)       
      link = browser.link(text: /weekly/).click
      pp link
    end

    url = "https://investing.com/equities/credit-agricole-technical"
    watir_scraper(url)

这是错误：

40: from app.rb:47:in `'

39: from app.rb:32:in `watir_scraper'

38: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/watir-6.16.5/lib/watir/elements/element.rb:145:in `click'

37: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/watir-6.16.5/lib/watir/elements/element.rb:789:in `element_call'

36: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/watir-6.16.5/lib/watir/elements/element.rb:154:in `block in click'

35: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/common/element.rb:74:in `click'

34: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/w3c/bridge.rb:371:in `click_element'

33: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/w3c/bridge.rb:567:in `execute'

32: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/bridge.rb:167:in `execute'

31: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/http/common.rb:64:in `call'

30: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/http/default.rb:114:in `request'

29: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/http/common.rb:88:in `create_response'

28: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/http/common.rb:88:in `new'

27: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/response.rb:34:in `initialize'

26: from /Users/remicarette/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.3/lib/selenium/webdriver/remote/response.rb:72:in `assert_ok'

25: from 25 libsystem_pthread.dylib 0x00007fff5aaa440d thread_start + 13

24: from 24 libsystem_pthread.dylib 0x00007fff5aaa8249 _pthread_start + 66

23: from 23 libsystem_pthread.dylib 0x00007fff5aaa52eb _pthread_body + 126

22: from 22 chromedriver 0x000000010b434e67 chromedriver + 3673703

21: from 21 chromedriver 0x000000010b416014 chromedriver + 3547156

20: from 20 chromedriver 0x000000010b3e0f07 chromedriver + 3329799

19: from 19 chromedriver 0x000000010b3f91b8 chromedriver + 3428792

18: from 18 chromedriver 0x000000010b3cd069 chromedriver + 3248233

17: from 17 chromedriver 0x000000010b3f86d8 chromedriver + 3426008

16: from 16 chromedriver 0x000000010b3f8940 chromedriver + 3426624

15: from 15 chromedriver 0x000000010b3ecc1f chromedriver + 3378207

14: from 14 chromedriver 0x000000010b0ce8a5 chromedriver + 108709

13: from 13 chromedriver 0x000000010b0cd7e2 chromedriver + 104418

12: from 12 chromedriver 0x000000010b0f1bf3 chromedriver + 252915

11: from 11 chromedriver 0x000000010b0fba37 chromedriver + 293431

10: from 10 chromedriver 0x000000010b0f1c4e chromedriver + 253006

9: from 9 chromedriver 0x000000010b0cfa66 chromedriver + 113254

8: from 8 chromedriver 0x000000010b0f1a72 chromedriver + 252530

7: from 7 chromedriver 0x000000010b0cfe66 chromedriver + 114278

6: from 6 chromedriver 0x000000010b0d63fb chromedriver + 140283

5: from 5 chromedriver 0x000000010b0d71a9 chromedriver + 143785

4: from 4 chromedriver 0x000000010b0d8d19 chromedriver + 150809

3: from 3 chromedriver 0x000000010b0da569 chromedriver + 157033

2: from 2 chromedriver 0x000000010b15fcef chromedriver + 703727

1: from 1 chromedriver 0x000000010b3bf133 chromedriver + 3191091 0x000000010b42f129 chromedriver + 3649833: element click intercepted: Element ... is not clickable at point (544, 704). Other element would receive the click: ... (Selenium::WebDriver::Error::ElementClickInterceptedError) (Session info: chrome=76.0.3809.100)

我希望一切都可以帮助您理解我的问题。我想知道我是否可以使用 Mechanize 或 Watir 抓取数据。如果没有，哪些工具可以完成这项工作？

非常感谢！

Answer 1

我不认为这正是您要找的东西，但它可能会让您更接近一些。

使用 HTTP 嗅探器，发现您尝试点击的 link 生成 POST。 POST 的响应可以通过以下方式获得：

def mechanize_poster(url)
  agent = Mechanize.new
  headers = {
    'X-Requested-With' => 'XMLHttpRequest',
    'User-Agent' => 'Mac Safari',
    'Content-Type' => 'application/x-www-form-urlencoded',
    'Referer' => 'https://www.investing.com/equities/credit-agricole-technical'
  }
  fields = {
    period: 'week',
    viewType: 'normal',
    pairID: '407'
  }
  page = agent.post(url, fields, headers)
  p page
end

我认为您需要使用一些 Nokogiri 来获取数据值。

Answer 2

您可以使用 POST 对请求和 bs4 执行此操作。与我看到的其他答案中的想法相同，但使用循环来提供所有请求的值。我只是使用开发工具在单击 5hr、Daily 等时监视网络流量，然后观察 xhr 调用。

import requests
from bs4 import BeautifulSoup as bs

headers = { 'User-Agent': 'Mozilla/5.0',
            'Content-Type': 'application/x-www-form-urlencoded',
            'Referer': 'https://www.investing.com/equities/credit-agricole-technical',
            'X-Requested-With': 'XMLHttpRequest'}

body = {'pairID' : 407, 'period': '', 'viewType' : 'normal'}
periods = {'5hr': 18000, 'Daily': 86400, 'Weekly': 'week'}

with requests.Session() as s:
    for k, v in periods.items():
        body['period'] = v
        r = s.post('https://www.investing.com/instruments/Service/GetTechincalData', data = body, headers = headers)
        soup = bs(r.content, 'lxml')
        for i in soup.select('#techStudiesInnerWrap .summaryTableLine'):
            print(k, ' : ' , ' '.join([j.text for j in i.select('span')]))

输出：

Answer 3

您在 Watir 中看到的错误来自 webdriver，它表明如果有人试图点击那个 link，页面上的其他一些元素将被点击（因为其他元素与 link 重叠） =18=].

可能默认的浏览器尺寸较小，而您正在处理的 'reactive' 设计不会缩小到低于给定尺寸（常见问题）

首先尝试设置屏幕尺寸，使其与您将要使用的尺寸相似（例如 1024x768 或更大）@browser.window.resize_to(1920, 1080)

如何抓取网站投资以获得股票的每项技术分析

How to scrape the website investing to get every technical analysis for a stock

javascript

ruby

mechanize

watir

web-scraping

机械化

瓦提尔

如何抓取网站投资以获​​得股票的每项技术分析

How to scrape the website investing to get every technical analysis for a stock

javascript

ruby

mechanize

watir

web-scraping

机械化

瓦提尔

如何抓取网站投资以获得股票的每项技术分析