免费 heroku 帐户中的应用程序未 运行
Application not running in free heroku account
我有一个免费的 heroku 帐户，我已经在其中部署了我的 rails 应用程序。我还启用了 redis heroku 插件。我的应用程序抓取网站，在我的本地环境中，它需要 93 分钟才能完成。所以我决定把它放在后台任务中并 运行 放在那里。我按照一些教程在免费的 heroku 帐户中获取后台任务 运行ning，所以我使用了独角兽服务器。在本地测试时，我的应用程序运行良好，但是一旦我将它部署到 heroku 并浏览我的项目站点，它就不会 work.Here 是 heroku log
的结果
2016-09-16T17:40:40.934711+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.943447+00:00 app[web.1]: E, [2016-09-16T17:40:40.943337 #3] ERROR -- : reaped #<Process::Status: pid 13987 exit 1> worker=2
2016-09-16T17:40:40.951820+00:00 app[web.1]: E, [2016-09-16T17:40:40.946468 #13992] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.951822+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.951825+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.951827+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.951828+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.951829+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.961433+00:00 app[web.1]: E, [2016-09-16T17:40:40.961347 #3] ERROR -- : reaped #<Process::Status: pid 13992 exit 1> worker=1
2016-09-16T17:40:40.966761+00:00 app[web.1]: E, [2016-09-16T17:40:40.953957 #13996] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.966764+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.966765+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.966766+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.966768+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.966769+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.966770+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.966771+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.968861+00:00 app[web.1]: E, [2016-09-16T17:40:40.944646 #13990] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.968863+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.968864+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.968865+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.968866+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.968867+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.968872+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.968871+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977007+00:00 app[web.1]: E, [2016-09-16T17:40:40.976925 #3] ERROR -- : reaped #<Process::Status: pid 13990 exit 1> worker=0
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.977910+00:00 app[web.1]: E, [2016-09-16T17:40:40.965528 #13999] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.977913+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.977915+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.977916+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.977917+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977919+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.981913+00:00 app[web.1]: E, [2016-09-16T17:40:40.981796 #3] ERROR -- : reaped #<Process::Status: pid 13996 exit 1> worker=2
2016-09-16T17:40:40.982020+00:00 app[web.1]: E, [2016-09-16T17:40:40.981968 #3] ERROR -- : reaped #<Process::Status: pid 13999 exit 1> worker=1
2016-09-16T17:40:40.998699+00:00 app[web.1]: E, [2016-09-16T17:40:40.985434 #14004] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.998703+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.998704+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.998705+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.998706+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.998707+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.998709+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.998710+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.013015+00:00 app[web.1]: E, [2016-09-16T17:40:41.012891 #3] ERROR -- : reaped #<Process::Status: pid 14004 exit 1> worker=0
2016-09-16T17:40:41.013269+00:00 app[web.1]: E, [2016-09-16T17:40:40.992746 #14009] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.013271+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.013273+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.013274+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.013275+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.013277+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.015082+00:00 app[web.1]: E, [2016-09-16T17:40:40.991259 #14006] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.015083+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.015084+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.015085+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.015087+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.015089+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.029456+00:00 app[web.1]: E, [2016-09-16T17:40:41.019600 #14013] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.029459+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.029465+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.029466+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.033517+00:00 app[web.1]: E, [2016-09-16T17:40:41.032905 #3] ERROR -- : reaped #<Process::Status: pid 14006 exit 1> worker=1
2016-09-16T17:40:41.033606+00:00 app[web.1]: E, [2016-09-16T17:40:41.033566 #3] ERROR -- : reaped #<Process::Status: pid 14009 exit 1> worker=2
2016-09-16T17:40:41.034410+00:00 app[web.1]: E, [2016-09-16T17:40:41.034370 #3] ERROR -- : reaped #<Process::Status: pid 14013 exit 1> worker=0
2016-09-16T17:40:41.067342+00:00 app[web.1]: E, [2016-09-16T17:40:41.057962 #14018] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.067346+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.067348+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.067350+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.067351+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.067352+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.077263+00:00 app[web.1]: E, [2016-09-16T17:40:41.077156 #3] ERROR -- : reaped #<Process::Status: pid 14018 exit 1> worker=1
这是我的独角兽配置
unicorn.rb
before_fork do |server, worker|
@sidekiq_pid ||= spawn("bundle exec sidekiq -c 2")
end
worker_processes 3
after_fork do |server, worker|
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
Sidekiq.configure_server do |config|
config.redis = { :size => 5 }
end
end
过程文件
web: bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb
首先，设置你的 procfile 以正确使用 worker dynos（我使用 puma for web 所以改变它以反映你的独角兽设置）：
web: bundle exec puma -C config/puma.rb
worker: bundle exec rake jobs:work
现在，创建 rake 任务非常简单。打开 lib/tasks 文件夹并添加一个名为 "scraper.rake" 的文件。这是您要放置 运行 抓取程序的代码的地方。由于您没有显示您的爬虫代码，我添加了一些我的代码，所以您有一个例子。首先，需要 运行 抓取工具所需的任何外部文件或配置。然后将所有内容包装在命名空间中（我选择了 :daily），但您可以选择任何内容以帮助使其更具描述性。然后，任务行应该被完全复制。 “:environment do”部分只是告诉 rails 使用模型和环境。而 运行宁耙。
require 'capybara/poltergeist'
namespace :daily do
desc "Scrapes hourly for much needed dataaaaa"
task :scrape => :environment do
Capybara.register_driver(:poltergeist) { |app| Capybara::Poltergeist::Driver.new(app, {js_errors: true, inspector: true, debug: false})} #, extensions: ["vendor/poltergeist/phantom_config.js"]}) }
Capybara.default_max_wait_time = 12
Capybara.default_driver = :poltergeist # configure Capybara to use poltergeist as the driver
Capybara.javascript_driver = :poltergeist
Capybara.current_driver = :poltergeist
@session = Capybara.current_session
EnvisionAccount.find_each do |envision_account|
# account = Scraper::HourlyScraper.new
daily_sign_in(envision_account)
end
end
end
一旦你的代码像这样设置好了，你就可以从命令行运行你的爬虫了：bundle exec rake daily:scrape如果一切正常，你只需要进入Heroku，添加调度程序插件，然后告诉它 运行 在您需要的任何时间间隔执行 rake 命令。另请注意，如果您使用水豚和闹鬼进行抓取，Heroku 需要一个构建包，因此 运行 this:
heroku buildpacks:add --index 1 https://github.com/stomita/heroku-buildpack-phantomjs
享受
我有一个免费的 heroku 帐户，我已经在其中部署了我的 rails 应用程序。我还启用了 redis heroku 插件。我的应用程序抓取网站，在我的本地环境中，它需要 93 分钟才能完成。所以我决定把它放在后台任务中并 运行 放在那里。我按照一些教程在免费的 heroku 帐户中获取后台任务 运行ning，所以我使用了独角兽服务器。在本地测试时，我的应用程序运行良好，但是一旦我将它部署到 heroku 并浏览我的项目站点，它就不会 work.Here 是 heroku log
2016-09-16T17:40:40.934711+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.943447+00:00 app[web.1]: E, [2016-09-16T17:40:40.943337 #3] ERROR -- : reaped #<Process::Status: pid 13987 exit 1> worker=2
2016-09-16T17:40:40.951820+00:00 app[web.1]: E, [2016-09-16T17:40:40.946468 #13992] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.951822+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.951825+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.951827+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.951828+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.951829+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.961433+00:00 app[web.1]: E, [2016-09-16T17:40:40.961347 #3] ERROR -- : reaped #<Process::Status: pid 13992 exit 1> worker=1
2016-09-16T17:40:40.966761+00:00 app[web.1]: E, [2016-09-16T17:40:40.953957 #13996] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.966764+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.966765+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.966766+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.966768+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.966769+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.966770+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.966771+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.968861+00:00 app[web.1]: E, [2016-09-16T17:40:40.944646 #13990] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.968863+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.968864+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.968865+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.968866+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.968867+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.968872+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.968871+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977007+00:00 app[web.1]: E, [2016-09-16T17:40:40.976925 #3] ERROR -- : reaped #<Process::Status: pid 13990 exit 1> worker=0
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.977910+00:00 app[web.1]: E, [2016-09-16T17:40:40.965528 #13999] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.977913+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.977915+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.977916+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.977917+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977919+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.981913+00:00 app[web.1]: E, [2016-09-16T17:40:40.981796 #3] ERROR -- : reaped #<Process::Status: pid 13996 exit 1> worker=2
2016-09-16T17:40:40.982020+00:00 app[web.1]: E, [2016-09-16T17:40:40.981968 #3] ERROR -- : reaped #<Process::Status: pid 13999 exit 1> worker=1
2016-09-16T17:40:40.998699+00:00 app[web.1]: E, [2016-09-16T17:40:40.985434 #14004] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.998703+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.998704+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.998705+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.998706+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.998707+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.998709+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.998710+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.013015+00:00 app[web.1]: E, [2016-09-16T17:40:41.012891 #3] ERROR -- : reaped #<Process::Status: pid 14004 exit 1> worker=0
2016-09-16T17:40:41.013269+00:00 app[web.1]: E, [2016-09-16T17:40:40.992746 #14009] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.013271+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.013273+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.013274+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.013275+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.013277+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.015082+00:00 app[web.1]: E, [2016-09-16T17:40:40.991259 #14006] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.015083+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.015084+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.015085+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.015087+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.015089+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.029456+00:00 app[web.1]: E, [2016-09-16T17:40:41.019600 #14013] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.029459+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.029465+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.029466+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.033517+00:00 app[web.1]: E, [2016-09-16T17:40:41.032905 #3] ERROR -- : reaped #<Process::Status: pid 14006 exit 1> worker=1
2016-09-16T17:40:41.033606+00:00 app[web.1]: E, [2016-09-16T17:40:41.033566 #3] ERROR -- : reaped #<Process::Status: pid 14009 exit 1> worker=2
2016-09-16T17:40:41.034410+00:00 app[web.1]: E, [2016-09-16T17:40:41.034370 #3] ERROR -- : reaped #<Process::Status: pid 14013 exit 1> worker=0
2016-09-16T17:40:41.067342+00:00 app[web.1]: E, [2016-09-16T17:40:41.057962 #14018] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.067346+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.067348+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.067350+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.067351+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.067352+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.077263+00:00 app[web.1]: E, [2016-09-16T17:40:41.077156 #3] ERROR -- : reaped #<Process::Status: pid 14018 exit 1> worker=1
这是我的独角兽配置
unicorn.rb
before_fork do |server, worker|
@sidekiq_pid ||= spawn("bundle exec sidekiq -c 2")
end
worker_processes 3
after_fork do |server, worker|
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
Sidekiq.configure_server do |config|
config.redis = { :size => 5 }
end
end
过程文件
web: bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb
首先，设置你的 procfile 以正确使用 worker dynos（我使用 puma for web 所以改变它以反映你的独角兽设置）：
web: bundle exec puma -C config/puma.rb
worker: bundle exec rake jobs:work
现在，创建 rake 任务非常简单。打开 lib/tasks 文件夹并添加一个名为 "scraper.rake" 的文件。这是您要放置 运行 抓取程序的代码的地方。由于您没有显示您的爬虫代码，我添加了一些我的代码，所以您有一个例子。首先，需要 运行 抓取工具所需的任何外部文件或配置。然后将所有内容包装在命名空间中（我选择了 :daily），但您可以选择任何内容以帮助使其更具描述性。然后，任务行应该被完全复制。 “:environment do”部分只是告诉 rails 使用模型和环境。而 运行宁耙。
require 'capybara/poltergeist'
namespace :daily do
desc "Scrapes hourly for much needed dataaaaa"
task :scrape => :environment do
Capybara.register_driver(:poltergeist) { |app| Capybara::Poltergeist::Driver.new(app, {js_errors: true, inspector: true, debug: false})} #, extensions: ["vendor/poltergeist/phantom_config.js"]}) }
Capybara.default_max_wait_time = 12
Capybara.default_driver = :poltergeist # configure Capybara to use poltergeist as the driver
Capybara.javascript_driver = :poltergeist
Capybara.current_driver = :poltergeist
@session = Capybara.current_session
EnvisionAccount.find_each do |envision_account|
# account = Scraper::HourlyScraper.new
daily_sign_in(envision_account)
end
end
end
一旦你的代码像这样设置好了，你就可以从命令行运行你的爬虫了：bundle exec rake daily:scrape如果一切正常，你只需要进入Heroku，添加调度程序插件，然后告诉它 运行 在您需要的任何时间间隔执行 rake 命令。另请注意，如果您使用水豚和闹鬼进行抓取，Heroku 需要一个构建包，因此 运行 this:
heroku buildpacks:add --index 1 https://github.com/stomita/heroku-buildpack-phantomjs
享受