错误 运行 rails 上的抓取命令 ruby
error running a scrape command ruby on rails
我正在尝试在一台新计算机上重新设置我的应用程序，并且 运行 正在努力构建数据库。当我 运行 我的第一个耙子 scraper:scrape 时，这是我得到的错误。我不确定为什么会收到此错误任何帮助都会让我开心..干杯！
Art West@ARTWESTIV ~/desktop/duckduckjeep-master
$ rake scraper:scrape --trace
** Invoke scraper:scrape (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute scraper:scrape
rake aborted!
NoMethodError: undefined method `value' for nil:NilClass
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:17:in `block (3 levels) in <top (required)>'
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in `loop'
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in `block (2 levels) in <top (required)>'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/bin/rake:33:in `<top (required)>'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in `load'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in `<main>'
Tasks: TOP => scraper:scrape
这是我的 scraper.rake
namespace :scraper do
desc "Fetch Craigslist posts from 3Taps"
task scrape: :environment do
require 'open-uri'
require 'json'
# Set API token and URL
auth_token = "b077632d17da8857e2fa92c053115e43"
polling_url = "http://polling.3taps.com/poll"
# Grab data until up-to-date
loop do
# Specify request parameters
params = {
auth_token: auth_token,
anchor: Anchor.first.value,
source:"CRAIG",
category_group: "VVVV",
category: "VAUT",
'location.country' => "USA",
retvals: "location,external_url,heading,body,timestamp,price,images,annotations"
}
# Prepare API request
uri = URI.parse(polling_url)
uri.query = URI.encode_www_form(params)
# Submit request
result = JSON.parse(open(uri).read)
# Display results to screen
#puts result["postings"].first["annotations"]["year"]
Anchor.first.update(value: result["anchor"])
puts Anchor.first.value
break if result["postings"].empty?
# #store results in Database
result["postings"].each do |posting|
#ADD HARD FILTER (IN PROGRESS....)
if posting["annotations"]["make"] == "Jeep"
#create new post
@post= Post.new
@post.heading = posting["heading"]
@post.body = posting["body"]
@post.price = posting["price"]
@post.neighborhood = posting["location"]["locality"]
@post.external_url = posting["external_url"]
@post.timestamp = posting["timestamp"]
@post.year = posting ["annotations"]["year"] if posting ["annotations"]["year"].present?
@post.make = posting ["annotations"]["make"] if posting ["annotations"]["make"].present?
@post.model = posting ["annotations"]["model"] if posting ["annotations"]["model"].present?
@post.title_status = posting ["annotations"]["title_status"] if posting ["annotations"]["title_status"].present?
@post.transmission = posting ["annotations"]["transmission"] if posting ["annotations"]["transmission"].present?
@post.mileage = posting ["annotations"]["mileage"] if posting ["annotations"]["mileage"].present?
@post.source_account = posting ["annotations"]["source_account"] if posting ["annotations"]["source_account"].present?
@post.phone = posting ["annotations"]["phone"] if posting ["annotations"]["phone"].present?
@post.lat = posting["location"]["lat"]
@post.lng = posting["location"]["long"]
@post.zipcode = posting["location"]["zipcode"]
#Save Post
@post.save
# Loop over images and save to Image database
posting["images"].each do |image|
@image = Image.new
@image.url = image["full"]
@image.post_id = @post.id
@image.save
end
end
end
end
end
desc "Destroy All Posting Data"
task destroy_all_posts: :environment do
Post.destroy_all
end
desc "Save neighborhood codes in a reference table"
task scrape_neighborhoods: :environment do
require 'open-uri'
require 'json'
# Set API token and URL
auth_token = "b077632d17da8857e2fa92c053115e43"
location_url = "http://reference.3taps.com/locations"
# Specify request parameters
params = {
auth_token: auth_token,
level: "locality",
country: "USA"
}
# Prepare API request
uri = URI.parse(location_url)
uri.query = URI.encode_www_form(params)
# Submit request
result = JSON.parse(open(uri).read)
# Display results to screen
# puts JSON.pretty_generate result
# Store results in database
result["locations"].each do |location|
@location = Location.new
@location.code = location["code"]
@location.name = location["short_name"]
@location.save
end
end
desc "Discard old data"
task discard_old_data: :environment do
Post.all.each do |post|
if post.created_at < 72.hours.ago
post.destoy
end
end
end
end
您的错误消息说明了一切：
NoMethodError: undefined method `value' for nil:NilClass
您正试图在 nil 上的某个时刻致电 value。而且，这可能发生在这里：
# Specify request parameters
params = {
auth_token: auth_token,
anchor: Anchor.first.value, # this is the line that's creating problem
source:"CRAIG",
category_group: "VVVV",
category: "VAUT",
'location.country' => "USA",
retvals: "location,external_url,heading,body,timestamp,price,images,annotations"
}
当你第一次调用这个：anchor: Anchor.first.value 时，你的 Anchor.first 是 nil 所以你试图调用：nil.value 这就是你的地方rake 任务失败并显示指定的错误消息。
确保您已填充数据库 table (anchors)，这样当您调用 Anchor.first 时，您不会得到 nil.
另一种避免此问题的方法是使用 try:
anchor: Anchor.first.try(:value)
这样，即使 Anchor.first returns nil.
你的 rake 任务也不会失败
我正在尝试在一台新计算机上重新设置我的应用程序，并且 运行 正在努力构建数据库。当我 运行 我的第一个耙子 scraper:scrape 时，这是我得到的错误。我不确定为什么会收到此错误任何帮助都会让我开心..干杯！
Art West@ARTWESTIV ~/desktop/duckduckjeep-master
$ rake scraper:scrape --trace
** Invoke scraper:scrape (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute scraper:scrape
rake aborted!
NoMethodError: undefined method `value' for nil:NilClass
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:17:in `block (3 levels) in <top (required)>'
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in `loop'
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in `block (2 levels) in <top (required)>'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/bin/rake:33:in `<top (required)>'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in `load'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in `<main>'
Tasks: TOP => scraper:scrape
这是我的 scraper.rake
namespace :scraper do
desc "Fetch Craigslist posts from 3Taps"
task scrape: :environment do
require 'open-uri'
require 'json'
# Set API token and URL
auth_token = "b077632d17da8857e2fa92c053115e43"
polling_url = "http://polling.3taps.com/poll"
# Grab data until up-to-date
loop do
# Specify request parameters
params = {
auth_token: auth_token,
anchor: Anchor.first.value,
source:"CRAIG",
category_group: "VVVV",
category: "VAUT",
'location.country' => "USA",
retvals: "location,external_url,heading,body,timestamp,price,images,annotations"
}
# Prepare API request
uri = URI.parse(polling_url)
uri.query = URI.encode_www_form(params)
# Submit request
result = JSON.parse(open(uri).read)
# Display results to screen
#puts result["postings"].first["annotations"]["year"]
Anchor.first.update(value: result["anchor"])
puts Anchor.first.value
break if result["postings"].empty?
# #store results in Database
result["postings"].each do |posting|
#ADD HARD FILTER (IN PROGRESS....)
if posting["annotations"]["make"] == "Jeep"
#create new post
@post= Post.new
@post.heading = posting["heading"]
@post.body = posting["body"]
@post.price = posting["price"]
@post.neighborhood = posting["location"]["locality"]
@post.external_url = posting["external_url"]
@post.timestamp = posting["timestamp"]
@post.year = posting ["annotations"]["year"] if posting ["annotations"]["year"].present?
@post.make = posting ["annotations"]["make"] if posting ["annotations"]["make"].present?
@post.model = posting ["annotations"]["model"] if posting ["annotations"]["model"].present?
@post.title_status = posting ["annotations"]["title_status"] if posting ["annotations"]["title_status"].present?
@post.transmission = posting ["annotations"]["transmission"] if posting ["annotations"]["transmission"].present?
@post.mileage = posting ["annotations"]["mileage"] if posting ["annotations"]["mileage"].present?
@post.source_account = posting ["annotations"]["source_account"] if posting ["annotations"]["source_account"].present?
@post.phone = posting ["annotations"]["phone"] if posting ["annotations"]["phone"].present?
@post.lat = posting["location"]["lat"]
@post.lng = posting["location"]["long"]
@post.zipcode = posting["location"]["zipcode"]
#Save Post
@post.save
# Loop over images and save to Image database
posting["images"].each do |image|
@image = Image.new
@image.url = image["full"]
@image.post_id = @post.id
@image.save
end
end
end
end
end
desc "Destroy All Posting Data"
task destroy_all_posts: :environment do
Post.destroy_all
end
desc "Save neighborhood codes in a reference table"
task scrape_neighborhoods: :environment do
require 'open-uri'
require 'json'
# Set API token and URL
auth_token = "b077632d17da8857e2fa92c053115e43"
location_url = "http://reference.3taps.com/locations"
# Specify request parameters
params = {
auth_token: auth_token,
level: "locality",
country: "USA"
}
# Prepare API request
uri = URI.parse(location_url)
uri.query = URI.encode_www_form(params)
# Submit request
result = JSON.parse(open(uri).read)
# Display results to screen
# puts JSON.pretty_generate result
# Store results in database
result["locations"].each do |location|
@location = Location.new
@location.code = location["code"]
@location.name = location["short_name"]
@location.save
end
end
desc "Discard old data"
task discard_old_data: :environment do
Post.all.each do |post|
if post.created_at < 72.hours.ago
post.destoy
end
end
end
end
您的错误消息说明了一切：
NoMethodError: undefined method `value' for nil:NilClass
您正试图在 nil 上的某个时刻致电 value。而且，这可能发生在这里：
# Specify request parameters
params = {
auth_token: auth_token,
anchor: Anchor.first.value, # this is the line that's creating problem
source:"CRAIG",
category_group: "VVVV",
category: "VAUT",
'location.country' => "USA",
retvals: "location,external_url,heading,body,timestamp,price,images,annotations"
}
当你第一次调用这个：anchor: Anchor.first.value 时，你的 Anchor.first 是 nil 所以你试图调用：nil.value 这就是你的地方rake 任务失败并显示指定的错误消息。
确保您已填充数据库 table (anchors)，这样当您调用 Anchor.first 时，您不会得到 nil.
另一种避免此问题的方法是使用 try:
anchor: Anchor.first.try(:value)
这样，即使 Anchor.first returns nil.