Rails如何正确使用Sidekiq处理后台任务
How to properly use Sidekiq to process background tasks in Rails
因此，我使用 https://github.com/Shopify/shopify_app 生成了一个 rails 应用程序 - 大部分情况下该应用程序都按预期工作 - 它的目标是从外部库存 management API，然后使用该库存 management 系统中的最新数量更新 Shopify 中的变体数量。
我的问题是，对外部 API 的初始 POST 请求会返回大量产品 - 有时需要 15 秒以上的时间。除此之外，我的应用程序的另一部分然后接受此响应，并且对于响应中也存在于 Shopify 中的每个产品，它将向 Shopify 发出 PUT 请求以更新变体数量。与初始请求一样，这也需要 10-15 秒以上。
我的问题是我在 Heroku 上托管该应用程序，结果我达到了他们的 30 秒请求超时限制。因此，我需要使用后台工作人员将至少一个上述请求（可能两个）偏移到工作队列。我已经使用了广泛推荐的 Sidekiq gem - https://github.com/mperham/sidekiq - 它很容易设置。
我的问题是我不知道如何从完成的 Sidekiq worker 作业中获取结果，然后在 Controller 中再次使用它 - 我也不知道这是否是最佳实践（我是Rails/App 开发有点新。
我已经包括了我的控制器（在将其分解为工作人员之前），该控制器目前正在运行下面的应用程序 - 我想我只需要一些建议 - 我这样做是否正确 - 这些逻辑中的一些是否应该在模型中，如果是这样，该模型将如何与控制器通信，然后 Sidekiq 将如何适应所有这些。
感谢任何建议或帮助，谢谢。
class StockManagementController < ShopifyApp::AuthenticatedController
require 'uri'
require 'net/http'
require 'json'
require 'nokogiri'
require 'open-uri'
require 'rexml/document'
def new
@token = StockManagementController.new
end
def get_token
url = URI('https://external.api.endpoint/api/v1/AuthToken')
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
@HEROKU_ENV_USERNAME = ENV['HEROKU_ENV_USERNAME']
@HEROKU_ENV_PASSWORD = ENV['HEROKU_ENV_PASSWORD']
request = Net::HTTP::Post.new(url)
request['content-type'] = 'application/x-www-form-urlencoded'
request['cache-control'] = 'no-cache'
request.body = 'username=' + @HEROKU_ENV_USERNAME + '&password=' + @HEROKU_ENV_PASSWORD + '&grant_type=password'
response = http.request(request)
responseJSON = JSON.parse(response.read_body)
session[:accessToken] = responseJSON['access_token']
if session[:accessToken]
flash[:notice] = 'StockManagement token generation was successful.'
redirect_to '/StockManagement/product_quantity'
else
flash[:alert] = 'StockManagement token generation was unsuccessful.'
end
end
def product_quantity
REXML::Document.entity_expansion_text_limit = 1_000_000
@theToken = session[:accessToken]
if @theToken
url = URI('https://external.api.endpoint/api/v1/ProductQuantity')
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(url)
request['authorization'] = 'bearer ' + @theToken + ''
request['content-type'] = 'application/xml'
request['cache-control'] = 'no-cache'
response = http.request(request)
responseBody = response.read_body
finalResponse = Hash.from_xml(responseBody).to_json
resultQuantity = JSON.parse finalResponse
@connectionType = resultQuantity['AutomatorResponse']['Type']
@successResponse = resultQuantity['AutomatorResponse']['Success']
@errorResponse = resultQuantity['AutomatorResponse']['ErrorMsg']
productQuantityResponse = resultQuantity['AutomatorResponse']['ResponseString']
xmlResponse = Hash.from_xml(productQuantityResponse).to_json
jsonResponse = JSON.parse xmlResponse
@fullResponse = jsonResponse['StockManagement']['Company']['InventoryQuantitiesByLocation']['InventoryQuantity']
# This hash is used to store the final list of items that we need in order to display the item's we've synced, and to show the number of items we've sycned successfully.
@finalList = Hash.new
# This array is used to contain the available products - this is used later on as a way of only rendering
@availableProducts = Array.new
# Here we get all of the variant data from Shopify.
@variants = ShopifyAPI::Variant.find(:all, params: {})
# For each peace of variant data, we push all of the available SKUs in the store to the @availableProducts Array for use later
@variants.each do |variant|
@availableProducts << variant.sku
end
#Our final list of products which will contain details from both the Stock Management company and Shopify - we will use this list to run api calls against each item
@finalProductList = Array.new
puts "Final product list has #{@fullResponse.length} items."
puts @fullResponse.inspect
# We look through every item in the response from Company
@fullResponse.each_with_index do |p, index|
# We get the Quantity and Product Code
@productQTY = p["QtyOnHand"].to_f.round
@productCode = p["Code"].upcase
# If the product code is found in the list of available products in the Shopify store...
if @availableProducts.include? @productCode
@variants.each do |variant|
if @productCode === variant.sku
if @productQTY != 0
@finalProductList << {
"sku" => variant.sku,
"inventory_quantity" => variant.inventory_quantity,
"old_inventory_quantity" => variant.old_inventory_quantity,
"id" => variant.id,
"company_sku" => @productCode,
"company_qty" => @productQTY
}
end
end
end
end
end
# If we get a successful response from StockManagement, proceed...
if @finalProductList
flash[:notice] = 'StockManagement product quantity check was successful.'
puts "Final product list has #{@finalProductList.length} items."
puts @finalProductList
@finalProductList.each do |item|
@productSKU = item["sku"]
@productInventoryQuantity = item["inventory_quantity"]
@productOldInventoryQuantity = item["old_inventory_quantity"]
@productID = item["id"]
@companySKU = item["company_sku"]
@companyQTY = item["company_qty"]
url = URI("https://example.myshopify.com/admin/variants/#{@productID}.json")
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Put.new(url)
request["content-type"] = 'application/json'
request["authorization"] = 'Basic KJSHDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDF'
request["cache-control"] = 'no-cache'
request.body = "{\n\t\"variant\": {\n\t\t\"id\": #{@productID},\n\t\t\"inventory_quantity\": #{@companyQTY},\n\t\t\"old_inventory_quantity\": #{@productOldInventoryQuantity}\n\t}\n}"
# This is the line that actually runs the put request to update the quantity.
response = http.request(request)
# Finally, we populate the finalList has with response information.
@finalList[@companySKU] = ["","You had #{@productOldInventoryQuantity} in stock, now you have #{@companyQTY} in stock."]
end
else
# If the overall sync failed, we flash an alert.
flash[:alert] = 'Quantity synchronisation was unsuccessful.'
end
# Lastly we get the final number of items that were synchronised.
@synchronisedItems = @finalList.length
# We flash this notification, letting the user known how many products were successfully synchronised.
flash[:notice] = "#{@synchronisedItems} product quantities were synchronised successfully."
# We then pretty print this to the console for debugging purposes.
pp @finalList
else
flash[:alert] = @errorResponse
end
end
end
首先，您的 product_quantity 方法太长了。你应该把它分成更小的部分。第二，http.verify_mode = OpenSSL::SSL::VERIFY_NONE 不应该在生产中完成。您提供的示例以及您的问题太复杂，因此难以回答。听起来您需要对设计模式有基本的了解，这不是一个特定的 ruby 问题。
如果您的应用需要在控制器内部进行实时 API 调用，这是一个糟糕的设计。您不想让任何类型的请求最多等待几秒钟。您应该首先考虑为什么需要提出这些请求。如果它是您需要快速访问的数据，您应该编写后台作业以按计划抓取数据并将其存储在您自己的数据库中。
如果您的应用的用户发出需要等待 API 响应的请求，您可以编写一个 worker 来处理获取 API 数据并最终将响应发送到用户的浏览器可能使用 actioncable.
对于您的常量定义，您可能应该在初始化程序中执行此操作，您将保留在 my_app_root/config/initializers/constants.rb 中，它会在运行时加载到您的应用程序中。您可以在需要的地方使用 te ENV[] 语法调用它们，但如果您更喜欢更简单的常量，请删除 @，因为 ruby 中的命名约定是实例对象。
#app_root/config/initializers/constants.rb
HEROKU_ENV_USERNAME = ENV['HEROKU_ENV_USERNAME']
HEROKU_ENV_PASSWORD = ENV['HEROKU_ENV_PASSWORD']
因此，我使用 https://github.com/Shopify/shopify_app 生成了一个 rails 应用程序 - 大部分情况下该应用程序都按预期工作 - 它的目标是从外部库存 management API，然后使用该库存 management 系统中的最新数量更新 Shopify 中的变体数量。
我的问题是，对外部 API 的初始 POST 请求会返回大量产品 - 有时需要 15 秒以上的时间。除此之外，我的应用程序的另一部分然后接受此响应，并且对于响应中也存在于 Shopify 中的每个产品，它将向 Shopify 发出 PUT 请求以更新变体数量。与初始请求一样，这也需要 10-15 秒以上。
我的问题是我在 Heroku 上托管该应用程序，结果我达到了他们的 30 秒请求超时限制。因此，我需要使用后台工作人员将至少一个上述请求（可能两个）偏移到工作队列。我已经使用了广泛推荐的 Sidekiq gem - https://github.com/mperham/sidekiq - 它很容易设置。
我的问题是我不知道如何从完成的 Sidekiq worker 作业中获取结果，然后在 Controller 中再次使用它 - 我也不知道这是否是最佳实践（我是Rails/App 开发有点新。
我已经包括了我的控制器（在将其分解为工作人员之前），该控制器目前正在运行下面的应用程序 - 我想我只需要一些建议 - 我这样做是否正确 - 这些逻辑中的一些是否应该在模型中，如果是这样，该模型将如何与控制器通信，然后 Sidekiq 将如何适应所有这些。
感谢任何建议或帮助，谢谢。
class StockManagementController < ShopifyApp::AuthenticatedController
require 'uri'
require 'net/http'
require 'json'
require 'nokogiri'
require 'open-uri'
require 'rexml/document'
def new
@token = StockManagementController.new
end
def get_token
url = URI('https://external.api.endpoint/api/v1/AuthToken')
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
@HEROKU_ENV_USERNAME = ENV['HEROKU_ENV_USERNAME']
@HEROKU_ENV_PASSWORD = ENV['HEROKU_ENV_PASSWORD']
request = Net::HTTP::Post.new(url)
request['content-type'] = 'application/x-www-form-urlencoded'
request['cache-control'] = 'no-cache'
request.body = 'username=' + @HEROKU_ENV_USERNAME + '&password=' + @HEROKU_ENV_PASSWORD + '&grant_type=password'
response = http.request(request)
responseJSON = JSON.parse(response.read_body)
session[:accessToken] = responseJSON['access_token']
if session[:accessToken]
flash[:notice] = 'StockManagement token generation was successful.'
redirect_to '/StockManagement/product_quantity'
else
flash[:alert] = 'StockManagement token generation was unsuccessful.'
end
end
def product_quantity
REXML::Document.entity_expansion_text_limit = 1_000_000
@theToken = session[:accessToken]
if @theToken
url = URI('https://external.api.endpoint/api/v1/ProductQuantity')
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(url)
request['authorization'] = 'bearer ' + @theToken + ''
request['content-type'] = 'application/xml'
request['cache-control'] = 'no-cache'
response = http.request(request)
responseBody = response.read_body
finalResponse = Hash.from_xml(responseBody).to_json
resultQuantity = JSON.parse finalResponse
@connectionType = resultQuantity['AutomatorResponse']['Type']
@successResponse = resultQuantity['AutomatorResponse']['Success']
@errorResponse = resultQuantity['AutomatorResponse']['ErrorMsg']
productQuantityResponse = resultQuantity['AutomatorResponse']['ResponseString']
xmlResponse = Hash.from_xml(productQuantityResponse).to_json
jsonResponse = JSON.parse xmlResponse
@fullResponse = jsonResponse['StockManagement']['Company']['InventoryQuantitiesByLocation']['InventoryQuantity']
# This hash is used to store the final list of items that we need in order to display the item's we've synced, and to show the number of items we've sycned successfully.
@finalList = Hash.new
# This array is used to contain the available products - this is used later on as a way of only rendering
@availableProducts = Array.new
# Here we get all of the variant data from Shopify.
@variants = ShopifyAPI::Variant.find(:all, params: {})
# For each peace of variant data, we push all of the available SKUs in the store to the @availableProducts Array for use later
@variants.each do |variant|
@availableProducts << variant.sku
end
#Our final list of products which will contain details from both the Stock Management company and Shopify - we will use this list to run api calls against each item
@finalProductList = Array.new
puts "Final product list has #{@fullResponse.length} items."
puts @fullResponse.inspect
# We look through every item in the response from Company
@fullResponse.each_with_index do |p, index|
# We get the Quantity and Product Code
@productQTY = p["QtyOnHand"].to_f.round
@productCode = p["Code"].upcase
# If the product code is found in the list of available products in the Shopify store...
if @availableProducts.include? @productCode
@variants.each do |variant|
if @productCode === variant.sku
if @productQTY != 0
@finalProductList << {
"sku" => variant.sku,
"inventory_quantity" => variant.inventory_quantity,
"old_inventory_quantity" => variant.old_inventory_quantity,
"id" => variant.id,
"company_sku" => @productCode,
"company_qty" => @productQTY
}
end
end
end
end
end
# If we get a successful response from StockManagement, proceed...
if @finalProductList
flash[:notice] = 'StockManagement product quantity check was successful.'
puts "Final product list has #{@finalProductList.length} items."
puts @finalProductList
@finalProductList.each do |item|
@productSKU = item["sku"]
@productInventoryQuantity = item["inventory_quantity"]
@productOldInventoryQuantity = item["old_inventory_quantity"]
@productID = item["id"]
@companySKU = item["company_sku"]
@companyQTY = item["company_qty"]
url = URI("https://example.myshopify.com/admin/variants/#{@productID}.json")
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Put.new(url)
request["content-type"] = 'application/json'
request["authorization"] = 'Basic KJSHDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDF'
request["cache-control"] = 'no-cache'
request.body = "{\n\t\"variant\": {\n\t\t\"id\": #{@productID},\n\t\t\"inventory_quantity\": #{@companyQTY},\n\t\t\"old_inventory_quantity\": #{@productOldInventoryQuantity}\n\t}\n}"
# This is the line that actually runs the put request to update the quantity.
response = http.request(request)
# Finally, we populate the finalList has with response information.
@finalList[@companySKU] = ["","You had #{@productOldInventoryQuantity} in stock, now you have #{@companyQTY} in stock."]
end
else
# If the overall sync failed, we flash an alert.
flash[:alert] = 'Quantity synchronisation was unsuccessful.'
end
# Lastly we get the final number of items that were synchronised.
@synchronisedItems = @finalList.length
# We flash this notification, letting the user known how many products were successfully synchronised.
flash[:notice] = "#{@synchronisedItems} product quantities were synchronised successfully."
# We then pretty print this to the console for debugging purposes.
pp @finalList
else
flash[:alert] = @errorResponse
end
end
end
首先，您的 product_quantity 方法太长了。你应该把它分成更小的部分。第二，http.verify_mode = OpenSSL::SSL::VERIFY_NONE 不应该在生产中完成。您提供的示例以及您的问题太复杂，因此难以回答。听起来您需要对设计模式有基本的了解，这不是一个特定的 ruby 问题。
如果您的应用需要在控制器内部进行实时 API 调用，这是一个糟糕的设计。您不想让任何类型的请求最多等待几秒钟。您应该首先考虑为什么需要提出这些请求。如果它是您需要快速访问的数据，您应该编写后台作业以按计划抓取数据并将其存储在您自己的数据库中。
如果您的应用的用户发出需要等待 API 响应的请求，您可以编写一个 worker 来处理获取 API 数据并最终将响应发送到用户的浏览器可能使用 actioncable.
对于您的常量定义，您可能应该在初始化程序中执行此操作，您将保留在 my_app_root/config/initializers/constants.rb 中，它会在运行时加载到您的应用程序中。您可以在需要的地方使用 te ENV[] 语法调用它们，但如果您更喜欢更简单的常量，请删除 @，因为 ruby 中的命名约定是实例对象。
#app_root/config/initializers/constants.rb
HEROKU_ENV_USERNAME = ENV['HEROKU_ENV_USERNAME']
HEROKU_ENV_PASSWORD = ENV['HEROKU_ENV_PASSWORD']