在 rails 中使用插入查询遍历大数据的最佳方法是什么

Question

我必须插入大数据，比方说 20k 我怀疑我是否编写了优化查询。

它的作用：

1. 获取满足某些条件的 sql 用户的记录 它在 merge_user_records
的活动记录数组中获取超过 1k-20k 的用户
2. 从 1k-20k 用户的 activerecord 数组中分批处理 100 个用户的切片
3. 遍历merge_user条记录，从merge_user_records
中的User模型用户user_id中查找用户
4. 仍在循环中，用户调用方法 construct_user_notifications 为每个用户插入 user_notifications。
5. 仍在循环中查找用户的设备。
6. 运行 设备中的一个循环，用于在每个设备上发送推送通知。
7. 循环结束

这是代码。

merge_users = MergeField.get_user_field_values(notification_template.merge_field, scope_users) #=> **returns users 1k - 20k**
if merge_users.present?
  merge_users.each_slice(100) do |record|
    record.each do |user_record|
      user = User.find_by(id: user_record.user_id)
      text = notification_template.title
      notification_template.description = MustacheDescription.render(notification_template, user_record)
      text += " " + notification_template.description
      Rails.logger.info "Merge field message: #{notification_template.description}"
      construct_user_notifications(notification_template, user_record.user_id) #=> **this calls another method below which actually create notifications for every user.**
      badge = (notification_template.display_screen == "suggestion") ? user.unread_suggestion_notifications.count : user.unread_option_notifications.count
      devices = user.devices.with_notification_token
      if devices.present?
        devices.each do |device|
          PushNotification.notify_ios(text, device.notification_token, badge, {screen: notification_template.display_screen})
          Rails.logger.info "Sending push to user_id #{user_record.user_id} token #{device.notification_token}"
        end
      end
    end
  end
end

def self.construct_user_notifications(notification_template, user_id)
  notification_template.user_notifications.build.tap do |user_notification|
    user_notification.title = notification_template.title
    user_notification.subtitle = notification_template.subtitle
    user_notification.description = notification_template.description
    user_notification.merge_field = notification_template.merge_field
    user_notification.cta = notification_template.cta
    user_notification.cta_key = notification_template.cta_key
    user_notification.secondary_cta = notification_template.secondary_cta
    user_notification.secondary_cta_key = notification_template.secondary_cta_key
    user_notification.show_useful = notification_template.show_useful
    user_notification.category = notification_template.category
    user_notification.display_screen = notification_template.display_screen
    user_notification.sent_at = Time.current
    user_notification.user_id = user_id
    user_notification.filter_preferences = notification_template.filter_preferences
    user_notification.save
  end
end

我已经为 100 位用户测试了这个，需要 30-40 秒。天知道 20k 用户在产品中需要多少。

Answer 1

我建议将循环的内部内容包装在一个 t运行saction 块中，这将在最后 运行 所有查询一次完成，而不是零碎的。这会将每个用户的所有查询分组到一个 t运行saction 中，同时 运行:

merge_users.each_slice(100) do |record|
  ActiveRecord::Base.transaction do
  // code
  end

  if devices.present?
    devices.each do |device|
      PushNotification.notify_ios(text,device.notification_token,badge,{screen: notification_template.display_screen})
      Rails.logger.info  "Sending push to user_id #{user_record.user_id} token #{device.notification_token}"
    end
  end
end

您可以在此处找到有关 t运行sactions 的更多信息：

http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html

最后，我建议不要直接在块中执行 PushNotification.notifify_ios，而应使用 DelayedJob 或类似于 运行 的后台作业。这意味着所有方法调用将在代码本身具有 运行 之后在后台处理，而不是在循环本身内处理。

看起来像：

if devices.present?
    devices.each do |device|
      PushNotification.delay.notify_ios(text,device.notification_token,badge,{screen: notification_template.display_screen})
      Rails.logger.info  "Sending push to user_id #{user_record.user_id} token #{device.notification_token}"
    end
  end

https://github.com/collectiveidea/delayed_job

Answer 2

1。 find_by 在循环中

user = User.find_by(id: user.user_id) 会查询 20k 次！我们可以通过将其置于 each 循环之外来避免这种情况：

merge_users.each_slice(100) do |users|
  users = User.where(id: users.map(&:user_id))
  users.each do |user|
    # loop
  end
end

---

2。记录器级别

将 info 更改为 debug。磁盘 IO 总是很慢。

---

3。 construct_user_notifications 函数

notification_template.user_notifications.build 将分配 20k 个对象。 GC 也会有问题。

请只构建属性，稍后保存。

例如：

def self.construct_user_notifications(notification_template, user_id)
  {
      title: notification_template.title,
      subtitle: notification_template.subtitle,
      description: notification_template.description,
      merge_field: notification_template.merge_field,
      cta: notification_template.cta,
      cta_key: notification_template.cta_key,
      secondary_cta: notification_template.secondary_cta,
      secondary_cta_key: notification_template.secondary_cta_key,
      show_useful: notification_template.show_useful,
      category: notification_template.category,
      display_screen: notification_template.display_screen,
      sent_at: Time.current,
      user_id: user_id,
      filter_preferences: notification_template.filter_preferences, 
      # more attributes
  }
end

---

4。 badge 个查询

badge = (notification_template.display_screen == "suggestion") ? user.unread_suggestion_notifications.count : user.unread_option_notifications.count

这是不必要的，除非 devices 存在。

以后可以查询徽章。

---

5。推送通知

PushNotification.notify_ios(text, device.notification_token, badge, {screen: notification_template.display_screen})

这可能有一些 http 请求，这很慢。

您应该使用 sidekiq 或 resque 在后台作业中完成。

6。保存 user_notifications

看看activerecord-import gem。批量插入效率更高

例子

merge_users = MergeField.get_user_field_values(notification_template.merge_field, scope_users) #=> **returns users 1k - 20k**

merge_users.each_slice(500) do |users|
  users = User.where(id: users.map(&:user_id))
  user_notifications = Set.new

  users.each do |user|
    text = notification_template.title
    notification_template.description = MustacheDescription.render(notification_template, user)
    text += " " + notification_template.description
    Rails.logger.debug "Merge field message: #{notification_template.description}"

    user_notifications.add construct_user_notifications(notification_template, user.user_id)

    # do this asynchronously
    push_notification(notification_template, user_id)
  end

  UserNotification.import(user_notifications.first.keys, user_notifications.to_a)
end

def self.push_notification(notification_template, user_id)
  devices = Device.where(user_id: user_id).with_notification_token.pluck(:notification_token)
  if devices.present?
    badge = (notification_template.display_screen == "suggestion") ? UnreadSuggestionNotification.where(user_id: user_id).count : UnreadOptionNotification.where(user_id: user_id).count
    devices.each do |device|
      PushNotification.notify_ios(text, device.notification_token, badge, {screen: notification_template.display_screen})
      Rails.logger.debug "Sending push to user_id #{user_id} token #{device.notification_token}"
    end
  end
end


def self.construct_user_notifications(notification_template, user_id)
  {
      title: notification_template.title,
      subtitle: notification_template.subtitle,
      description: notification_template.description,
      merge_field: notification_template.merge_field,
      cta: notification_template.cta,
      cta_key: notification_template.cta_key,
      secondary_cta: notification_template.secondary_cta,
      secondary_cta_key: notification_template.secondary_cta_key,
      show_useful: notification_template.show_useful,
      category: notification_template.category,
      display_screen: notification_template.display_screen,
      sent_at: Time.current,
      user_id: user_id,
      filter_preferences: notification_template.filter_preferences,
      # more attributes
  }
end