扩展 firebase firestore 预定事件

Question

我有一个每 4 小时运行一次的 pubsub 事件。我想在预定事件后保存所有用户的团队。我是批量写入，但批量写入的每次提交限制为 500 次写入。下面是一个类似于我正在尝试做的示例代码。

问题：如何自动缩放批量写入。

exports.updateNews = functions.pubsub
  .topic("my-scheduled-topic")
  .onPublish(message => {
    return axios
      .get(
        "https://newsapi.org/v2/top-headlines?apiKey=someKey&sources=espn-cric-info"
      )
      .then(result => {
        const batch = db.batch();
        result.data.articles.forEach(article => {
          const docRef = db.collection("news").doc();
          batch.set(docRef, article);
        });
        return batch.commit();
      })
      .then(result => {
        console.log(result);
        return result;
      })
      .catch(error => {
        console.log(error);
        return error;
      });
  });

Answer 1

如果您的批次遇到 500 次写入的限制，您可以改用 Promise.all()，如下所示。当 return 由 add() 方法调用的所有承诺都已解决时，它将 return 解决一个单一的承诺。

exports.updateNews = functions.pubsub
  .topic("my-scheduled-topic")
  .onPublish(message => {
    return axios
      .get(
        "https://newsapi.org/v2/top-headlines?apiKey=someKey&sources=espn-cric-info"
      )
      .then(result => {
        const promises = [];
        result.data.articles.forEach(article => {
           promises.push(db.collection("news").add(article));
        });
        return Promise.all(promises);
      })
      .then(results => {
        console.log(results);
        return null;
      })
      .catch(error => {
        console.log(error);
        return null;
      });
  });

Answer 2

我想我自己解决了需要专家意见：

我知道我的问题代码看起来不像答案中的代码我在提问之前简化了问题。

\\ reference the collection I wanted to save to or the place I wanna write
var userTeamsSave = db.collection("saveTeams");

\\ api call
  db.collection("news")
    .get()
    .then(querySnapshot => {

     \\ create Json array from query snapshot

      let users = [];
      querySnapshot.forEach(user => {
        users.push({ id: user.id, data: user.data() });
      });

      return users;

    })
    .then(users => {

      var counter = 0;
      var commitCounter = 0;
      var batches = [];

      \\ array of batches
      batches[commitCounter] = db.batch();
      users.forEach(user => {

      \\ limit batch write in 1 commit upto 499
        if (counter <= 200) {

          var thisRef = userTeamsSave.doc(user.id);
          batches[commitCounter].set(thisRef, user.data);
          counter = counter + 1;

        } else {
        \\ Reset Counter
          counter = 0;
          thisRef = userTeamsSave.doc(user.id);
          batches[commitCounter].set(thisRef, user.data);
          commitCounter = commitCounter + 1;
          batches[commitCounter] = db.batch();

        }
      });

      \\ return all batches
      return batches;
    })
    .then(fullBatch => {
      fullBatch.forEach(batch => {
        console.count("wrote batch");

       \\ commit all batches
        return batch.commit();
      });
      return;
    })
    .catch(error => {
      console.error(error);
      return error;
    });

扩展 firebase firestore 预定事件

Scaling firebase firestore scheduled event

firebase

google-cloud-pubsub

google-cloud-functions

google-cloud-firestore