Google Cloud Functions 不在 PubSub 上发布，超过超时

Question

首先感谢您阅读本文并尝试回答:)

在 GCP（Google 云平台）上，我有一个存储在云端的数据库 SQL，

2个云函数，第一个用于向数据库发出请求（这个请求可以有超过1000个结果）然后将结果发布到PubSub，第二个用于做网页抓取感谢木偶师。

第一个云函数代码：

//[Requirement]
const mysql = require('mysql')
const {SecretManagerServiceClient} = require('@google-cloud/secret-manager')
const ProjectID = process.env.secretID
const SqlPass = `projects/xxx`
const client = new SecretManagerServiceClient()
const {PubSub} = require('@google-cloud/pubsub');
const pubSubClient = new PubSub();
const topicName = "xxxxxx";
//[/Requirement]

exports.LaunchAudit = async () => {
    const dbSocketPath = "/cloudsql"
    const DB_USER = "xxx"
    const DB_PASS = await getSecret()
    const DB_NAME = "xxx"
    const CLOUD_SQL_CONNECTION_NAME = "xxx"
  
    //[SQL CONNEXION]
    let pool = mysql.createPool({
      connectionLimit: 1,
      socketPath: `${dbSocketPath}/${CLOUD_SQL_CONNECTION_NAME}`,
      user: DB_USER,
      password: DB_PASS,
      connectTimeout: 500,
      database: DB_NAME
    })
    //[/SQL CONNEXION]
    //set the request
    let sql = `select * from * where *;`
      //make the setted request 
    await pool.query(sql, async (e,results) => {
      //if there is an error send it
        if(e){
          throw e
        }
        //for each result of the query, log it and publish on PubSub ("Audit-property" topic)
        results.forEach(async element => {
          console.log(JSON.stringify(element))
          await msgPubSub(JSON.stringify(element))
        })
    })    
}

async function msgPubSub(data){
  //console.log(data)
  const messageBuffer = Buffer.from(data)
  try {
    const topicPublisher = await pubSubClient.topic(topicName).publish(messageBuffer)
    console.log("Message id: " + topicPublisher)
  } catch (error) {
    console.error(`Error while publishing message: ${error.message}`)
  }
}

首先，当它运行时，在 PubSub 主题上发布第一条消息需要很长时间，大约 6 分钟，为什么会有这种延迟？当我提出一个大请求（比如 500+ 结果）时，我遇到了超时错误：Total timeout of API google.pubsub.v1.Publisher exceeded 600000 milliseconds before any response was received.

我试过批量发布消息，给云函数增加一些内存，使用google-gax，但得到了相同的结果。

我正在使用 nodejs10。

第二个云函数消息部分代码：

exports.MainAudit =  async message => {
    const property = Buffer.from(message.data, 'base64').toString()
    const pProperty = JSON.parse(property)
    console.log(property)
}

package.json 依赖关系：

  "dependencies": {
    "@google-cloud/pubsub": "^2.6.0",
    "@google-cloud/secret-manager": "^3.2.0",
    "google-gax": "^2.9.2",
    "mysql": "^2.18.1",
    "node-fetch": "^2.6.1"
  }

日志+时间戳：

Answer 1

就像现在的代码一样，您正在为您发布的每条消息创建一个发布者的新实例。这是因为 pubSubClient.topic(topicName) 创建了一个用于发布到主题的实例。因此，您要为发送的每条消息支付建立连接的开销。相反，您希望一次性创建该对象并重新使用它：

const pubSubClient = new PubSub();
const topicName = "xxxxxx";
const topicPublisher = pubSubClient.topic(topicName)

但是，由于在 publish 调用中使用了 await 并且调用至 msgPubSub。 Pub/Sub 客户端库可以 batch messages togetherr for more efficient sending, but you'd need to allow multiple calls to publish to be outstanding to take advantage of it. You'd want to await on a Promise.all 从发布返回的承诺列表。

Google Cloud Functions 不在 PubSub 上发布，超过超时

Google Cloud Function don't publish on PubSub, Timeout exceeded

node.js

google-cloud-platform

google-cloud-pubsub