Node.js 上传图片 Stream.Readable 到 S3

Question

我的 lambda 是由来自浏览器的请求触发的。浏览器将图像发送为 multipart/form-data.

lambda 使用busboy 解析请求：

function parseForm(event: IHttpEvent) {
  return new Promise(
    (resolve, reject) => {
      const busboy = new Busboy({
        headers: event.headers,
        limits: { files: 10 },
      });
      const imageResponse = new Map<string, IImageParseResponse>();

      busboy.on("file", (id, file, filename, encoding, mimeType) => {
           imageResponse.set(id, { file, filename, mimeType });
      });

      busboy.on("error", (error) => reject(`Parse error: ${error}`));
      busboy.on("finish", () => resolve(imageResponse));

      busboy.write(event.body, event.isBase64Encoded ? "base64" : "binary");
      busboy.end();
    }
  );
}

当我解析请求时，我想将文件上传到 AWS S3。

export async function handler(event: IHttpEvent) {
  var res = await parseForm(event);
  const s3 = new S3Client({ region: "eu-central-1" });
  for (const [k, v] of res) {
    console.log(`File ${v.filename} ${v.mimeType} streaming`);
    const stream = new Readable().wrap(v.file);
    const upload = new Upload({
      client: s3,
      params: {
        Key: v.filename,
        Bucket: "my-image-bucket",
        Body: stream,
        ContentType: v.mimeType,
      },
    });
    upload.on("httpUploadProgress", (p) => console.log(p));
    const result = await upload.done();
    console.log(result);
    return result;
  }
}

这不起作用。但是，浏览器将收到 200 OK 和 null 正文响应。更让我困惑的是 console.log(result); 没有记录任何内容到控制台。

我的错误在哪里？我不完全了解流的机制。但据我了解，它会更节省内存。将来我计划一次上传多张图片。为了节省成本，我希望我的方法尽可能高效。

Answer 1

总的来说我犯了2个错误。

试图在 busboy
我没有正确等待上传到 s3 的完成才终止函数。

最后我得到了以下结果：

const s3 = new S3Client({ region: "eu-central-1" });
const { BUCKET_NAME, MAX_IMAGE_SIZE } = process.env;

export async function handler(event: IHttpEvent) {
  const results = await parseForm(event);
  const response = [];
  for (const r of results) {
    if (r.status === "fulfilled") {
      const value: any = r.value.result;
      response.push({
        id: r.value.id,
        key: value.Key,
        url: value.Location,
      });
    }
    if (r.status === "rejected")
      response.push({ id: r.reason.id, reason: r.reason.error });
  }
  return response;
}

async function doneHandler(
  id: string,
  uploadMap: Map<string, Upload>
): Promise<{ id: string; result: ServiceOutputTypes }> {
  try {
    var result = await uploadMap.get(id).done();
  } catch (e: any) {
    var error = e;
  } finally {
    uploadMap.delete(id);
    if (error) throw { id, error };
    return { id, result };
  }
}

function parseForm(event: IHttpEvent) {
  return new Promise( (resolve, reject) => {
      const busboy = new Busboy({
        headers: event.headers,
        limits: { files: 1, fileSize: parseInt(MAX_IMAGE_SIZE) },
      });

      const responses: Promise<{
        id: string;
        result: ServiceOutputTypes;
      }>[] = [];
      const uploads = new Map<string, Upload>();

      busboy.on("file", (id, file, filename, encoding, mimeType) => {
        uploads.set(
          id,
          new Upload({
            client: s3,
            params: {
              Bucket: BUCKET_NAME,
              Body: new Readable().wrap(file),
              Key: filename,
              ContentType: mimeType,
              ContentEncoding: encoding,
            },
          })
        );
        responses.push(doneHandler(id, uploads));

        file.on("limit", async () => {
          const aborts = [];
          for (const [k, upload] of uploads) {
            aborts.push(upload.abort());
          }
          await Promise.all(aborts);
          return reject(new Error("File is too big."));
        });
      });

      busboy.on("error", (error: any) => {
        reject(new Error(`Parse error: ${error}`));
      });
      busboy.on("finish", async () => {
        const res = await Promise.allSettled(responses);
        resolve(res);
      });

      busboy.write(event.body, event.isBase64Encoded ? "base64" : "binary");
      busboy.end();
    }
  );
}

此解决方案还处理文件限制并尝试中止所有待处理的 S3 上传

Node.js 上传图片 Stream.Readable 到 S3

Node.js upload Image Stream.Readable to S3

amazon-s3

node.js

busboy

aws-lambda

node-streams