AWS S3 Node.js SDK `NotImplemented` 错误与 Multer

Question

我正在尝试设置端点以将文件上传到 AWS S3 存储桶。我正在使用 Node.JS、Express、AWS S3 SDK 和 Multer 库。

这是我目前用于上传中间件的代码：

export const uploadMiddleware = multer({
  storage: {
    _handleFile(
      req: Request,
      file: Express.Multer.File,
      callback: (error?: any, info?: Partial<Express.Multer.File>) => void
    ) {
      const key = crypto.randomBytes(16).toString('hex');
      s3.send(
        new PutObjectCommand({
          Bucket: 'my-bucket',
          Body: file.stream,
          Key: key,
          ContentLength: file.size,
          ContentType: file.mimetype,
        }),
        (err, output) => {
          if (err) {
            console.log(err);
            callback(err, undefined);
          } else {
            console.log(output);
            callback(null, file);
          }
        }
      );
    },
    _removeFile(
      req: Request,
      file: Express.Multer.File,
      callback: (error: Error) => void
    ) {
      // @ts-ignore
      callback(null);
    },
  },
}).array('files');

但是，S3 SDK 出现此错误：

NotImplemented: A header you provided implies functionality that is not implemented
2021-07-11T00:18:48.557532998Z     at deserializeAws_restXmlPutObjectCommandError (/usr/src/app/node_modules/@aws-sdk/client-s3/protocols/Aws_restXml.ts:9515:39)
2021-07-11T00:18:48.557542041Z     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2021-07-11T00:18:48.557545571Z     at /usr/src/app/node_modules/@aws-sdk/middleware-serde/src/deserializerMiddleware.ts:18:20
2021-07-11T00:18:48.557548319Z     at /usr/src/app/node_modules/@aws-sdk/middleware-signing/src/middleware.ts:26:22
2021-07-11T00:18:48.557551272Z     at StandardRetryStrategy.retry (/usr/src/app/node_modules/@aws-sdk/middleware-retry/src/StandardRetryStrategy.ts:83:38)
2021-07-11T00:18:48.557554272Z     at /usr/src/app/node_modules/@aws-sdk/middleware-logger/src/loggerMiddleware.ts:22:22 {
2021-07-11T00:18:48.557557035Z   Code: 'NotImplemented',
2021-07-11T00:18:48.557559463Z   Header: 'Transfer-Encoding',
2021-07-11T00:18:48.557561825Z   RequestId: 'T6QNENNWC0X2WDXP',
2021-07-11T00:18:48.557564202Z   HostId: 'gJo7gOiFu9RjdvishH0t6EP8/BHDlBW913gicSlYh1eyJ/JkZaX9QSepRmdeq5Gxt4lg6aKMxxI=',
2021-07-11T00:18:48.557566724Z   '$fault': 'client',
2021-07-11T00:18:48.557569116Z   '$metadata': {
2021-07-11T00:18:48.557571528Z     httpStatusCode: 501,
2021-07-11T00:18:48.557573948Z     requestId: undefined,
2021-07-11T00:18:48.557576323Z     extendedRequestId: 'gJo7gOiFu9RjdvishH0t6EP8/BHDlBW913gicSlYh1eyJ/JkZaX9QSepRmdeq5Gxt4lg6aKMxxI=',
2021-07-11T00:18:48.557578821Z     cfId: undefined,
2021-07-11T00:18:48.557589903Z     attempts: 1,
2021-07-11T00:18:48.557592818Z     totalRetryDelay: 0
2021-07-11T00:18:48.557595524Z   },
2021-07-11T00:18:48.557597888Z   storageErrors: []
2021-07-11T00:18:48.557600254Z }

我尝试了多种可能的解决方案，例如including/excluding ContentLength 和ContentType，并使用file.buffer 而不是file.stream。我还尝试了相关 Whosebug 帖子中的其他解决方案，但没有任何效果。

请注意，我也知道存在用于 Multer/S3 互操作性的 NPM 包，但是 none 使用新的模块化 AWS 系统，因此我选择使用我自己的。

任何帮助将不胜感激，谢谢！

Answer 1

我必须在 handleFile 函数中将流转换为缓冲区：

async function streamToBuffer(stream: Stream): Promise<Buffer> {
  return new Promise<Buffer>((resolve, reject) => {
    const _buf: any[] = [];

    stream.on('data', (chunk) => _buf.push(chunk));
    stream.on('end', () => resolve(Buffer.concat(_buf)));
    stream.on('error', (err) => reject(err));
  });
}

export function handleFile(
  req: Request,
  file: Express.Multer.File,
  callback: (error?: any, info?: Express.Multer.File) => void
) {
  const key = generateId();

  streamToBuffer(file.stream)
    .then((buffer) =>
      s3.send(
        new PutObjectCommand({
          Bucket: config.aws.issueNewBucket,
          Body: buffer,
          Key: key,
          ACL: 'public-read',
          ContentType: file.mimetype,
        })
      )
    )
    .then(() => callback(null, file))
    .catch((err) => callback(err));
}

Answer 2

PutObject S3 API 需要已知的流长度。因此，如果缺少 'content-length' header，则 API 调用将失败。

替代解决方案是使用 @aws-sdk/lib-storage 库：

import { Stream } from 'stream';
import { S3 } from '@aws-sdk/client-s3';
import { Upload } from "@aws-sdk/lib-storage";
// skipped code

    const passThrough = new Stream.PassThrough();

    file.stream.pipe(passThrough);

    const upload = new Upload({
        client: new S3({}),
        params: {
            Bucket: 'my-bucket',
            Key: key,
            Body: passThrough,
            ContentType: file.mimetype,
        }
    });

    upload.on('s3 upload progress', (p) => console.log(p));

    await upload.done();

AWS S3 Node.js SDK `NotImplemented` 错误与 Multer

AWS S3 Node.js SDK `NotImplemented` Error with Multer

amazon-s3

amazon-web-services

node.js

express

multer