通过我的服务器将多个文件上传到 S3 时如何从失败中恢复？

Question

NodeJS 和 S3 的新手，我编写了以下探索性代码，通过我的 NodeJS 服务器将文件上传到 S3，而不将文件保存到磁盘或内存：

var express = require('express');
var Busboy = require('busboy');
var S3 = require('../utils/s3Util');
var router = express.Router(); // mounted at /uploads

router.post("/", function (req, res, next) {
    let bb = new Busboy({ headers: req.headers });
    const uploads = [];
    bb.on('file', (fieldname, stream, filename, encoding, mimeType) => {
        console.log(`Uploaded fieldname: ${fieldname}; filename: ${filename}, mimeType: ${mimeType}`);
        uploads.push(S3.svc.upload({ Bucket: 'my-test-bucket', Key: filename, Body: stream }).promise());
    });
    bb.on('finish', () => {
        console.log("# of promises:", uploads.length);
        Promise.all(uploads).then(retVals => {
            for (let i = 0; retVals && i < retVals.length; i++) {
                console.log(`File ${i + 1}::`, retVals[i]); 
            }
            res.end();
        }).catch(err => {
            console.log("Error::", err);
            res.status(500).send(`${err.name}: ${err.message}`);
        });
    });
    req.pipe(bb);
});

module.exports = router;

一般失败情况下，正在上传的x个文件中有1个或多个上传失败，如何处理？有些上传会成功，有些会失败。但是，在 catch 子句中，我不知道哪些失败了...

最好能够使此上传过程有点 事务性（即，要么所有上传成功，要么 none 成功）。当错误发生时，理想情况下我能够 "rollback" 成功上传的子集。

Answer 1

你可以这样做：

将一个对象推送到上传中，其中包含您需要重试的数据，因此：

uploads.push({ 
  fieldname, 
  filename, 
  mimeType,
  uploaded: S3.svc.upload({ Bucket: 'my-test-bucket', Key: filename, Body: stream })
    .promise()
    .then(() => true)
    .catch(() => false)
});

...

const failed = await 
  (Promise.all(uploads.map(async upload => ({...upload, uploaded: await upload.uploaded})))).then(u => u.filter(upload => !upload.uploaded))

const failedFiles = failed.join(', ')

console.log(`The following files failed to upload: ${failedFiles}`);

您需要让您的事件处理程序 async 在其中使用 await，例如：

bb.on('file', async (fieldname, stream, filename, encoding, mimeType) => {

Answer 2

我最终使用了以下代码，这是对@JoshWulf 的回答的扩展：

function handleUpload(req, res, bucket, key) {
    let bb = new Busboy({ headers: req.headers });
    const uploads = [];
    bb.on('file', (fieldname, stream, filename, encoding, mimeType) => {
        console.log(`Uploaded fieldname: ${fieldname}; filename: ${filename}, mimeType: ${mimeType}`);
        const params = { Bucket: bucket, Key: key, Body: stream, ContentType: mimeType };
        uploads.push({ filename, result: S3.svc.upload(params).promise().then(data => data).catch(err => err) });
    });
    bb.on('finish', async () => {
        const results = await Promise.all(uploads.map(async (upload) => ({ ...upload, result: await upload.result })));
        // handle success/failure with their respective objects
    });
    req.pipe(bb);
}

此处与@Josh Wulf's 回答的不同之处在于，在我的上传承诺中，我按原样返回返回的数据对象（如果成功）和返回的错误对象（如果失败）。这让我以后可以根据需要使用它们。

通过我的服务器将多个文件上传到 S3 时如何从失败中恢复？

How do I recover from failure when uploading multiple files via my server to S3?

javascript

amazon-s3

node.js

busboy