配置 Uppy 以使用 Laravel/Vue 的分段上传

Question

我想通了

这是缺失的部分。一旦我清理了我的代码，我会 post 一个答案，希望下一个必须处理这个问题的可怜人不必经历我经历过的同样的地狱;)

$command = $client->getCommand('UploadPart', array(
    'Bucket' => 'the-bucket-name',
    'Key' => $key,
    'PartNumber' => $partNumber,
    'UploadId' => $uploadId,
    'Body' => '',
));

$signedUrl = $client->createPresignedRequest($command, '+20 minutes');
$presignedUrl = (string)$signedUrl->getUri();
return response()->json(['url' => $presignedUrl]);

我正在尝试弄清楚如何配置我的服务器以使用 Uppy 使用 CompanionUrl 选项将分段上传上传到 AWS S3。 https://uppy.io/docs/aws-s3-multipart/#createMultipartUpload-file.

这是我想到走这条路的地方 https://github.com/transloadit/uppy/issues/1189#issuecomment-445521442。

我想不通，我觉得其他人也一直没有答案，所以我 post我正在 post考虑到目前为止我在尝试获得 Uppy 时的想法使用 Laravel/Vue.

处理分段上传

对于 Vue 组件，我有这个：

<template>
<div>
    <a id="uppy-trigger" @click="isUppyOpen = !isUppyOpen">Open Uppy</a>

    <dashboard-modal
        :uppy="uppy"
        :open="isUppyOpen"
        :props="{trigger: '#uppy-trigger'}"
    />
</div>
</template>

<script>
import Uppy from '@uppy/core'
import AwsS3Multipart from '@uppy/aws-s3-multipart';
import '@uppy/core/dist/style.css';
import '@uppy/dashboard/dist/style.css';

export default {
    components: {
        'dashboard-modal': DashboardModal,
    },

    data() {
        return {
            isUppyOpen: false,
        }
    },

    computed: {
        // Uppy Instance
        uppy: () => new Uppy({
            logger: Uppy.debugLogger
        }).use(AwsS3Multipart, {
            limit: 4,
            companionUrl: 'https://mysite.local/',
        }),
    },

    beforeDestroy () {
        this.uppy.close();
    },
}
</script>

然后对于路由，我已将其添加到我的 web.php 文件中。

// AWS S3 Multipart Upload Routes
Route::name('s3.multipart.')->prefix('s3/multipart')
    ->group(function () {
        Route::post('/', ['as' => 'createMultipartUpload', 'uses' => 'AwsS3MultipartController@createMultipartUpload']);
        Route::get('{uploadId}', ['as' => 'getUploadedParts', 'uses' => 'AwsS3MultipartController@getUploadedParts']);
        Route::get('{uploadId}/{partNumber}', ['as' => 'signPartUpload', 'uses' => 'AwsS3MultipartController@signPartUpload']);
        Route::post('{uploadId}/complete', ['as' => 'completeMultipartUpload', 'uses' => 'AwsS3MultipartController@completeMultipartUpload']);
        Route::delete('{uploadId}', ['as' => 'abortMultipartUpload', 'uses' => 'AwsS3MultipartController@abortMultipartUpload']);
    });

基本上发生的事情是我将“companionUrl”设置为“https://mysite.local/”，然后 Uppy 在将分段上传文件上传到这些路由时将发送五个请求，即“https://mysite.local/s3/multipart/createMultipartUpload”.

然后我创建了一个控制器来处理请求：

<?php

namespace App\Http\Controllers;

use Aws\S3\S3Client;
use Illuminate\Http\Request;

class AwsS3MultipartController extends Controller
{
    public function createMultipartUpload(Request $request)
    {
        $client = new S3Client([
            'version' => 'latest',
            'region'  => 'us-east-1',
        ]);

        $key = $request->has('filename') ? $request->get('filename') : null;

        $type = $request->has('type') ? $request->get('type') : null;

        if (!is_string($key)) {
            return response()->json(['error' => 's3: filename returned from "getKey" must be a string'], 500);
        }

        if (!is_string($type)) {
            return response()->json(['error' => 's3: content type must be a string'], 400);
        }

        $response = $client->createMultipartUpload([
            'Bucket'        => 'the-bucket-name',
            'Key'           => $key,
            'ContentType'   => $type,
            'Expires'       => 60
        ]);

        $mpuKey = !empty($response['Key']) ? $response['Key'] : null;
        $mpuUploadId = !empty($response['UploadId']) ? $response['UploadId'] : null;

        if (!$mpuKey || !$mpuUploadId) {
            return response()->json(['error' => 'Unable to process upload request.'], 400);
        }

        return response()->json([
            'key'       => $mpuKey,
            'uploadId'  => $mpuUploadId
        ]);
    }

    public function getUploadedParts($uploadId)
    {
        // Haven't configured this route yet as I haven't made it this far.
        return $uploadId;
    }

    public function signPartUpload(Request $request, $uploadId, $partNumber)
    {
        $client = new S3Client([
            'version' => 'latest',
            'region'  => 'us-east-1',
        ]);

        $key = $request->has('key') ? $request->get('key') : null;

        if (!is_string($key)) {
            return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
        }

        if (!intval($partNumber)) {
            return response()->json(['error' => 's3: the part number must be a number between 1 and 10000.'], 400);
        }

        // Creating a presigned URL. I don't think this is correct.
        $cmd = $client->getCommand('PutObject', [
            'Bucket'        => 'the-bucket-name',
            'Key'           => $key,
            'UploadId'      => $uploadId,
            'PartNumber'    => $partNumber,
        ]);

        $response = $client->createPresignedRequest($cmd, '+20 minutes');
        $presignedUrl = (string)$response->getUri();

        return response()->json(['url' => $presignedUrl]);
    }

    public function completeMultipartUpload(Request $request, $uploadId)
    {
        $client = new S3Client([
            'version' => 'latest',
            'region'  => 'us-east-1',
        ]);

        $key = $request->has('key') ? $request->get('key') : null;

        $parts = json_decode($request->getContent(), true)['parts'];

        if (!is_string($key)) {
            return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
        }

        if (!is_array($parts) || !$this->arePartsValid($parts)) {
            return response()->json(['error' => 's3: "parts" must be an array of {ETag, PartNumber} objects.'], 400);
        }


        // The completeMultipartUpload method fails with the following error.

        // "Error executing "CompleteMultipartUpload" on "https://the-bucket-name.s3.amazonaws.com/NewProject.png?uploadId=nlWLdbNgB9zgarpLBXnj17eOIGAmQM_xyBArymtwdM71fhbFvveggDmL6fz4blz.B95TLhMatDvodbMb5p2ZMKqdlLeLFoSW1qcu33aRQTlt6NbiP_dkDO90DFO.pWGH"; AWS HTTP error: Client error: `POST https://the-bucket-name.s3.amazonaws.com/NewProject.png?uploadId=nlWLdbNgB9zgarpLBXnj17eOIGAmQM_xyBArymtwdM71fhbFvveggDmL6fz4blz.B95TLhMatDvodbMb5p2ZMKqdlLeLFoSW1qcu33aRQTlt6NbiP_dkDO90DFO.pWGH` resulted in a `400 Bad Request` response:
        //     <Error><Code>InvalidPart</Code><Message>One or more of the specified parts could not be found.  The part may not have be (truncated...)
        //  InvalidPart (client): One or more of the specified parts could not be found.  The part may not have been uploaded, or the specified entity tag may not match the part's entity tag. - <Error><Code>InvalidPart</Code><Message>One or more of the specified parts could not be found.  The part may not have been uploaded, or the specified entity tag may not match the part's en"

        $result = $client->completeMultipartUpload([
            'Bucket'          => 'the-bucket-name',
            'Key'             => $key,
            'UploadId'        => $uploadId,
            'MultipartUpload' => [
                'Parts' => $parts,
            ],
        ]);

        return response()->json(['location' => $result['location']]);
    }

    public function abortMultipartUpload($uploadId)
    {
        // Haven't configured this route yet as I haven't made it this far.
        return $uploadId;
    }

    private function arePartsValid($parts)
    {
        // Validation for the parts will go here, but returning true for now.
        return true;
    }
}

我完全可以上传一个多部分文件PHP/server-side。但是对于大文件，这不会起作用，因为我必须等待上传在我的服务器上完成，然后将其分段上传到 AWS。

$s3_client = new S3Client([
    'version' => 'latest',
    'region'  => 'us-east-1',
]);
$bucket = 'the-bucket-name';
$tmp_name = $request->file('file')->getPathname();
$folder = Carbon::now()->format('Y/m/d/');
$filename = pathinfo($request->file('file')->getClientOriginalName(), PATHINFO_FILENAME);
$extension = $extension = pathinfo($request->file('file')->getClientOriginalName(), PATHINFO_EXTENSION);
$timestamp = Carbon::now()->format('H-i-s');
$name = "{$folder}{$filename}_{$timestamp}.{$extension}";

$response = $s3_client->createMultipartUpload([
    'Bucket' => $bucket,
    'Key'    => $name,
]);

$uploadId = $response['UploadId'];

$file = fopen($tmp_name, 'r');
$parts = [];
$partNumber = 1;
while (! feof($file)) {
    $result = $s3_client->uploadPart([
        'Bucket'     => $bucket,
        'Key'        => $name,
        'UploadId'   => $uploadId,
        'PartNumber' => $partNumber,
        'Body'       => fread($file, 5 * 1024 * 1024),
    ]);

    $parts[] = [
        'PartNumber' => $partNumber++,
        'ETag'       => $result['ETag'],
    ];
}

$result = $s3_client->completeMultipartUpload([
    'Bucket'          => $bucket,
    'Key'             => $name,
    'UploadId'        => $uploadId,
    'MultipartUpload' => [
        'Parts' => $parts,
    ],
]);

我认为正在发生的事情是 Uppy 正在处理客户端的 while 循环部分。为了做到这一点，我必须 return 预签名 URL Uppy 可以使用，但预签名 URL 我目前 returning 不是正确。

我注意到的一件事是，当我在纯服务器端启动分段上传时逐步执行 while 循环时，没有文件上传到我的存储桶，直到 completeMultipartUpload 方法是解雇了。但是，如果我逐步完成通过 Uppy 上传的部分，这些部分似乎是作为最终文件上传的，每个部分只是覆盖前一部分。然后我留下了文件的一个片段，即 43.5MB 文件的最后 3.5MB。

Answer 1

以下是我如何让 Uppy、Vue 和 Laravel 一起玩得很好。

Vue 组件：

<template>
<div>
    <a id="uppy-trigger" @click="isUppyOpen = !isUppyOpen">Open Uppy</a>

    <dashboard-modal
        :uppy="uppy"
        :open="isUppyOpen"
        :props="{trigger: '#uppy-trigger'}"
    />
</div>
</template>

<script>
import Uppy from '@uppy/core'
import AwsS3Multipart from '@uppy/aws-s3-multipart';
import '@uppy/core/dist/style.css';
import '@uppy/dashboard/dist/style.css';

export default {
    components: {
        'dashboard-modal': DashboardModal,
    },

    data() {
        return {
            isUppyOpen: false,
        }
    },

    computed: {
        // Uppy Instance
        uppy: () => new Uppy({
            logger: Uppy.debugLogger
        }).use(AwsS3Multipart, {
            limit: 4,
            companionUrl: 'https://mysite.local/',
        }),
    },

    beforeDestroy () {
        this.uppy.close();
    },
}
</script>

路由：

// AWS S3 Multipart Upload Routes
Route::name('s3.multipart.')->prefix('s3/multipart')
    ->group(function () {
        Route::post('/', ['as' => 'createMultipartUpload', 'uses' => 'AwsS3MultipartController@createMultipartUpload']);
        Route::get('{uploadId}', ['as' => 'getUploadedParts', 'uses' => 'AwsS3MultipartController@getUploadedParts']);
        Route::get('{uploadId}/{partNumber}', ['as' => 'signPartUpload', 'uses' => 'AwsS3MultipartController@signPartUpload']);
        Route::post('{uploadId}/complete', ['as' => 'completeMultipartUpload', 'uses' => 'AwsS3MultipartController@completeMultipartUpload']);
        Route::delete('{uploadId}', ['as' => 'abortMultipartUpload', 'uses' => 'AwsS3MultipartController@abortMultipartUpload']);
    });

控制者：

<?php

namespace App\Http\Controllers;

use Aws\S3\S3Client;
use Carbon\Carbon;
use Exception;
use Illuminate\Http\Request;

class AwsS3MultipartController extends Controller
{
    private $bucket;
    private $client;

    public function __construct()
    {
        $this->bucket = 'the-name-of-the-bucket';

        $this->client = new S3Client([
            'version' => 'latest',
            'region'  => 'us-east-1',
        ]);
    }

    /**
     * Create/initiate the multipart upload
     * @param Request $request 
     * @return JsonResponse 
     */
    public function createMultipartUpload(Request $request)
    {
        // Get the filename and type from request
        $filename = $request->has('filename') ? $request->get('filename') : null;
        $type = $request->has('type') ? $request->get('type') : null;

        // Check filename
        if (!is_string($filename)) {
            return response()->json(['error' => 's3: filename returned from "getKey" must be a string'], 500);
        }

        // Check type
        if (!is_string($type)) {
            return response()->json(['error' => 's3: content type must be a string'], 400);
        }

        // Set up key equal to YYYY/MM/DD/filename_H-i-s.ext
        $fileBaseName = pathinfo($filename, PATHINFO_FILENAME);
        $extension = pathinfo($filename, PATHINFO_EXTENSION);
        $folder = Carbon::now()->format('Y/m/d/');
        $timestamp = Carbon::now()->format('H-i-s');
        $key = "{$folder}{$fileBaseName}_{$timestamp}.{$extension}";

        // Create/initiate the multipart upload
        try {
            $response = $this->client->createMultipartUpload([
                'Bucket'        => $this->bucket,
                'Key'           => $key,
                'ContentType'   => $type,
                'Expires'       => 60
            ]);
        } catch (Exception $e) {
            return response()->json(['error' => $e->getMessage()], 400);
        }

        // Multipart upload key and id
        $mpuKey = !empty($response['Key']) ? $response['Key'] : null;
        $mpuUploadId = !empty($response['UploadId']) ? $response['UploadId'] : null;

        // Check multipart upload key and id
        if (!$mpuKey || !$mpuUploadId) {
            return response()->json(['error' => 'Unable to process upload request.'], 400);
        }

        return response()->json([
            'key'       => $mpuKey,
            'uploadId'  => $mpuUploadId
        ]);
    }

    /**
     * Get parts that have been uploaded
     * @param Request $request 
     * @param string $uploadId 
     * @return JsonResponse 
     */
    public function getUploadedParts(Request $request, string $uploadId)
    {
        $key = $request->has('key') ? $request->get('key') : null;

        // Check key
        if (!is_string($key)) {
            return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
        }

        $parts = [];
        $getParts = true;
        $startAt = 0;

        // Get parts uploaded so far
        while ($getParts) {
            $partsPage = $this->listPartsPage($key, $uploadId, $startAt, $parts);

            if (isset($partsPage['error'])) {
                return response()->json(['error' => $partsPage['error']], 400);
            }

            if ($partsPage['isTruncated']) {
                $startAt = $partsPage['nextPartNumberMarker'];
            } else {
                $getParts = false;
            }
        }

        return response()->json(
            $parts,
        );
    }

    /**
     * Create a pre-signed URL for parts to be uploaded to
     * @param Request $request 
     * @param string $uploadId 
     * @param int $partNumber 
     * @return JsonResponse 
     */
    public function signPartUpload(Request $request, string $uploadId, int $partNumber)
    {
        $key = $request->has('key') ? $request->get('key') : null;

        // Check key
        if (!is_string($key)) {
            return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
        }

        // Check part number
        if (!intval($partNumber)) {
            return response()->json(['error' => 's3: the part number must be a number between 1 and 10000.'], 400);
        }

        // Create the upload part command and get the pre-signed URL
        try {
            $command = $this->client->getCommand('UploadPart', [
                'Bucket'        => $this->bucket,
                'Key'           => $key,
                'PartNumber'    => $partNumber,
                'UploadId'      => $uploadId,
                'Body'          => '',
            ]);

            $presignedUrl = $this->client->createPresignedRequest($command, '+20 minutes');
        } catch (Exception $e) {
            return response()->json(['error' => $e->getMessage()], 400);
        }

        // Convert the pre-signed URL to a string
        $presignedUrlString = (string)$presignedUrl->getUri();

        return response()->json(['url' => $presignedUrlString]);
    }

    /**
     * Complete the multipart upload
     * @param Request $request 
     * @param string $uploadId 
     * @return JsonResponse 
     */
    public function completeMultipartUpload(Request $request, string $uploadId)
    {
        $key = $request->has('key') ? $request->get('key') : null;

        $parts = json_decode($request->getContent(), true)['parts'];

        // Check the key
        if (!is_string($key)) {
            return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
        }

        // Check the parts
        if (!is_array($parts) || !$this->arePartsValid($parts)) {
            return response()->json(['error' => 's3: "parts" must be an array of {ETag, PartNumber} objects.'], 400);
        }

        // Complete the multipart upload
        try {
            $result = $this->client->completeMultipartUpload([
                'Bucket'          => $this->bucket,
                'Key'             => $key,
                'UploadId'        => $uploadId,
                'MultipartUpload' => [
                    'Parts' => $parts,
                ],
            ]);
        } catch (Exception $e) {
            return response()->json(['error' => $e->getMessage()], 400);
        }

        // Change forwardslash entities to forwardslashes
        $location = urldecode($result['Location']);

        return response()->json(['location' => $location]);
    }

    public function abortMultipartUpload(Request $request, $uploadId)
    {
        $key = $request->has('key') ? $request->get('key') : null;

        // Check the key
        if (!is_string($key)) {
            return response()->json(['error' => 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"'], 400);
        }

        // Cancel the multipart upload
        try {
            $response = $this->client->abortMultipartUpload([
                'Bucket' => $this->bucket,
                'Key' => $key,
                'UploadId' => $uploadId,
            ]);
        } catch (Exception $e) {
            //
        }

        return response()->json();
    }

    private function listPartsPage(string $key, string $uploadId, int $startAt, array &$parts)
    {
        // Configure response
        $response = [
            'isTruncated' => false,
        ];

        // Get list of parts uploaded
        try {
            $result = $this->client->listParts([
                'Bucket'            => $this->bucket,
                'Key'               => $key,
                'PartNumberMarker'  => $startAt,
                'UploadId'          => $uploadId,
            ]);
        } catch (Exception $e) {
            return ['error' => 's3: unable to continue upload. The upload may have been aborted.'];
        }

        // Add found parts to parts array
        if ($result->hasKey('Parts')) {
            array_push($parts, ...$result->get('Parts'));
        }

        // Check if parts are truncated
        if ($result->hasKey('IsTruncated') && $result->get('IsTruncated')) {
            $response['isTruncated'] = true;
            $response['nextPartNumberMarker'] = $result->get('NextPartNumberMarker');
        }

        return $response;
    }

    /**
     * Validate the parts for the multipart upload
     * @param array $parts An associative array of parts with PartNumber and ETag
     * @return bool 
     */
    private function arePartsValid(array $parts)
    {
        if (!is_array($parts)) {
            return false;
        }

        foreach ($parts as $part) {
            if (!is_int($part['PartNumber']) || !is_string($part['ETag'])) {
                return false;
            }
        }

        return true;
    }
}

Answer 2

您可以使用这个预构建的 laravel 包通过 laravel 和 uppy 轻松实现分段上传:

https://github.com/TappNetwork/laravel-uppy-s3-multipart-upload

配置 Uppy 以使用 Laravel/Vue 的分段上传

Configuring Uppy to Use Multipart Uploads with Laravel/Vue

amazon-s3

laravel

vue.js

aws-sdk

uppy