是否可以从浏览器在 amazon s3 上上传流？
Is it possible to upload stream on amazon s3 from browser?
我想捕获网络摄像头视频流，并直接将其流式传输到 S3 存储。
我了解到您可以通过流上传到 s3：
https://aws.amazon.com/blogs/aws/amazon-s3-multipart-upload/
我了解到您可以通过浏览器上传：
http://docs.aws.amazon.com/AmazonS3/latest/dev/HTTPPOSTExamples.html#HTTPPOSTExamplesFileUpload
但我仍然不知道如何实际去做。
我需要一个像上面那样将 getusermediastream 上传到 S3 的例子。
Buffer、Binary data、multipart upload、stream……这些我都不懂。我希望我知道的东西，但现在还不知道去哪里学。
目前，您不能简单地将媒体流传递给任何 S3 方法来自动进行分段上传。
但是，仍然有一个名为 dataavailable 的事件，它会在每个给定的时间间隔内生成视频块。所以我们可以订阅 dataavailable 并手动执行 S3 分段上传。
这种方法带来了一些复杂性：假设视频块每 1 秒生成一次，但我们不知道将视频块上传到 S3 需要多长时间。例如。由于连接速度的原因，上传可能需要 3 倍的时间。所以我们可能会在尝试同时发出多个 PUT 请求时遇到困难。
可能的解决方案是一个一个地上传块，直到上一个块才开始上传下一个块。一个是上传的。
以下是如何使用 Rx.js 和 AWS SDK 处理此问题的片段。请看我的评论。
// Configure the AWS. In this case for the simplicity I'm using access key and secret.
AWS.config.update({
credentials: {
accessKeyId: "YOUR_ACCESS_KEY",
secretAccessKey: "YOUR_SECRET_KEY",
region: "us-east-1"
}
});
const s3 = new AWS.S3();
const BUCKET_NAME = "video-uploads-123";
let videoStream;
// We want to see what camera is recording so attach the stream to video element.
navigator.mediaDevices
.getUserMedia({
audio: true,
video: { width: 1280, height: 720 }
})
.then(stream => {
console.log("Successfully received user media.");
const $mirrorVideo = document.querySelector("video#mirror");
$mirrorVideo.srcObject = stream;
// Saving the stream to create the MediaRecorder later.
videoStream = stream;
})
.catch(error => console.error("navigator.getUserMedia error: ", error));
let mediaRecorder;
const $startButton = document.querySelector("button#start");
$startButton.onclick = () => {
// Getting the MediaRecorder instance.
// I took the snippet from here: https://github.com/webrtc/samples/blob/gh-pages/src/content/getusermedia/record/js/main.js
let options = { mimeType: "video/webm;codecs=vp9" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.log(options.mimeType + " is not Supported");
options = { mimeType: "video/webm;codecs=vp8" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.log(options.mimeType + " is not Supported");
options = { mimeType: "video/webm" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.log(options.mimeType + " is not Supported");
options = { mimeType: "" };
}
}
}
try {
mediaRecorder = new MediaRecorder(videoStream, options);
} catch (e) {
console.error("Exception while creating MediaRecorder: " + e);
return;
}
//Generate the file name to upload. For the simplicity we're going to use the current date.
const s3Key = `video-file-${new Date().toISOString()}.webm`;
const params = {
Bucket: BUCKET_NAME,
Key: s3Key
};
let uploadId;
// We are going to handle everything as a chain of Observable operators.
Rx.Observable
// First create the multipart upload and wait until it's created.
.fromPromise(s3.createMultipartUpload(params).promise())
.switchMap(data => {
// Save the uploadId as we'll need it to complete the multipart upload.
uploadId = data.UploadId;
mediaRecorder.start(15000);
// Then track all 'dataavailable' events. Each event brings a blob (binary data) with a part of video.
return Rx.Observable.fromEvent(mediaRecorder, "dataavailable");
})
// Track the dataavailable event until the 'stop' event is fired.
// MediaRecorder emits the "stop" when it was stopped AND have emitted all "dataavailable" events.
// So we are not losing data. See the docs here: https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/stop
.takeUntil(Rx.Observable.fromEvent(mediaRecorder, "stop"))
.map((event, index) => {
// Show how much binary data we have recorded.
const $bytesRecorded = document.querySelector("span#bytesRecorded");
$bytesRecorded.textContent =
parseInt($bytesRecorded.textContent) + event.data.size; // Use frameworks in prod. This is just an example.
// Take the blob and it's number and pass down.
return { blob: event.data, partNumber: index + 1 };
})
// This operator means the following: when you receive a blob - start uploading it.
// Don't accept any other uploads until you finish uploading: http://reactivex.io/rxjs/class/es6/Observable.js~Observable.html#instance-method-concatMap
.concatMap(({ blob, partNumber }) => {
return (
s3
.uploadPart({
Body: blob,
Bucket: BUCKET_NAME,
Key: s3Key,
PartNumber: partNumber,
UploadId: uploadId,
ContentLength: blob.size
})
.promise()
// Save the ETag as we'll need it to complete the multipart upload
.then(({ ETag }) => {
// How how much bytes we have uploaded.
const $bytesUploaded = document.querySelector("span#bytesUploaded");
$bytesUploaded.textContent =
parseInt($bytesUploaded.textContent) + blob.size;
return { ETag, PartNumber: partNumber };
})
);
})
// Wait until all uploads are completed, then convert the results into an array.
.toArray()
// Call the complete multipart upload and pass the part numbers and ETags to it.
.switchMap(parts => {
return s3
.completeMultipartUpload({
Bucket: BUCKET_NAME,
Key: s3Key,
UploadId: uploadId,
MultipartUpload: {
Parts: parts
}
})
.promise();
})
.subscribe(
({ Location }) => {
// completeMultipartUpload returns the location, so show it.
const $location = document.querySelector("span#location");
$location.textContent = Location;
console.log("Uploaded successfully.");
},
err => {
console.error(err);
if (uploadId) {
// Aborting the Multipart Upload in case of any failure.
// Not to get charged because of keeping it pending.
s3
.abortMultipartUpload({
Bucket: BUCKET_NAME,
UploadId: uploadId,
Key: s3Key
})
.promise()
.then(() => console.log("Multipart upload aborted"))
.catch(e => console.error(e));
}
}
);
};
const $stopButton = document.querySelector("button#stop");
$stopButton.onclick = () => {
// After we call .stop() MediaRecorder is going to emit all the data it has via 'dataavailable'.
// And then finish our stream by emitting 'stop' event.
mediaRecorder.stop();
};
button {
margin: 0 3px 10px 0;
padding-left: 2px;
padding-right: 2px;
width: 99px;
}
button:last-of-type {
margin: 0;
}
p.borderBelow {
margin: 0 0 20px 0;
padding: 0 0 20px 0;
}
video {
height: 232px;
margin: 0 12px 20px 0;
vertical-align: top;
width: calc(20em - 10px);
}
video:last-of-type {
margin: 0 0 20px 0;
}
<div id="container">
<video id="mirror" autoplay muted></video>
<div>
<button id="start">Start Streaming</button>
<button id="stop">Stop Streaming</button>
</div>
<div>
<span>Recorded: <span id="bytesRecorded">0</span> bytes</span>;
<span>Uploaded: <span id="bytesUploaded">0</span> bytes</span>
</div>
<div>
<span id="location"></span>
</div>
</div>
<!-- include adapter for srcObject shim -->
<script src="https://webrtc.github.io/adapter/adapter-latest.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/aws-sdk/2.175.0/aws-sdk.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.6/Rx.js"></script>
注意事项：
- 所有分段上传都需要完成或中止。如果您将其永远悬而未决，您将被收取费用。见 "Note" here.
- 您上传的每个块（最后一个除外）必须大于 5 MB。否则将抛出错误。查看详情 here。所以你需要调整 timeframe/resolution.
- 当您实例化 SDK 时，请确保存在具有
s3:PutObject 权限的策略。
- 您需要在存储桶 CORS 配置中公开 ETag。这是 CORS 配置的示例：
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<ExposeHeader>ETag</ExposeHeader>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
限制：
- 小心，因为 MediaRecorder API 仍未被广泛采用。在产品中使用它之前，请务必检查您是否访问了 caniuse.com。
我想捕获网络摄像头视频流，并直接将其流式传输到 S3 存储。
我了解到您可以通过流上传到 s3： https://aws.amazon.com/blogs/aws/amazon-s3-multipart-upload/
我了解到您可以通过浏览器上传： http://docs.aws.amazon.com/AmazonS3/latest/dev/HTTPPOSTExamples.html#HTTPPOSTExamplesFileUpload
但我仍然不知道如何实际去做。
我需要一个像上面那样将 getusermediastream 上传到 S3 的例子。
Buffer、Binary data、multipart upload、stream……这些我都不懂。我希望我知道的东西，但现在还不知道去哪里学。
目前，您不能简单地将媒体流传递给任何 S3 方法来自动进行分段上传。
但是，仍然有一个名为 dataavailable 的事件，它会在每个给定的时间间隔内生成视频块。所以我们可以订阅 dataavailable 并手动执行 S3 分段上传。
这种方法带来了一些复杂性：假设视频块每 1 秒生成一次，但我们不知道将视频块上传到 S3 需要多长时间。例如。由于连接速度的原因，上传可能需要 3 倍的时间。所以我们可能会在尝试同时发出多个 PUT 请求时遇到困难。
可能的解决方案是一个一个地上传块，直到上一个块才开始上传下一个块。一个是上传的。 以下是如何使用 Rx.js 和 AWS SDK 处理此问题的片段。请看我的评论。
// Configure the AWS. In this case for the simplicity I'm using access key and secret.
AWS.config.update({
credentials: {
accessKeyId: "YOUR_ACCESS_KEY",
secretAccessKey: "YOUR_SECRET_KEY",
region: "us-east-1"
}
});
const s3 = new AWS.S3();
const BUCKET_NAME = "video-uploads-123";
let videoStream;
// We want to see what camera is recording so attach the stream to video element.
navigator.mediaDevices
.getUserMedia({
audio: true,
video: { width: 1280, height: 720 }
})
.then(stream => {
console.log("Successfully received user media.");
const $mirrorVideo = document.querySelector("video#mirror");
$mirrorVideo.srcObject = stream;
// Saving the stream to create the MediaRecorder later.
videoStream = stream;
})
.catch(error => console.error("navigator.getUserMedia error: ", error));
let mediaRecorder;
const $startButton = document.querySelector("button#start");
$startButton.onclick = () => {
// Getting the MediaRecorder instance.
// I took the snippet from here: https://github.com/webrtc/samples/blob/gh-pages/src/content/getusermedia/record/js/main.js
let options = { mimeType: "video/webm;codecs=vp9" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.log(options.mimeType + " is not Supported");
options = { mimeType: "video/webm;codecs=vp8" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.log(options.mimeType + " is not Supported");
options = { mimeType: "video/webm" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.log(options.mimeType + " is not Supported");
options = { mimeType: "" };
}
}
}
try {
mediaRecorder = new MediaRecorder(videoStream, options);
} catch (e) {
console.error("Exception while creating MediaRecorder: " + e);
return;
}
//Generate the file name to upload. For the simplicity we're going to use the current date.
const s3Key = `video-file-${new Date().toISOString()}.webm`;
const params = {
Bucket: BUCKET_NAME,
Key: s3Key
};
let uploadId;
// We are going to handle everything as a chain of Observable operators.
Rx.Observable
// First create the multipart upload and wait until it's created.
.fromPromise(s3.createMultipartUpload(params).promise())
.switchMap(data => {
// Save the uploadId as we'll need it to complete the multipart upload.
uploadId = data.UploadId;
mediaRecorder.start(15000);
// Then track all 'dataavailable' events. Each event brings a blob (binary data) with a part of video.
return Rx.Observable.fromEvent(mediaRecorder, "dataavailable");
})
// Track the dataavailable event until the 'stop' event is fired.
// MediaRecorder emits the "stop" when it was stopped AND have emitted all "dataavailable" events.
// So we are not losing data. See the docs here: https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/stop
.takeUntil(Rx.Observable.fromEvent(mediaRecorder, "stop"))
.map((event, index) => {
// Show how much binary data we have recorded.
const $bytesRecorded = document.querySelector("span#bytesRecorded");
$bytesRecorded.textContent =
parseInt($bytesRecorded.textContent) + event.data.size; // Use frameworks in prod. This is just an example.
// Take the blob and it's number and pass down.
return { blob: event.data, partNumber: index + 1 };
})
// This operator means the following: when you receive a blob - start uploading it.
// Don't accept any other uploads until you finish uploading: http://reactivex.io/rxjs/class/es6/Observable.js~Observable.html#instance-method-concatMap
.concatMap(({ blob, partNumber }) => {
return (
s3
.uploadPart({
Body: blob,
Bucket: BUCKET_NAME,
Key: s3Key,
PartNumber: partNumber,
UploadId: uploadId,
ContentLength: blob.size
})
.promise()
// Save the ETag as we'll need it to complete the multipart upload
.then(({ ETag }) => {
// How how much bytes we have uploaded.
const $bytesUploaded = document.querySelector("span#bytesUploaded");
$bytesUploaded.textContent =
parseInt($bytesUploaded.textContent) + blob.size;
return { ETag, PartNumber: partNumber };
})
);
})
// Wait until all uploads are completed, then convert the results into an array.
.toArray()
// Call the complete multipart upload and pass the part numbers and ETags to it.
.switchMap(parts => {
return s3
.completeMultipartUpload({
Bucket: BUCKET_NAME,
Key: s3Key,
UploadId: uploadId,
MultipartUpload: {
Parts: parts
}
})
.promise();
})
.subscribe(
({ Location }) => {
// completeMultipartUpload returns the location, so show it.
const $location = document.querySelector("span#location");
$location.textContent = Location;
console.log("Uploaded successfully.");
},
err => {
console.error(err);
if (uploadId) {
// Aborting the Multipart Upload in case of any failure.
// Not to get charged because of keeping it pending.
s3
.abortMultipartUpload({
Bucket: BUCKET_NAME,
UploadId: uploadId,
Key: s3Key
})
.promise()
.then(() => console.log("Multipart upload aborted"))
.catch(e => console.error(e));
}
}
);
};
const $stopButton = document.querySelector("button#stop");
$stopButton.onclick = () => {
// After we call .stop() MediaRecorder is going to emit all the data it has via 'dataavailable'.
// And then finish our stream by emitting 'stop' event.
mediaRecorder.stop();
};
button {
margin: 0 3px 10px 0;
padding-left: 2px;
padding-right: 2px;
width: 99px;
}
button:last-of-type {
margin: 0;
}
p.borderBelow {
margin: 0 0 20px 0;
padding: 0 0 20px 0;
}
video {
height: 232px;
margin: 0 12px 20px 0;
vertical-align: top;
width: calc(20em - 10px);
}
video:last-of-type {
margin: 0 0 20px 0;
}
<div id="container">
<video id="mirror" autoplay muted></video>
<div>
<button id="start">Start Streaming</button>
<button id="stop">Stop Streaming</button>
</div>
<div>
<span>Recorded: <span id="bytesRecorded">0</span> bytes</span>;
<span>Uploaded: <span id="bytesUploaded">0</span> bytes</span>
</div>
<div>
<span id="location"></span>
</div>
</div>
<!-- include adapter for srcObject shim -->
<script src="https://webrtc.github.io/adapter/adapter-latest.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/aws-sdk/2.175.0/aws-sdk.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.6/Rx.js"></script>
注意事项：
- 所有分段上传都需要完成或中止。如果您将其永远悬而未决，您将被收取费用。见 "Note" here.
- 您上传的每个块（最后一个除外）必须大于 5 MB。否则将抛出错误。查看详情 here。所以你需要调整 timeframe/resolution.
- 当您实例化 SDK 时，请确保存在具有
s3:PutObject权限的策略。 - 您需要在存储桶 CORS 配置中公开 ETag。这是 CORS 配置的示例：
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<ExposeHeader>ETag</ExposeHeader>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
限制：
- 小心，因为 MediaRecorder API 仍未被广泛采用。在产品中使用它之前，请务必检查您是否访问了 caniuse.com。