CloudBlockBlob.DownloadToStream 对比 DownloadRangeToStream
CloudBlockBlob.DownloadToStream vs DownloadRangeToStream
正在尝试使用 ASP.NET azure SDK 从 blob 存储下载图像..
我在另一个 post 中读到 DownloadToStream 确实将 blob 分解成更小的块并并行下载它们以提高性能。我相信这就是 DownloadRangeToStream 的用途。
我还没有找到任何文档或代码来证实这个关于 DownloadToStream 的声明,我对此持怀疑态度,因为它与直接从 blob 下载的运行时间相同 url(每次下载 0.5-3 秒).这是我的两种下载方法的代码,性能大致相同。
使用CloudBlockBlob.DownloadToStream:
private Bitmap DownloadFromBlob(String set) {
CloudStorageAccount storageAccount = CloudStorageAccount.Parse( CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("templates");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(set + ".png");
using (var memoryStream = new MemoryStream()) {
blockBlob.DownloadToStream(memoryStream);
return (memoryStream == null) ? null : (Bitmap)Image.FromStream(memoryStream);
}
}
使用Image.FromStream:
private Bitmap DownloadImageFromUrl(string url) {
try {
using (WebClient client = new WebClient()) {
byte[] data = client.DownloadData(url);
using (MemoryStream mem = (data == null) ? null : new MemoryStream(data)) {
return (data == null || mem == null) ? null : (Bitmap)Image.FromStream(mem);
}
}
} catch (WebException e) {
return null;
}
}
我正在尝试增加 0.5-12 MB 图像的下载时间。我尝试为这些图像实现我自己的 DownloadRangeToStream 方法,其代码如下。我需要这样做还是 DownloadToStream 已经为我做了?此方法产生与上述 DownloadFromBlob 方法相同的运行时..
使用 downloadRangeToStream:
private Image getImageFromStream(string set)
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("templates");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(set + ".png");
using (MemoryStream ms = new MemoryStream())
{
ParallelDownloadBlob(ms, blockBlob);
return (ms == null) ? null : Image.FromStream(ms);
}
}
private static void ParallelDownloadBlob(Stream outPutStream, CloudBlockBlob blob)
{
blob.FetchAttributes();
int bufferLength = 1 * 1024 * 1024;//1 MB chunk
long blobRemainingLength = blob.Properties.Length;
Queue<KeyValuePair<long, long>> queues = new Queue<KeyValuePair<long, long>>();
long offset = 0;
while (blobRemainingLength > 0)
{
long chunkLength = (long)Math.Min(bufferLength, blobRemainingLength);
queues.Enqueue(new KeyValuePair<long, long>(offset, chunkLength));
offset += chunkLength;
blobRemainingLength -= chunkLength;
}
Parallel.ForEach(queues,
new ParallelOptions()
{
//Gets or sets the maximum number of concurrent tasks
MaxDegreeOfParallelism = 10
}, (queue) =>
{
using (var ms = new MemoryStream())
{
blob.DownloadRangeToStream(ms, queue.Key, queue.Value);
lock (outPutStream)
{
outPutStream.Position = queue.Key;
var bytes = ms.ToArray();
outPutStream.Write(bytes, 0, bytes.Length);
}
}
});
}
据我了解,CloudBlockBlob.DownloadToStream
和 Image.FromStream
都只会发送下载流的请求,您可以利用 Fiddler 来捕获流量,如下所示:
使用 DownloadRangeToStream
时,您可以将 blob 分解成更小的块并自行并行下载它们以提高性能。这是我的代码片段,你可以参考一下。
private static void ParallelDownloadBlob(Stream outPutStream, CloudBlockBlob blob)
{
blob.FetchAttributes();
int bufferLength = 1 * 1024 * 1024;//1 MB chunk
long blobRemainingLength = blob.Properties.Length;
Queue<KeyValuePair<long, long>> queues = new Queue<KeyValuePair<long, long>>();
long offset = 0;
while (blobRemainingLength > 0)
{
long chunkLength = (long)Math.Min(bufferLength, blobRemainingLength);
queues.Enqueue(new KeyValuePair<long, long>(offset, chunkLength));
offset += chunkLength;
blobRemainingLength -= chunkLength;
}
Parallel.ForEach(queues,
new ParallelOptions()
{
//Gets or sets the maximum number of concurrent tasks
MaxDegreeOfParallelism = 10
}, (queue) =>
{
using (var ms = new MemoryStream())
{
blob.DownloadRangeToStream(ms, queue.Key, queue.Value);
lock (outPutStream)
{
outPutStream.Position = queue.Key;
var bytes = ms.ToArray();
outPutStream.Write(bytes, 0, bytes.Length);
}
}
});
}
结果:
另外,还有一些关于upload/download blob的并行博客,你可以参考一下(blog1 and blog2)。
正在尝试使用 ASP.NET azure SDK 从 blob 存储下载图像..
我在另一个 post 中读到 DownloadToStream 确实将 blob 分解成更小的块并并行下载它们以提高性能。我相信这就是 DownloadRangeToStream 的用途。
我还没有找到任何文档或代码来证实这个关于 DownloadToStream 的声明,我对此持怀疑态度,因为它与直接从 blob 下载的运行时间相同 url(每次下载 0.5-3 秒).这是我的两种下载方法的代码,性能大致相同。
使用CloudBlockBlob.DownloadToStream:
private Bitmap DownloadFromBlob(String set) {
CloudStorageAccount storageAccount = CloudStorageAccount.Parse( CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("templates");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(set + ".png");
using (var memoryStream = new MemoryStream()) {
blockBlob.DownloadToStream(memoryStream);
return (memoryStream == null) ? null : (Bitmap)Image.FromStream(memoryStream);
}
}
使用Image.FromStream:
private Bitmap DownloadImageFromUrl(string url) {
try {
using (WebClient client = new WebClient()) {
byte[] data = client.DownloadData(url);
using (MemoryStream mem = (data == null) ? null : new MemoryStream(data)) {
return (data == null || mem == null) ? null : (Bitmap)Image.FromStream(mem);
}
}
} catch (WebException e) {
return null;
}
}
我正在尝试增加 0.5-12 MB 图像的下载时间。我尝试为这些图像实现我自己的 DownloadRangeToStream 方法,其代码如下。我需要这样做还是 DownloadToStream 已经为我做了?此方法产生与上述 DownloadFromBlob 方法相同的运行时..
使用 downloadRangeToStream:
private Image getImageFromStream(string set)
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("templates");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(set + ".png");
using (MemoryStream ms = new MemoryStream())
{
ParallelDownloadBlob(ms, blockBlob);
return (ms == null) ? null : Image.FromStream(ms);
}
}
private static void ParallelDownloadBlob(Stream outPutStream, CloudBlockBlob blob)
{
blob.FetchAttributes();
int bufferLength = 1 * 1024 * 1024;//1 MB chunk
long blobRemainingLength = blob.Properties.Length;
Queue<KeyValuePair<long, long>> queues = new Queue<KeyValuePair<long, long>>();
long offset = 0;
while (blobRemainingLength > 0)
{
long chunkLength = (long)Math.Min(bufferLength, blobRemainingLength);
queues.Enqueue(new KeyValuePair<long, long>(offset, chunkLength));
offset += chunkLength;
blobRemainingLength -= chunkLength;
}
Parallel.ForEach(queues,
new ParallelOptions()
{
//Gets or sets the maximum number of concurrent tasks
MaxDegreeOfParallelism = 10
}, (queue) =>
{
using (var ms = new MemoryStream())
{
blob.DownloadRangeToStream(ms, queue.Key, queue.Value);
lock (outPutStream)
{
outPutStream.Position = queue.Key;
var bytes = ms.ToArray();
outPutStream.Write(bytes, 0, bytes.Length);
}
}
});
}
据我了解,CloudBlockBlob.DownloadToStream
和 Image.FromStream
都只会发送下载流的请求,您可以利用 Fiddler 来捕获流量,如下所示:
使用 DownloadRangeToStream
时,您可以将 blob 分解成更小的块并自行并行下载它们以提高性能。这是我的代码片段,你可以参考一下。
private static void ParallelDownloadBlob(Stream outPutStream, CloudBlockBlob blob)
{
blob.FetchAttributes();
int bufferLength = 1 * 1024 * 1024;//1 MB chunk
long blobRemainingLength = blob.Properties.Length;
Queue<KeyValuePair<long, long>> queues = new Queue<KeyValuePair<long, long>>();
long offset = 0;
while (blobRemainingLength > 0)
{
long chunkLength = (long)Math.Min(bufferLength, blobRemainingLength);
queues.Enqueue(new KeyValuePair<long, long>(offset, chunkLength));
offset += chunkLength;
blobRemainingLength -= chunkLength;
}
Parallel.ForEach(queues,
new ParallelOptions()
{
//Gets or sets the maximum number of concurrent tasks
MaxDegreeOfParallelism = 10
}, (queue) =>
{
using (var ms = new MemoryStream())
{
blob.DownloadRangeToStream(ms, queue.Key, queue.Value);
lock (outPutStream)
{
outPutStream.Position = queue.Key;
var bytes = ms.ToArray();
outPutStream.Write(bytes, 0, bytes.Length);
}
}
});
}
结果:
另外,还有一些关于upload/download blob的并行博客,你可以参考一下(blog1 and blog2)。