下载许多 zip 中的大文件。正确实施

Question

我正在使用 Angular 12 开发一个应用程序，它允许用户存储大量文件，例如图像和视频。（有些视频可能大于 1GB）。反正都是很重的文件。

在此应用程序中，有必要放置一个按钮“将所有内容下载为 ZIP”问题是我目前必须使用 JSZip 处理此下载的方式，它使我的计算机非常慢，而且它不报告文件被压缩之前的进度，也就是说，它在 0% 下载时花费 20 分钟，甚至更晚它开始报告进度。

这是我目前正在实施的解决方案：

 <button (click)="downloadAll()">Download as Zip</button>

然后在我的 TS 文件中实现函数

downloadAll() {
  // This is the function where I get the access links of all the uploaded files, which can weigh more than 11GB in total.
  // But here I only get an array of links
  const urls = this.PROJECT.resources.map(u => u.link); 
  this.downloadService.downloadAll(urls, this.downloadCallback); // my download service
}


// this is my callback function that I send to the service
downloadCallback(metaData) {
  const percent = metaData.percent;
  setTimeout(() => {
    console.log(percent);
    if (percent >= 100) { console.log('Zip Downloaded'); }
  }, 10);
}

我的 Donwload 服务具有 Donwload All 功能，我将所有重新收集的文件转换为一个 zip。

downloadAll(urls: string[], callback?: any) {
  let count = 0;
  const zip = new JSZip();
  urls.forEach(u => {
    // for each link I undestand that I need to get the Binary Content
    const filename = u.split('/')[u.split('/').length - 1];
    // I think that this function where the binary content of a file is obtained through a link is what causes the download to be so slow and to consume a lot of resources
    JSZipUtils.getBinaryContent(u, (err, data) => {
      if (err) { throw err;  }
      zip.file(filename, data, { binary: true });
      count++;
      if (count === urls.length) {
        // This function works relatively normal and reports progress as expected.
        zip.generateAsync({ type: 'blob' }, (value) => callback(value)).then((content) => {
          const objectUrl: string = URL.createObjectURL(content);
          const link: any = document.createElement('a');
          link.download = 'resources.zip';
          link.href = objectUrl;
          link.click();
        });
      }
    });
  });
}

所以基本上我的问题是：是否有正确的方法来实现在 zip 中下载多个大文件？

任何时候用户都需要知道他们的下载进度，但是正如我重复的那样，在获取每个文件的二进制内容时，这样做需要很长时间，直到很长时间之后生成的文件开始下载。

在 zip 中下载大文件的正确方法是什么？

Answer 1

您可以使用 OneZip 而不是 JSZip。

OneZip 提供了一些侦听器，您可以使用它们来改进 UI/UX。

const onezip = require('onezip');
const path = require('path');
const cwd = process.cwd();
const name = 'pipe.tar.gz';
const from = cwd + '/pipe-io';
const to = path.join(cwd, name);

const pack = onezip.pack(from, to, [
    'LICENSE',
    'README.md',
    'package.json',
]);

pack.on('file', (name) => {
    console.log(name);
});

pack.on('start', () => {
    console.log('start packing');
});

pack.on('progress', (percent) => {        // this will help you with what you are seeking
    console.log(percent + '%');
});

pack.on('error', (error) => {
    console.error(error);
});

pack.on('end', () => {
    console.log('done');
});

关于滞后，我认为这是由您的系统配置而不是进程引起的。这是一个繁重的过程，比其他简单的任务需要更多的时间。

但您可以运行在后台执行任务，并在可以下载时通知用户。它将改善您的用户体验并减少过程中的烦人。

Answer 2

所以这引起了我的注意，并尝试使用可观察对象构建它。

简短回答：无法在浏览器中使用 Jszip 压缩大文件，我觉得它也应该在后端完成。

https://stuk.github.io/jszip/documentation/limitations.html

但如果需要，这里是压缩的正确实现：

我构建的项目包含一个流式传输文件的后端和将获取文件并使用进度条压缩它们的前端。

我试图压缩两个总共 2.4G 的电影文件，但在达到 100% 时失败了错误是：

Uncaught (in promise): RangeError: Array buffer allocation failed

https://github.com/ericaskari-playground/playground-angular/tree/master/download-and-zip-with-progressbar

我用一个 observable 包装了 jszip：

import {
    HttpClient,
    HttpEvent,
    HttpEventType,
    HttpHeaderResponse,
    HttpProgressEvent,
    HttpResponse,
    HttpUserEvent
} from '@angular/common/http';
import {Injectable} from '@angular/core';
import {combineLatest, Observable} from 'rxjs';
import {scan} from 'rxjs/operators';
import JSZip from 'jszip';

export class DownloadModel {
    link: string = '';
    fileSize: number = 0;
    fileName: string = '';
}

export interface DownloadStatus<T> {
    progress: number;
    state: 'PENDING' | 'IN_PROGRESS' | 'DONE' | 'SENT';
    body: T | null;
    httpEvent: HttpEvent<unknown> | null;
    downloadModel: DownloadModel;
}

export interface ZipStatus<T> {
    progress: number;
    state: 'PENDING' | 'DOWNLOADING' | 'DOWNLOADED' | 'ZIPPING' | 'DONE';
    body: {
        downloadModel: DownloadModel,
        downloaded: Blob | null
    }[];
    zipFile: Blob | null;
    httpEvent: HttpEvent<unknown> | null;
}

@Injectable({providedIn: 'root'})
export class DownloadService {
    constructor(private httpClient: HttpClient) {
    }

    public downloadMultiple(downloadLinks: DownloadModel[]): Observable<DownloadStatus<Blob>[]> {
        return combineLatest(downloadLinks.map((downloadModel): Observable<DownloadStatus<Blob>> => {
                    return this.httpClient.get(downloadModel.link, {
                        reportProgress: true,
                        observe: 'events',
                        responseType: 'blob'
                    }).pipe(
                        scan((uploadStatus: DownloadStatus<Blob>, httpEvent: HttpEvent<Blob>, index: number): DownloadStatus<Blob> => {
                            if (this.isHttpResponse(httpEvent)) {

                                return {
                                    progress: 100,
                                    state: 'DONE',
                                    body: httpEvent.body,
                                    httpEvent,
                                    downloadModel
                                };
                            }

                            if (this.isHttpSent(httpEvent)) {

                                return {
                                    progress: 0,
                                    state: 'PENDING',
                                    body: null,
                                    httpEvent,
                                    downloadModel

                                };
                            }

                            if (this.isHttpUserEvent(httpEvent)) {

                                return {
                                    progress: 0,
                                    state: 'PENDING',
                                    body: null,
                                    httpEvent,
                                    downloadModel
                                };
                            }

                            if (this.isHttpHeaderResponse(httpEvent)) {

                                return {
                                    progress: 0,
                                    state: 'PENDING',
                                    body: null,
                                    httpEvent,
                                    downloadModel
                                };
                            }

                            if (this.isHttpProgressEvent(httpEvent)) {

                                return {
                                    progress: Math.round((100 * httpEvent.loaded) / downloadModel.fileSize),
                                    state: 'IN_PROGRESS',
                                    httpEvent,
                                    body: null,
                                    downloadModel
                                };
                            }


                            console.log(httpEvent);

                            throw new Error('unknown HttpEvent');

                        }, {state: 'PENDING', progress: 0, body: null, httpEvent: null} as DownloadStatus<Blob>));
                }
            )
        );

    }

    zipMultiple(downloadMultiple: Observable<DownloadStatus<Blob>[]>): Observable<ZipStatus<Blob>> {

        return new Observable<ZipStatus<Blob>>(((subscriber) => {

            downloadMultiple.pipe(
                scan((uploadStatus: ZipStatus<Blob>, httpEvent: DownloadStatus<Blob>[], index: number): ZipStatus<Blob> => {
                    if (httpEvent.some((x) => x.state === 'SENT')) {
                        return {
                            state: 'PENDING',
                            body: [],
                            httpEvent: null,
                            progress: httpEvent.reduce((prev, curr) => prev + curr.progress, 0) / httpEvent.length / 2,
                            zipFile: null
                        };
                    }
                    if (httpEvent.some((x) => x.state === 'PENDING')) {
                        return {
                            state: 'PENDING',
                            body: [],
                            httpEvent: null,
                            progress: httpEvent.reduce((prev, curr) => prev + curr.progress, 0) / httpEvent.length / 2,
                            zipFile: null
                        };
                    }

                    if (httpEvent.some((x) => x.state === 'IN_PROGRESS')) {
                        return {
                            state: 'DOWNLOADING',
                            body: [],
                            httpEvent: null,
                            progress: httpEvent.reduce((prev, curr) => prev + curr.progress, 0) / httpEvent.length / 2,
                            zipFile: null
                        };
                    }

                    if (httpEvent.every((x) => x.state === 'DONE')) {
                        return {
                            state: 'DOWNLOADED',
                            body: httpEvent.map(x => {
                                return {
                                    downloadModel: x.downloadModel,
                                    downloaded: x.body
                                };
                            }),
                            httpEvent: null,
                            progress: 50,
                            zipFile: null
                        };
                    }

                    throw new Error('ZipStatus<Blob> unhandled switch case');

                }, {state: 'PENDING', progress: 0, body: [], httpEvent: null, zipFile: null} as ZipStatus<Blob>)
            ).subscribe({
                next: (zipStatus) => {
                    if (zipStatus.state !== 'DOWNLOADED') {
                        subscriber.next(zipStatus);
                    } else {
                        this.zip(zipStatus.body.map(x => {
                            return {
                                fileData: x.downloaded as Blob,
                                fileName: x.downloadModel.fileName
                            };
                        })).subscribe({
                            next: (data) => {
                                // console.log('zipping next');
                                subscriber.next(data);
                            },
                            complete: () => {
                                // console.log('zipping complete');
                                subscriber.complete();
                            },
                            error: (error) => {
                                // console.log('zipping error');

                            }
                        });

                    }
                },
                complete: () => {
                    // console.log('zip$ source complete: ');

                },
                error: (error) => {
                    // console.log('zip$ source error: ', error);
                }
            });


        }));

    }

    private zip(files: { fileName: string, fileData: Blob }[]): Observable<ZipStatus<Blob>> {
        return new Observable((subscriber) => {
            const zip = new JSZip();

            files.forEach(fileModel => {
                zip.file(fileModel.fileName, fileModel.fileData);
            });

            zip.generateAsync({type: "blob", streamFiles: true}, (metadata) => {
                subscriber.next({
                    state: 'ZIPPING',
                    body: [],
                    httpEvent: null,
                    progress: Math.trunc(metadata.percent / 2) + 50,
                    zipFile: null
                });

            }).then(function (content) {
                subscriber.next({
                    state: 'DONE',
                    body: [],
                    httpEvent: null,
                    progress: 100,
                    zipFile: content
                });

                subscriber.complete();
            });
        });
    }

    private isHttpSent<T>(event: HttpEvent<T>): event is HttpResponse<T> {
        return event.type === HttpEventType.Sent;
    }

    private isHttpResponse<T>(event: HttpEvent<T>): event is HttpResponse<T> {
        return event.type === HttpEventType.Response;
    }


    private isHttpHeaderResponse<T>(event: HttpEvent<T>): event is HttpHeaderResponse {
        return event.type === HttpEventType.ResponseHeader;
    }


    private isHttpUserEvent<T>(event: HttpEvent<T>): event is HttpUserEvent<T> {
        return event.type === HttpEventType.User;
    }

    private isHttpProgressEvent(event: HttpEvent<Blob>): event is HttpProgressEvent {
        return (
            event.type === HttpEventType.DownloadProgress ||
            event.type === HttpEventType.UploadProgress
        );
    }
}

并且您可以从存储库 link

中找到用于在同一磁贴中下载多个文件的代码

这里是调用它的地方：

export class AppComponent {
    constructor(
        private http: HttpClient,
        private downloadService: DownloadService
    ) {
    }

    start() {
        console.log('start');
        const file1 = new DownloadModel();
        file1.link = 'http://localhost:3000?name=1.jpg';
        file1.fileSize = 41252062;
        file1.fileName = '1.jpg';
        const file2 = new DownloadModel();
        file2.link = 'http://localhost:3000?name=2.jpg';
        file2.fileSize = 39986505;
        file2.fileName = '2.jpg';

        const download$ = this.downloadService.downloadMultiple([file1, file2]).pipe(tap({
            next: (data) => {
                // console.log('download$ next: ', data);
            },
            complete: () => {
                // console.log('download$ complete: ');

            },
            error: (error) => {
                // console.log('download$ error: ', error);

            }
        }));

        const zip$ = this.downloadService.zipMultiple(download$);

        zip$
            .pipe(distinctUntilKeyChanged('progress')).subscribe({
            next: (data) => {
                console.log('zip$ next: ', data);

                if (data.zipFile) {
                    const downloadAncher = document.createElement("a");
                    downloadAncher.style.display = "none";
                    downloadAncher.href = URL.createObjectURL(data.zipFile);
                    downloadAncher.download = 'images.zip';
                    downloadAncher.click();
                }
            },
            complete: () => {
                console.log('zip$ complete: ');

            },
            error: (error) => {
                console.log('zip$ error: ', error);

            }
        });
    }
}

下载许多 zip 中的大文件。正确实施

Download Many heavy files in zip. Correct implementation

typescript

jszip

angular