使用 NodeJS 下载限速文件的最佳方法是什么？

Question

我正在尝试找出使用 NodeJS 的内置 HTTPS 模块限速下载文件的最佳方法。（在底部有一个完整的 python 实现我正在尝试做的事情。）我已经编写了两个不同的函数，它们似乎都按预期完成了工作。

download1 函数，检查当前秒是否超过速度限制，如果是，则暂停下载并创建一个超时，该超时会在恢复下载的那一秒结束时触发。

download2 但是，不是创建超时，而是创建一个每 1000 毫秒触发一次的间隔，并在下载暂停时恢复下载。

我想知道这两种方法中哪一种更好，或者我是否应该采用完全不同的方法。

函数如下：

export const download1 = (url: string, fileName: string, speedLimitInKb: number) => {
    return new Promise((resolve, _reject) => {
        https.get(url, res => {
            const stream = fs.createWriteStream(fileName);
            let totalSize = 0;
            let size = 0;
            let speedLimit = kbToBytes(speedLimitInKb);
            let startDate: number;
            let lastSecond = Date.now();

            res.pipe(stream);

            res.once("resume", () => {
                startDate = Date.now();
                console.log(`Started at ${new Date(startDate)}`)
            })

            res.on("data", (chunk) => {
                size += chunk.length;
                const now = Date.now();
                if (now - lastSecond > 1000) {
                    lastSecond = Date.now();
                    totalSize += size;
                    size = 0;
                } else if (size >= speedLimit) {
                    res.pause();
                    setTimeout(() => res.resume(), 1000 - (now - lastSecond));
                }
            });

            res.on("resume", () => {
                lastSecond = Date.now();
                totalSize += size;
                size = 0;
            })

            res.on("end", () => {
                const elapsed = (Date.now() - startDate) / 1000;
                totalSize += size
                stream.end();
                console.log(`${bytesToMb(totalSize)} mb of data downloaded in ${elapsed} seconds with a speed of ${bytesToKb(totalSize) / elapsed}`)
                resolve(undefined);
            });

            res.on("error", console.log);
        })
    })
};

export const download2 = (url: string, fileName: string, speedLimitInKb: number) => {
    return new Promise((resolve, _reject) => {
        https.get(url, res => {
            const stream = fs.createWriteStream(fileName);
            let totalSize = 0;
            let size = 0;
            let speedLimit = kbToBytes(speedLimitInKb);
            let startDate: number;
            res.pipe(stream);

            res.once("resume", () => {
                startDate = Date.now();
                console.log(`Started at ${new Date(startDate)}`)
            })

            const interval = setInterval(() => {
                if (res.isPaused()) {
                    res.resume();
                }
                totalSize += size;
                size = 0;
            }, 1000);

            res.on("data", (chunk) => {
                size += chunk.length;
                if (size >= speedLimit) {
                    res.pause();
                }
            });

            res.on("end", () => {
                clearInterval(interval);
                const elapsed = (Date.now() - startDate) / 1000;
                totalSize += size
                stream.end();
                console.log(`${bytesToMb(totalSize)} mb of data downloaded in ${elapsed} seconds with a speed of ${bytesToKb(totalSize) / elapsed}`)
                resolve(undefined);
            });

            res.on("error", console.log);
        });
    })
}

附加功能：

export const bytesToKb = (bytes: number) => bytes / 1024;
export const kbToMb = (kb: number) => kb / 1024;
export const kbToBytes = (kb: number) => kb * 1024;
export const mbToKb = (mb: number) => mb * 1024;
export const mbToBytes = (mb: number) => mb * 1024 * 1024;
export const bytesToMb = (bytes: number) => bytes / 1024 / 1024;
export const bytesToGb = (bytes: number) => bytes / 1024 / 1024 / 1024;
export const secondsToMs = (seconds: number) => seconds * 1000;
export const msToSeconds = (ms: number) => ms / 1000;

我已经写了一个 Python 版本来实现我想要实现的目标，它适用于任何速度限制和文件大小。我想弄清楚如何在 nodejs 中实现它：

import requests
import time

def download(url, file_name, speed_limit_in_kb):
    start = time.time()
    size = 0
    total_size = 0
    with open(file_name, "wb") as f:
        with requests.get(url, stream=True) as res:
            last_second = time.time()
            for part in res.iter_content(1024):
                f.write(part)
                total_size += len(part)
                size += len(part)
                offset = time.time() - last_second
                if offset > 1:
                    size = 0
                    last_second = time.time()
                elif size > (1024 * speed_limit_in_kb):
                    time.sleep(1 - offset)
                    size = 0
                    last_second = time.time()
    elapsed = time.time() - start
    print(f"{total_size / 1024 / 1024} mb of data downloaded in {elapsed} seconds with a speed of {total_size / 1024 / elapsed}")

Answer 1

这类问题肯定会得到自以为是的答案。就个人而言，我会使用 nodejs 的 built-in 流功能来进行节流。使用这种方法的观察结果：

最少的代码
代码依赖于 (nodejs) 库代码而不是自定义代码
高性能。最小的开销，也 wrt。内存
缺点：对于那些不熟悉流的人来说，代码似乎很复杂

import fs from "fs";
import https from "https";
import stream from "stream";
import util from "util";

async function downloadWithBackpressure(url, filename, byteRate) {
    let totalBytesDownloaded = 0;
    const timeBeforeStart = Date.now();

    await util.promisify(stream.pipeline)(

        // Start the download stream
        await new Promise(resolve => https.get(url, resolve)),

        // Throttle data by combining setTimeout with a stream.Transform
        new stream.Transform({
            transform: async (chunk, encoding, next) => {
                // Accumulate the total number of bytes received
                totalBytesDownloaded += chunk.byteLength;

                // Sleep to throttle towards desired transfer speed
                const sleepMs = Math.max(0, (totalBytesDownloaded / byteRate * 1000) - Date.now() + timeBeforeStart);
                sleepMs && await new Promise(resolve => setTimeout(resolve, sleepMs));

                // Propagate the chunk to the stream writable
                next(null, chunk);
            }
        }),

        // Save the file to disk
        fs.createWriteStream(filename)
    );
}

使用 NodeJS 下载限速文件的最佳方法是什么？

What is the best way to download a file with a speed limit using NodeJS?

javascript

download

node.js

python-requests

typescript