限制 gsutil -m 线程
Limit gsutil -m threads
有没有办法限制 gsutil -m 命令生成的线程数？我可以说 gsutil -m --threads=4 之类的话来正好生成四个线程吗？
您应该将 boto 配置文件中的 parallel_thread_count 和 parallel_process_count 值设置为 4
Gsutil Top-Level Command-Line Options
-m flag
Causes supported operations (acl ch, acl set, cp, mv, rm, rsync, and
setmeta) to run in parallel. This can significantly improve
performance if you are performing operations on a large number of
files over a reasonably fast network connection.
gsutil performs the specified operation using a combination of
multi-threading and multi-processing, using a number of threads and
processors determined by the parallel_thread_count and
parallel_process_count values set in the boto configuration file. You
might want to experiment with these values, as the best values can
vary based on a number of factors, including network speed, number of
CPUs, and available memory.
Using the -m option may make your performance worse if you are using a
slower network, such as the typical network speeds offered by
non-business home network plans. It can also make your performance
worse for cases that perform all operations locally (e.g., gsutil
rsync, where both source and destination URLs are on the local disk),
because it can "thrash" your local disk.
If a download or upload operation using parallel transfer fails before
the entire transfer is complete (e.g. failing after 300 of 1000 files
have been transferred), you will need to restart the entire transfer.
Also, although most commands will normally fail upon encountering an
error when the -m flag is disabled, all commands will continue to try
all operations when -m is enabled with multiple threads or processes,
and the number of failed operations (if any) will be reported as an
exception at the end of the command's execution.
有没有办法限制 gsutil -m 命令生成的线程数？我可以说 gsutil -m --threads=4 之类的话来正好生成四个线程吗？
您应该将 boto 配置文件中的 parallel_thread_count 和 parallel_process_count 值设置为 4
Gsutil Top-Level Command-Line Options
-m flag
Causes supported operations (acl ch, acl set, cp, mv, rm, rsync, and setmeta) to run in parallel. This can significantly improve performance if you are performing operations on a large number of files over a reasonably fast network connection.
gsutil performs the specified operation using a combination of multi-threading and multi-processing, using a number of threads and processors determined by the parallel_thread_count and parallel_process_count values set in the boto configuration file. You might want to experiment with these values, as the best values can vary based on a number of factors, including network speed, number of CPUs, and available memory.
Using the -m option may make your performance worse if you are using a slower network, such as the typical network speeds offered by non-business home network plans. It can also make your performance worse for cases that perform all operations locally (e.g., gsutil rsync, where both source and destination URLs are on the local disk), because it can "thrash" your local disk.
If a download or upload operation using parallel transfer fails before the entire transfer is complete (e.g. failing after 300 of 1000 files have been transferred), you will need to restart the entire transfer.
Also, although most commands will normally fail upon encountering an error when the -m flag is disabled, all commands will continue to try all operations when -m is enabled with multiple threads or processes, and the number of failed operations (if any) will be reported as an exception at the end of the command's execution.