GNU 并行 |管道命令

Question

我在使用 GNU parallel 方面是全新的，我需要您的建议运行使用 GNU[在下面的命令中使用 GNU parallel:

/home/admin/Gfinal/decoder/decdr.pl --gh --w14b /data/tmp/KRX12/a.bin | perl /home/admin/decout/decoder/flow.pl >> /data/tmp/decodedgfile/out_1.txt

我将运行这个命令放在文件列表 (.bin) 上，那么使用 GNU[=25= 实现该命令的最佳（最快）方法是什么？ ] parallel 注意到命令第一部分的输出 (/home/admin/Gfinal/decoder/decdr.pl --gh --w14b) 非常大 (> 2 GB)。

如有任何帮助，我们将不胜感激。

Answer 1

这里有一些关于 gnu-parallel / parallel 的精彩视频

参考 youtube Part 1: GNU Parallel script processing and execution

这是来自 GNU 网站的 link 平台特定信息。

参考 gnu parallel download information

"Multiple input sources

GNU parallel can take multiple input sources given on the command line. GNU parallel then generates all combinations of the input sources:

parallel echo ::: A B C ::: D E F

Output (the order may be different):

A D

A E

A F

B D

B E ............

The input sources can be files:

parallel -a abc-file -a def-file echo"

参考 GNU-Parallel-Tutorial

参考管道

Pipe capacity A pipe has a limited capacity. If the pipe is full, then a write(2) will block or fail, depending on whether the O_NONBLOCK flag is set (see below). Different implementations have different limits for the pipe capacity. Applications should not rely on a particular capacity: an application should be designed so that a reading process consumes data as soon as it is available, so that a writing process does not remain blocked.
  In Linux versions before 2.6.11, the capacity of a pipe was the same
   as the system page size (e.g., 4096 bytes on i386).  Since Linux
   2.6.11, the pipe capacity is 65536 bytes.  Since Linux 2.6.35, the
   default pipe capacity is 65536 bytes, but the capacity can be queried
   and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations.
   See fcntl(2) for more information.
PIPE_BUF POSIX.1 says that write(2)s of less than PIPE_BUF bytes must be atomic: the output data is written to the pipe as a contiguous sequence. Writes of more than PIPE_BUF bytes may be nonatomic: the kernel may interleave the data with data written by other processes. POSIX.1 requires PIPE_BUF to be at least 512 bytes. (On Linux, PIPE_BUF is 4096 bytes.) The precise semantics depend on whether the file descriptor is nonblocking (O_NONBLOCK), whether there are multiple writers to the pipe, and on n, the number of bytes to be written:

参考 man7.org pipe

您可以查看 fcntl F_GETPIPE_SZ 和 F_SETPIPE_SZ 操作以获取更多信息。

参考 fcntl

祝一切顺利

Answer 2

这行得通吗：

parallel /home/admin/Gfinal/decoder/decdr.pl --gh --w14b {} '|' perl  /home/admin/decout/decoder/flow.pl >> /data/tmp/decodedgfile/out_1.txt ::: /data/tmp/KRX12/*.bin

（如果 flow.pl 的输出超出您的磁盘 I/O 的处理能力，请尝试 parallel --compress）。

或者也许：

parallel /home/admin/Gfinal/decoder/decdr.pl --gh --w14b {} '|' perl  /home/admin/decout/decoder/flow.pl '>>' /data/tmp/decodedgfile/out_{#}.txt ::: /data/tmp/KRX12/*.bin

这取决于您想要单个输出文件还是每个输入文件一个。

另外花一个小时浏览教程。您的命令行会因此爱上您。 man parallel_tutorial

GNU 并行 |管道命令

GNU Parallel | pipe command

linux

greenplum

gnu-parallel