为什么 subprocess.Popen 参数长度限制小于 OS 报告的长度?
Why is the subprocess.Popen argument length limit smaller than what the OS reports?
我在 运行宁 Python 3.4.3 Linux 3.16.0。我想使用 subprocess.Popen
到 运行 一个带有长单个参数的命令(一个复杂的 Bash 调用),大约 200KiB。
根据getconf
和xargs
,这应该在我的能力范围内:
$ getconf ARG_MAX
2097152
$ xargs --show-limits < /dev/null
Your environment variables take up 3364 bytes
POSIX upper limit on argument length (this system): 2091740
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088376
Size of command buffer we are actually using: 131072
但是,Python 失败了,限制非常小:
>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (131072-4096)), shell=True, executable='/bin/bash')
<subprocess.Popen object at 0x7f4613b58410>
>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (262144-4096)), shell=True, executable='/bin/bash')
Traceback (most recent call last):
[...]
OSError: [Errno 7] Argument list too long
请注意,Python 限制与 "actually using" 命令缓冲区 xargs
报告大致相同。这表明 xargs
在某种程度上足够聪明,可以从较小的限制开始并根据需要增加它,但 Python 不是。
问题:
- 为什么 Python 限制小于 OS 2MiB 的限制?
- 我可以提高 Python 限制吗?
- 如果是,怎么做?
This 其他问题与您的相似,但 Windows。在那种情况下,您可以通过避免 shell=True
选项来绕过任何 shell 限制。
否则,您可以向 subprocess.Popen()
提供文件列表,就像在那个场景中所做的那样,并按照@Aaron Digulla 的建议。
单个字符串参数的最大大小限制为131072。与python无关:
~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null
~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/null
bash: /bin/echo: Argument list too long
实际上是 MAX_ARG_STRLEN
决定了单个字符串的最大大小:
And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072).
This might become relevant if you generate a long call like "sh -c 'generated with long arguments'".
(pointed out by Xan Lopez and Ralf Wildenhues)
参见 unix.stackexchange
上的 this discussion of ARG_MAX
, under "Number of arguments and maximum length of one argument", and this question。
你可以在binfmts.h
看到它:
/*
* These are the maximum length and maximum number of strings passed to the
* execve() system call. MAX_ARG_STRLEN is essentially random but serves to
* prevent the kernel from being unduly impacted by misaddressed pointers.
* MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
*/
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF
~$ echo $(( $(getconf PAGE_SIZE)*32 ))
131072
您可以传递多个长度为 131071
:
的字符串
subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w"))
但单个字符串 arg 不能超过 131071 字节。
我在 运行宁 Python 3.4.3 Linux 3.16.0。我想使用 subprocess.Popen
到 运行 一个带有长单个参数的命令(一个复杂的 Bash 调用),大约 200KiB。
根据getconf
和xargs
,这应该在我的能力范围内:
$ getconf ARG_MAX
2097152
$ xargs --show-limits < /dev/null
Your environment variables take up 3364 bytes
POSIX upper limit on argument length (this system): 2091740
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088376
Size of command buffer we are actually using: 131072
但是,Python 失败了,限制非常小:
>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (131072-4096)), shell=True, executable='/bin/bash')
<subprocess.Popen object at 0x7f4613b58410>
>>> subprocess.Popen('echo %s > /dev/null' % ('a' * (262144-4096)), shell=True, executable='/bin/bash')
Traceback (most recent call last):
[...]
OSError: [Errno 7] Argument list too long
请注意,Python 限制与 "actually using" 命令缓冲区 xargs
报告大致相同。这表明 xargs
在某种程度上足够聪明,可以从较小的限制开始并根据需要增加它,但 Python 不是。
问题:
- 为什么 Python 限制小于 OS 2MiB 的限制?
- 我可以提高 Python 限制吗?
- 如果是,怎么做?
This 其他问题与您的相似,但 Windows。在那种情况下,您可以通过避免 shell=True
选项来绕过任何 shell 限制。
否则,您可以向 subprocess.Popen()
提供文件列表,就像在那个场景中所做的那样,并按照@Aaron Digulla 的建议。
单个字符串参数的最大大小限制为131072。与python无关:
~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null
~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/null
bash: /bin/echo: Argument list too long
实际上是 MAX_ARG_STRLEN
决定了单个字符串的最大大小:
And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072). This might become relevant if you generate a long call like "sh -c 'generated with long arguments'". (pointed out by Xan Lopez and Ralf Wildenhues)
参见 unix.stackexchange
上的 this discussion of ARG_MAX
, under "Number of arguments and maximum length of one argument", and this question。
你可以在binfmts.h
看到它:
/*
* These are the maximum length and maximum number of strings passed to the
* execve() system call. MAX_ARG_STRLEN is essentially random but serves to
* prevent the kernel from being unduly impacted by misaddressed pointers.
* MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
*/
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF
~$ echo $(( $(getconf PAGE_SIZE)*32 ))
131072
您可以传递多个长度为 131071
:
subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w"))
但单个字符串 arg 不能超过 131071 字节。