Valgrind 1017 calloc/memcopy of struct dirent OK, 1018th -- 无效读取
Valgrind 1017 calloc/memcopy of struct dirent OK, 1018th -- invalid read
全部，我不知道我达到了什么限制，或者这是否是 valgrind、libc 或 me 的问题，但我需要知道这是否是可重现的，如果是这样，这个问题在哪里。我已将问题归结为可在我的 2 个 AMD 机器上生产的 MCVE。基本上，我动态分配 struct dirent * 指针，然后为每个成功的 readdir 分配一个 struct dirent。 valgrind 没有对 1017 的投诉，但是在数字 1018 上，我得到一个 invalid read 错误（不涉及重新分配），例如
==9881== Invalid read of size 8
==9881== at 0x4C2F316: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x40098E: main (readdir_mcve.c:35)
==9881== Address 0x51df070 is 0 bytes after a block of size 32,816 alloc'd
==9881== at 0x4C2ABD0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x4EE93E3: __alloc_dir (in /usr/lib/libc-2.23.so)
==9881== by 0x4EE94D2: opendir_tail (in /usr/lib/libc-2.23.so)
==9881== by 0x400802: main (readdir_mcve.c:9)
（block of size 32,816 看起来很奇怪，但我没有找到任何帮助来分解它）
代码将要打开的目录名作为第一个参数，然后将要读取的文件限制作为第二个参数（默认为1000）：
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <dirent.h>
int main (int argc, char **argv) {
DIR *dp = opendir (argc > 1 ? argv[1] : "."); /* open directory (. default) */
struct dirent *de = NULL, **dlist = NULL; /* ptr and ptr2ptr to dirent */
size_t nptrs = argc > 2 ? (size_t)strtoul (argv[2], NULL, 10) : 1000,
i = 0, idx = 0; /* index, allocation counter */
if (!dp) {
fprintf (stderr, "error: opendir failed.\n");
return 1;
}
/* allocate nptrs dirent pointers */
if (!(dlist = calloc (nptrs, sizeof *dlist))) {
fprintf (stderr, "error: virtual memory exhausted - dlist\n");
return 1;
}
while ((de = readdir (dp))) {
/* skip dot files */
if (!strcmp (de->d_name, ".") || !strcmp (de->d_name, ".."))
continue;
if (!(dlist[idx] = calloc (1, sizeof **dlist))) { /* alloc dirent */
fprintf (stderr, "error: dlist memory allocation failed\n");
return 1;
}
memcpy (dlist[idx++], de, sizeof *de); /* copy de to dlist[idx] */
if (idx == nptrs) /* post-check/realloc, insures sentinel NULL */
break;
}
closedir (dp);
for (i = 0; i < idx; i++) {
printf (" file[%3zu] : %s\n", i, dlist[i]->d_name);
free (dlist[i]);
}
free (dlist);
return 0;
}
您可以创建一个简单的测试目录：
$ mkdir readdir_tst
$ for i in {1..1024}; do
printf -v fname "file%04d" "$i"
touch "readdir_tst/$fname"
done
然后一切正常读取 1014 文件名（1017 分配）：
$ valgrind ./bin/readdir_mcve readdir_tst 1014 > /dev/null
==9880== Memcheck, a memory error detector
==9880== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9880== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9880== Command: ./bin/readdir_mcve readdir_tst 1014
==9880==
==9880==
==9880== HEAP SUMMARY:
==9880== in use at exit: 0 bytes in 0 blocks
==9880== total heap usage: 1,017 allocs, 1,017 frees, 328,944 bytes allocated
==9880==
==9880== All heap blocks were freed -- no leaks are possible
==9880==
==9880== For counts of detected and suppressed errors, rerun with: -v
==9880== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
但是在文件 1015（分配 1018）上我遇到了一个 __alloc_dir 问题并且 valgrind 抛出了一个 Invalid read of size 8 ... is 0 bytes after a block of size 32,816 alloc'd:
$ valgrind ./bin/readdir_mcve readdir_tst 1015 > /dev/null
==9881== Memcheck, a memory error detector
==9881== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9881== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9881== Command: ./bin/readdir_mcve readdir_tst 1015
==9881==
==9881== Invalid read of size 8
==9881== at 0x4C2F316: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x40098E: main (readdir_mcve.c:35)
==9881== Address 0x51df070 is 0 bytes after a block of size 32,816 alloc'd
==9881== at 0x4C2ABD0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x4EE93E3: __alloc_dir (in /usr/lib/libc-2.23.so)
==9881== by 0x4EE94D2: opendir_tail (in /usr/lib/libc-2.23.so)
==9881== by 0x400802: main (readdir_mcve.c:9)
==9881==
==9881==
==9881== HEAP SUMMARY:
==9881== in use at exit: 0 bytes in 0 blocks
==9881== total heap usage: 1,018 allocs, 1,018 frees, 329,232 bytes allocated
==9881==
==9881== All heap blocks were freed -- no leaks are possible
==9881==
==9881== For counts of detected and suppressed errors, rerun with: -v
==9881== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
代码继续读取和打印所有目录条目，但让我感到困惑的是 valgrind 错误。我让它重新分配而不是在达到分配限制时调用 break 并且它处理 /usr/bin 中的 4000+ 文件除了 valgrind 错误之外没有任何问题。 （因为与 MCVE 无关而被删除）。我在 SO 上找到的最接近的东西是 Valgrind malloc leaks，但这在此处不适用。任何人都可以重现这个吗？如果可以，这是 valgrind、libc 还是 me？
注意： 我得到与 libc-2.18.so 相同的结果。
GNU libc dirent.h 提供了更多信息
在被答案指向正确的方向并继续搜索之后，似乎有很多方法 libc 可以 确定 [=45= 的长度].这将取决于编译器可用的各种定义。在dirent.h中说明：
46 /* This file defines `struct dirent'.
47
48 It defines the macro `_DIRENT_HAVE_D_NAMLEN' iff there is a `d_namlen'
49 member that gives the length of `d_name'.
...
59 */
...
67 /* These macros extract size information from a `struct dirent *'.
68 They may evaluate their argument multiple times, so it must not
69 have side effects. Each of these may involve a relatively costly
70 call to `strlen' on some systems, so these values should be cached.
71
72 _D_EXACT_NAMLEN (DP) returns the length of DP->d_name, not including
73 its terminating null character.
74
75 _D_ALLOC_NAMLEN (DP) returns a size at least (_D_EXACT_NAMLEN (DP) + 1);
76 that is, the allocation size needed to hold the DP->d_name string.
77 Use this macro when you don't need the exact length, just an upper bound.
78 This macro is less likely to require calling `strlen' than _D_EXACT_NAMLEN.
79 */
80
81 #ifdef _DIRENT_HAVE_D_NAMLEN
82 # define _D_EXACT_NAMLEN(d) ((d)->d_namlen)
83 # define _D_ALLOC_NAMLEN(d) (_D_EXACT_NAMLEN (d) + 1)
84 #else
85 # define _D_EXACT_NAMLEN(d) (strlen ((d)->d_name))
86 # ifdef _DIRENT_HAVE_D_RECLEN
87 # define _D_ALLOC_NAMLEN(d) (((char *) (d) + (d)->d_reclen) - &(d)->d_name[0])
88 # else
89 # define _D_ALLOC_NAMLEN(d) (sizeof (d)->d_name > 1 ? sizeof (d)->d_name : \
90 _D_EXACT_NAMLEN (d) + 1)
91 # endif
92 #endif
...
虽然根据不同的定义可以采用多种不同的编译路径，但如果设置了 __USE_XOPEN2K8，则出现最易读的路径：
221 #ifdef __USE_XOPEN2K8
222 ...
230 # ifdef __USE_MISC
231 # ifndef MAXNAMLEN
232 /* Get the definitions of the POSIX.1 limits. */
233 # include <bits/posix1_lim.h>
234
235 /* `MAXNAMLEN' is the BSD name for what POSIX calls `NAME_MAX'. */
236 # ifdef NAME_MAX
237 # define MAXNAMLEN NAME_MAX
238 # else
239 # define MAXNAMLEN 255
240 # endif
241 # endif
242 # endif
所以在这种情况下，d_name 是 NAME_MAX 或 255，具体取决于 NAME_MAX 定义（因此 256 由 _D_ALLOC_NAMLEN (DP)宏）。感谢 unwind 为我指明了正确的方向。我不知道我们是否能知道为什么 1017 struct dirent 分配没有问题以及为什么 valgrind 开始抱怨数字 1018 的确切答案，但至少我们现在明白问题的根源在哪里以及为什么用 memcpy 复制 struct dirent 可能会带来问题。
不能这样复制strucft dirent，好像the manual page和代码不同步
struct dirent
{
#ifndef __USE_FILE_OFFSET64
__ino_t d_ino; /* File serial number. */
#else
__ino64_t d_ino;
#endif
unsigned short int d_reclen; /* Length of the whole `struct dirent'. */
unsigned char d_type; /* File type, possibly unknown. */
unsigned char d_namlen; /* Length of the file name. */
/* Only this member is in the POSIX standard. */
char d_name[1]; /* File name (actually longer). */
};
显然，由于 d_name 被声明为 [1]，您将无法使用 sizeof 获得正确的大小。您需要做更多巧妙的存储，即 strdup() 名称或其他内容（如果您只对名称感兴趣）。
我不是 100% 确定为什么这会导致破损，但我敢打赌你看到了某种 UB（请注意，如果你的副本达到 printf()，你将死于阅读会触发 UB 的字符串）。
全部，我不知道我达到了什么限制，或者这是否是 valgrind、libc 或 me 的问题，但我需要知道这是否是可重现的，如果是这样，这个问题在哪里。我已将问题归结为可在我的 2 个 AMD 机器上生产的 MCVE。基本上，我动态分配 struct dirent * 指针，然后为每个成功的 readdir 分配一个 struct dirent。 valgrind 没有对 1017 的投诉，但是在数字 1018 上，我得到一个 invalid read 错误（不涉及重新分配），例如
==9881== Invalid read of size 8
==9881== at 0x4C2F316: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x40098E: main (readdir_mcve.c:35)
==9881== Address 0x51df070 is 0 bytes after a block of size 32,816 alloc'd
==9881== at 0x4C2ABD0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x4EE93E3: __alloc_dir (in /usr/lib/libc-2.23.so)
==9881== by 0x4EE94D2: opendir_tail (in /usr/lib/libc-2.23.so)
==9881== by 0x400802: main (readdir_mcve.c:9)
（block of size 32,816 看起来很奇怪，但我没有找到任何帮助来分解它）
代码将要打开的目录名作为第一个参数，然后将要读取的文件限制作为第二个参数（默认为1000）：
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <dirent.h>
int main (int argc, char **argv) {
DIR *dp = opendir (argc > 1 ? argv[1] : "."); /* open directory (. default) */
struct dirent *de = NULL, **dlist = NULL; /* ptr and ptr2ptr to dirent */
size_t nptrs = argc > 2 ? (size_t)strtoul (argv[2], NULL, 10) : 1000,
i = 0, idx = 0; /* index, allocation counter */
if (!dp) {
fprintf (stderr, "error: opendir failed.\n");
return 1;
}
/* allocate nptrs dirent pointers */
if (!(dlist = calloc (nptrs, sizeof *dlist))) {
fprintf (stderr, "error: virtual memory exhausted - dlist\n");
return 1;
}
while ((de = readdir (dp))) {
/* skip dot files */
if (!strcmp (de->d_name, ".") || !strcmp (de->d_name, ".."))
continue;
if (!(dlist[idx] = calloc (1, sizeof **dlist))) { /* alloc dirent */
fprintf (stderr, "error: dlist memory allocation failed\n");
return 1;
}
memcpy (dlist[idx++], de, sizeof *de); /* copy de to dlist[idx] */
if (idx == nptrs) /* post-check/realloc, insures sentinel NULL */
break;
}
closedir (dp);
for (i = 0; i < idx; i++) {
printf (" file[%3zu] : %s\n", i, dlist[i]->d_name);
free (dlist[i]);
}
free (dlist);
return 0;
}
您可以创建一个简单的测试目录：
$ mkdir readdir_tst
$ for i in {1..1024}; do
printf -v fname "file%04d" "$i"
touch "readdir_tst/$fname"
done
然后一切正常读取 1014 文件名（1017 分配）：
$ valgrind ./bin/readdir_mcve readdir_tst 1014 > /dev/null
==9880== Memcheck, a memory error detector
==9880== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9880== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9880== Command: ./bin/readdir_mcve readdir_tst 1014
==9880==
==9880==
==9880== HEAP SUMMARY:
==9880== in use at exit: 0 bytes in 0 blocks
==9880== total heap usage: 1,017 allocs, 1,017 frees, 328,944 bytes allocated
==9880==
==9880== All heap blocks were freed -- no leaks are possible
==9880==
==9880== For counts of detected and suppressed errors, rerun with: -v
==9880== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
但是在文件 1015（分配 1018）上我遇到了一个 __alloc_dir 问题并且 valgrind 抛出了一个 Invalid read of size 8 ... is 0 bytes after a block of size 32,816 alloc'd:
$ valgrind ./bin/readdir_mcve readdir_tst 1015 > /dev/null
==9881== Memcheck, a memory error detector
==9881== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9881== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9881== Command: ./bin/readdir_mcve readdir_tst 1015
==9881==
==9881== Invalid read of size 8
==9881== at 0x4C2F316: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x40098E: main (readdir_mcve.c:35)
==9881== Address 0x51df070 is 0 bytes after a block of size 32,816 alloc'd
==9881== at 0x4C2ABD0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881== by 0x4EE93E3: __alloc_dir (in /usr/lib/libc-2.23.so)
==9881== by 0x4EE94D2: opendir_tail (in /usr/lib/libc-2.23.so)
==9881== by 0x400802: main (readdir_mcve.c:9)
==9881==
==9881==
==9881== HEAP SUMMARY:
==9881== in use at exit: 0 bytes in 0 blocks
==9881== total heap usage: 1,018 allocs, 1,018 frees, 329,232 bytes allocated
==9881==
==9881== All heap blocks were freed -- no leaks are possible
==9881==
==9881== For counts of detected and suppressed errors, rerun with: -v
==9881== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
代码继续读取和打印所有目录条目，但让我感到困惑的是 valgrind 错误。我让它重新分配而不是在达到分配限制时调用 break 并且它处理 /usr/bin 中的 4000+ 文件除了 valgrind 错误之外没有任何问题。 （因为与 MCVE 无关而被删除）。我在 SO 上找到的最接近的东西是 Valgrind malloc leaks，但这在此处不适用。任何人都可以重现这个吗？如果可以，这是 valgrind、libc 还是 me？
注意： 我得到与 libc-2.18.so 相同的结果。
GNU libc dirent.h 提供了更多信息
在被答案指向正确的方向并继续搜索之后，似乎有很多方法 libc 可以 确定 [=45= 的长度].这将取决于编译器可用的各种定义。在dirent.h中说明：
46 /* This file defines `struct dirent'.
47
48 It defines the macro `_DIRENT_HAVE_D_NAMLEN' iff there is a `d_namlen'
49 member that gives the length of `d_name'.
...
59 */
...
67 /* These macros extract size information from a `struct dirent *'.
68 They may evaluate their argument multiple times, so it must not
69 have side effects. Each of these may involve a relatively costly
70 call to `strlen' on some systems, so these values should be cached.
71
72 _D_EXACT_NAMLEN (DP) returns the length of DP->d_name, not including
73 its terminating null character.
74
75 _D_ALLOC_NAMLEN (DP) returns a size at least (_D_EXACT_NAMLEN (DP) + 1);
76 that is, the allocation size needed to hold the DP->d_name string.
77 Use this macro when you don't need the exact length, just an upper bound.
78 This macro is less likely to require calling `strlen' than _D_EXACT_NAMLEN.
79 */
80
81 #ifdef _DIRENT_HAVE_D_NAMLEN
82 # define _D_EXACT_NAMLEN(d) ((d)->d_namlen)
83 # define _D_ALLOC_NAMLEN(d) (_D_EXACT_NAMLEN (d) + 1)
84 #else
85 # define _D_EXACT_NAMLEN(d) (strlen ((d)->d_name))
86 # ifdef _DIRENT_HAVE_D_RECLEN
87 # define _D_ALLOC_NAMLEN(d) (((char *) (d) + (d)->d_reclen) - &(d)->d_name[0])
88 # else
89 # define _D_ALLOC_NAMLEN(d) (sizeof (d)->d_name > 1 ? sizeof (d)->d_name : \
90 _D_EXACT_NAMLEN (d) + 1)
91 # endif
92 #endif
...
虽然根据不同的定义可以采用多种不同的编译路径，但如果设置了 __USE_XOPEN2K8，则出现最易读的路径：
221 #ifdef __USE_XOPEN2K8
222 ...
230 # ifdef __USE_MISC
231 # ifndef MAXNAMLEN
232 /* Get the definitions of the POSIX.1 limits. */
233 # include <bits/posix1_lim.h>
234
235 /* `MAXNAMLEN' is the BSD name for what POSIX calls `NAME_MAX'. */
236 # ifdef NAME_MAX
237 # define MAXNAMLEN NAME_MAX
238 # else
239 # define MAXNAMLEN 255
240 # endif
241 # endif
242 # endif
所以在这种情况下，d_name 是 NAME_MAX 或 255，具体取决于 NAME_MAX 定义（因此 256 由 _D_ALLOC_NAMLEN (DP)宏）。感谢 unwind 为我指明了正确的方向。我不知道我们是否能知道为什么 1017 struct dirent 分配没有问题以及为什么 valgrind 开始抱怨数字 1018 的确切答案，但至少我们现在明白问题的根源在哪里以及为什么用 memcpy 复制 struct dirent 可能会带来问题。
不能这样复制strucft dirent，好像the manual page和代码不同步
struct dirent
{
#ifndef __USE_FILE_OFFSET64
__ino_t d_ino; /* File serial number. */
#else
__ino64_t d_ino;
#endif
unsigned short int d_reclen; /* Length of the whole `struct dirent'. */
unsigned char d_type; /* File type, possibly unknown. */
unsigned char d_namlen; /* Length of the file name. */
/* Only this member is in the POSIX standard. */
char d_name[1]; /* File name (actually longer). */
};
显然，由于 d_name 被声明为 [1]，您将无法使用 sizeof 获得正确的大小。您需要做更多巧妙的存储，即 strdup() 名称或其他内容（如果您只对名称感兴趣）。
我不是 100% 确定为什么这会导致破损，但我敢打赌你看到了某种 UB（请注意，如果你的副本达到 printf()，你将死于阅读会触发 UB 的字符串）。