使用 gdb 调试应用程序 运行 不同的 libc (ld-linux.so)

Debug with gdb an application running with different libc (ld-linux.so)

我想在嵌入式系统上使用 gdb 调试应用程序。此应用程序需要更新的 glibc、线程并通过调用 Linux 动态加载器 ld-linux-x86-64.so.2 正常工作。我想要的是附加 gdb 并查看符号和堆栈跟踪,但加载程序“干扰”gdb.

这是一个示例 test.c 文件:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>
#include <pthread.h>

void *subThread(void *arg) {
        printf("Thread started after '%d' chars.\n", (long *)arg);
        sleep(2);
        printf("Thread '%d' ended.\n", (long *)arg);
        pthread_exit(NULL);

}
int main() {
        pid_t pid = getpid();
        if (pid > 0) {
                printf("The process id is %d.\n", pid);
                printf("Type 'q' to exit, 't' to spawn a thread.\n");

                char readChar;
                long count = 0;
                do {
                        count++;
                        readChar = getchar();
                        if (readChar == 't') {
                                // Spawn a thread
                                pthread_t thread_id;
                                pthread_create(&thread_id, NULL, subThread, (void *) count);
                                pthread_join(thread_id, NULL);
                        }
                } while(readChar != 'q');

                printf("Main ended.\n");
                return(0);
        } else {
                return(1);
        }
}

我在具有特定 glibc 版本的系统中编译此源文件,如下所示:

othersystem:~# gcc -O2 -g -lpthread -o test test.c

然后我将 test 二进制文件移动到我的嵌入式系统中,它有一个 较旧的 glibc 版本并且不能运行 test 直接,所以我也在复制加载程序和所需的库。另请参阅我的 gdb 设置。

[~/test] # ls
ld-linux-x86-64.so.2*  libc.so.6*  libpthread.so.0*  libthread_db.so.1  test*

[~/test] # cat ~/.gdbinit 
set auto-load safe-path /
set pagination off
set libthread-db-search-path /root/test

现在我 运行 代码并按 CTRLZ 暂停 我想附加gdb 使用进程 id。问题是 gdb 看到加载程序而不是应用程序,并且不加载调试符号。

[~/test] # /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test 
The process id is 30622.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+  Stopped(SIGTSTP)        /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test

[~/test] # gdb /root/test/test 30622
GNU gdb (GDB) 10.1
Reading symbols from /root/test/test...
Attaching to program: /root/test/test, process 30622

warning: Build ID mismatch between current exec-file /root/test/test
and automatically determined exec-file /root/test/ld-linux-x86-64.so.2
exec-file-mismatch handling is currently "ask"
Load new symbol table from "/root/test/ld-linux-x86-64.so.2"? (y or n) y
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Reading symbols from /root/test/libpthread.so.0...
(No debugging symbols found in /root/test/libpthread.so.0)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/root/test/libthread_db.so.1".
Reading symbols from /root/test/libc.so.6...
(No debugging symbols found in /root/test/libc.so.6)
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Reading symbols from /lib/libgcc_s.so.1...
0x00007fa1ef590494 in read () from /root/test/libc.so.6
(gdb) bt
#0  0x00007fa1ef590494 in read () from /root/test/libc.so.6
#1  0x00007fa1ef522670 in _IO_file_underflow () from /root/test/libc.so.6
#2  0x00007fa1ef5237b2 in _IO_default_uflow () from /root/test/libc.so.6
#3  0x00007fa1ef51de68 in getc () from /root/test/libc.so.6
#4  0x00007fa1ef68b110 in ?? ()
#5  0x00007fa1ef68b290 in ?? ()
#6  0x00007fa1ef4a2700 in ?? ()
#7  0x00007ffcdae50bb8 in ?? ()
#8  0x0000000000000000 in ?? ()

我不知道如何告诉 gdb/root/test/test 加载符号;有一些有趣的文章 here and here and also an answer on StackOveflow,但是 none 有效...我无法获得使用 add-symbol-file 指令的正确地址。 我试过这样:

[~/test] # cat /proc/30622/maps | grep "r-xp" | grep "/root/test/test" | awk -F'-' '{printf }'
7fa1ef68b000

[~/test] # objdump -s --section=".text" /root/test/test | grep Contents -A 1 | tail -n 1 | awk -F' ' '{printf }'                      
10c0

所以地址应该是0x7fa1ef68b000 + 0x10c0 = 0x7fa1ef68c0c0,对吧?但是当我将其输入 gdb 时,您可以看到它没有获取调试符号:

(gdb) add-symbol-file /root/test/test 0x7fa1ef68c0c0
add symbol table from file "/root/test/test" at
        .text_addr = 0x7fa1ef68c0c0
(y or n) y
Reading symbols from /root/test/test...
(gdb) bt
#0  0x00007fa1ef590494 in read () from /root/test/libc.so.6
#1  0x00007fa1ef522670 in _IO_file_underflow () from /root/test/libc.so.6
#2  0x00007fa1ef5237b2 in _IO_default_uflow () from /root/test/libc.so.6
#3  0x00007fa1ef51de68 in getc () from /root/test/libc.so.6
#4  0x00007fa1ef68b110 in ?? ()
#5  0x00007fa1ef68b290 in ?? ()
#6  0x00007fa1ef4a2700 in ?? ()
#7  0x00007ffcdae50bb8 in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb) 

有没有办法在不手动指定地址的情况下加载调试符号?如果我编译二进制文件以硬编码解释器路径,如下所示:

othersystem:~# gcc -O2 -g -lpthread -Wl,-rpath /root/test -Wl,--dynamic-linker=/root/test/ld-linux-x86-64.so.2 -o test2 test.c

然后一切正常:

[~/test] # ldd /root/test/test2
        linux-vdso.so.1 (0x00007ffcaa59c000)
        libpthread.so.0 => /root/test/libpthread.so.0 (0x00007f7265e1d000)
        libc.so.6 => /root/test/libc.so.6 (0x00007f7265c5c000)
        /root/test/ld-linux-x86-64.so.2 (0x00007f7265e40000)

[~/test] # /root/test/test2
The process id is 23264.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+  Stopped(SIGTSTP)        ./test2

[~/test] # gdb /root/test/test2 23264
GNU gdb (GDB) 10.1
Reading symbols from /root/test/test2...
Attaching to program: /root/test/test2, process 23264
Reading symbols from /root/test/libpthread.so.0...
(No debugging symbols found in /root/test/libpthread.so.0)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/root/test/libthread_db.so.1".
Reading symbols from /root/test/libc.so.6...
(No debugging symbols found in /root/test/libc.so.6)
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)

Program received signal SIGTSTP, Stopped (user).
0x00007ffba4e7d461 in read () from /root/test/libc.so.6
(gdb) bt
#0  0x00007ffba4e7d461 in read () from /root/test/libc.so.6
#1  0x00007ffba4e0f670 in _IO_file_underflow () from /root/test/libc.so.6
#2  0x00007ffba4e107b2 in _IO_default_uflow () from /root/test/libc.so.6
#3  0x00005613ef88a110 in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#4  main () at test.c:24

编辑

正如 Employed Russian 所问,我正在添加地图和 readelf 的完整输出(当然我必须使用新的 pid):

[~/test] # /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test 
The process id is 16873.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+  Stopped(SIGTSTP)        /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test

[~/test] # cat /proc/16873/maps
7f5e3767b000-7f5e3767e000 rw-p 00000000 00:00 0 
7f5e3767e000-7f5e376a0000 r--p 00000000 00:10 64323010                   /root/test/libc.so.6
7f5e376a0000-7f5e377e8000 r-xp 00022000 00:10 64323010                   /root/test/libc.so.6
7f5e377e8000-7f5e37834000 r--p 0016a000 00:10 64323010                   /root/test/libc.so.6
7f5e37834000-7f5e37835000 ---p 001b6000 00:10 64323010                   /root/test/libc.so.6
7f5e37835000-7f5e37839000 r--p 001b6000 00:10 64323010                   /root/test/libc.so.6
7f5e37839000-7f5e3783b000 rw-p 001ba000 00:10 64323010                   /root/test/libc.so.6
7f5e3783b000-7f5e3783f000 rw-p 00000000 00:00 0 
7f5e3783f000-7f5e37845000 r--p 00000000 00:10 64323011                   /root/test/libpthread.so.0
7f5e37845000-7f5e37854000 r-xp 00006000 00:10 64323011                   /root/test/libpthread.so.0
7f5e37854000-7f5e3785a000 r--p 00015000 00:10 64323011                   /root/test/libpthread.so.0
7f5e3785a000-7f5e3785b000 r--p 0001a000 00:10 64323011                   /root/test/libpthread.so.0
7f5e3785b000-7f5e3785c000 rw-p 0001b000 00:10 64323011                   /root/test/libpthread.so.0
7f5e3785c000-7f5e37862000 rw-p 00000000 00:00 0 
7f5e37862000-7f5e37863000 r--p 00000000 00:10 64323013                   /root/test/test
7f5e37863000-7f5e37864000 r-xp 00001000 00:10 64323013                   /root/test/test
7f5e37864000-7f5e37865000 r--p 00002000 00:10 64323013                   /root/test/test
7f5e37865000-7f5e37866000 r--p 00002000 00:10 64323013                   /root/test/test
7f5e37866000-7f5e37867000 rw-p 00003000 00:10 64323013                   /root/test/test
7f5e37867000-7f5e37868000 r--p 00000000 00:10 64323009                   /root/test/ld-linux-x86-64.so.2
7f5e37868000-7f5e37886000 r-xp 00001000 00:10 64323009                   /root/test/ld-linux-x86-64.so.2
7f5e37886000-7f5e3788e000 r--p 0001f000 00:10 64323009                   /root/test/ld-linux-x86-64.so.2
7f5e3788e000-7f5e3788f000 r--p 00026000 00:10 64323009                   /root/test/ld-linux-x86-64.so.2
7f5e3788f000-7f5e37890000 rw-p 00027000 00:10 64323009                   /root/test/ld-linux-x86-64.so.2
7f5e37890000-7f5e37891000 rw-p 00000000 00:00 0 
7f5e3905f000-7f5e39080000 rw-p 00000000 00:00 0                          [heap]
7fff1fcd3000-7fff1fcf4000 rw-p 00000000 00:00 0                          [stack]
7fff1fd82000-7fff1fd85000 r--p 00000000 00:00 0                          [vvar]
7fff1fd85000-7fff1fd87000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

[~/test] # readelf -Wl /root/test/test

Elf file type is DYN (Shared object file)
Entry point 0x1160
There are 11 program headers, starting at offset 64

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  PHDR           0x000040 0x0000000000000040 0x0000000000000040 0x000268 0x000268 R   0x8
  INTERP         0x0002a8 0x00000000000002a8 0x00000000000002a8 0x00001c 0x00001c R   0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x000768 0x000768 R   0x1000
  LOAD           0x001000 0x0000000000001000 0x0000000000001000 0x0002fd 0x0002fd R E 0x1000
  LOAD           0x002000 0x0000000000002000 0x0000000000002000 0x000210 0x000210 R   0x1000
  LOAD           0x002dd8 0x0000000000003dd8 0x0000000000003dd8 0x000290 0x0002a8 RW  0x1000
  DYNAMIC        0x002de8 0x0000000000003de8 0x0000000000003de8 0x0001f0 0x0001f0 RW  0x8
  NOTE           0x0002c4 0x00000000000002c4 0x00000000000002c4 0x000044 0x000044 R   0x4
  GNU_EH_FRAME   0x002098 0x0000000000002098 0x0000000000002098 0x000044 0x000044 R   0x4
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0x10
  GNU_RELRO      0x002dd8 0x0000000000003dd8 0x0000000000003dd8 0x000228 0x000228 R   0x1

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt 
   03     .init .plt .plt.got .text .fini 
   04     .rodata .eh_frame_hdr .eh_frame 
   05     .init_array .fini_array .dynamic .got .got.plt .data .bss 
   06     .dynamic 
   07     .note.ABI-tag .note.gnu.build-id 
   08     .eh_frame_hdr 
   09     
   10     .init_array .fini_array .dynamic .got

[~/test] # readelf -WS /root/test/test
There are 37 section headers, starting at offset 0x4b18:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        00000000000002a8 0002a8 00001c 00   A  0   0  1
  [ 2] .note.ABI-tag     NOTE            00000000000002c4 0002c4 000020 00   A  0   0  4
  [ 3] .note.gnu.build-id NOTE            00000000000002e4 0002e4 000024 00   A  0   0  4
  [ 4] .gnu.hash         GNU_HASH        0000000000000308 000308 000028 00   A  5   0  8
  [ 5] .dynsym           DYNSYM          0000000000000330 000330 000168 18   A  6   1  8
  [ 6] .dynstr           STRTAB          0000000000000498 000498 0000da 00   A  0   0  1
  [ 7] .gnu.version      VERSYM          0000000000000572 000572 00001e 02   A  5   0  2
  [ 8] .gnu.version_r    VERNEED         0000000000000590 000590 000040 00   A  6   2  8
  [ 9] .rela.dyn         RELA            00000000000005d0 0005d0 0000d8 18   A  5   0  8
  [10] .rela.plt         RELA            00000000000006a8 0006a8 0000c0 18  AI  5  23  8
  [11] .init             PROGBITS        0000000000001000 001000 000017 00  AX  0   0  4
  [12] .plt              PROGBITS        0000000000001020 001020 000090 10  AX  0   0 16
  [13] .plt.got          PROGBITS        00000000000010b0 0010b0 000008 08  AX  0   0  8
  [14] .text             PROGBITS        00000000000010c0 0010c0 000231 00  AX  0   0 16
  [15] .fini             PROGBITS        00000000000012f4 0012f4 000009 00  AX  0   0  4
  [16] .rodata           PROGBITS        0000000000002000 002000 000097 00   A  0   0  8
  [17] .eh_frame_hdr     PROGBITS        0000000000002098 002098 000044 00   A  0   0  4
  [18] .eh_frame         PROGBITS        00000000000020e0 0020e0 000130 00   A  0   0  8
  [19] .init_array       INIT_ARRAY      0000000000003dd8 002dd8 000008 08  WA  0   0  8
  [20] .fini_array       FINI_ARRAY      0000000000003de0 002de0 000008 08  WA  0   0  8
  [21] .dynamic          DYNAMIC         0000000000003de8 002de8 0001f0 10  WA  6   0  8
  [22] .got              PROGBITS        0000000000003fd8 002fd8 000028 08  WA  0   0  8
  [23] .got.plt          PROGBITS        0000000000004000 003000 000058 08  WA  0   0  8
  [24] .data             PROGBITS        0000000000004058 003058 000010 00  WA  0   0  8
  [25] .bss              NOBITS          0000000000004070 003068 000010 00  WA  0   0 16
  [26] .comment          PROGBITS        0000000000000000 003068 00001c 01  MS  0   0  1
  [27] .debug_aranges    PROGBITS        0000000000000000 003084 000040 00      0   0  1
  [28] .debug_info       PROGBITS        0000000000000000 0030c4 000626 00      0   0  1
  [29] .debug_abbrev     PROGBITS        0000000000000000 0036ea 000200 00      0   0  1
  [30] .debug_line       PROGBITS        0000000000000000 0038ea 00020c 00      0   0  1
  [31] .debug_str        PROGBITS        0000000000000000 003af6 000308 01  MS  0   0  1
  [32] .debug_loc        PROGBITS        0000000000000000 003dfe 0000e8 00      0   0  1
  [33] .debug_ranges     PROGBITS        0000000000000000 003ee6 000090 00      0   0  1
  [34] .symtab           SYMTAB          0000000000000000 003f78 000780 18     35  52  8
  [35] .strtab           STRTAB          0000000000000000 0046f8 0002bc 00      0   0  1
  [36] .shstrtab         STRTAB          0000000000000000 0049b4 000160 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

最终解决方案

为了方便(将来)阅读,我在这里报告了 Employed Russian 解决方案后的输出。

[~/test] # /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test 
The process id is 16873.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+  Stopped(SIGTSTP)        /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test

[~/test] # cat /proc/16873/maps | grep "/root/test/test" | awk -F'-' '{print "0x" ; exit; }'
0x7f5e37862000
[~/test] # readelf -WS /root/test/test | grep '\.text ' | awk '{print "0x" }'
0x0010c0

[~/test] # gdb /root/test/test 16873
GNU gdb (GDB) 10.1
Reading symbols from /root/test/test...
Attaching to program: /root/test/test, process 16873

warning: Build ID mismatch between current exec-file /root/test/test
and automatically determined exec-file /root/test/ld-linux-x86-64.so.2
exec-file-mismatch handling is currently "ask"
Load new symbol table from "/root/test/ld-linux-x86-64.so.2"? (y or n) y
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Reading symbols from /root/test/libpthread.so.0...
(No debugging symbols found in /root/test/libpthread.so.0)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/root/test/libthread_db.so.1".
Reading symbols from /root/test/libc.so.6...
(No debugging symbols found in /root/test/libc.so.6)
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)

Program received signal SIGTTIN, Stopped (tty input).
0x00007f5e37768461 in read () from /root/test/libc.so.6
(gdb) bt
#0  0x00007f5e37768461 in read () from /root/test/libc.so.6
#1  0x00007f5e376fa670 in _IO_file_underflow () from /root/test/libc.so.6
#2  0x00007f5e376fb7b2 in _IO_default_uflow () from /root/test/libc.so.6
#3  0x00007f5e37863110 in ?? ()
#4  0x00007f5e37863290 in ?? ()
#5  0x00007f5e37863160 in ?? ()
#6  0x00007fff1fcf2578 in ?? ()
#7  0x0000000000000000 in ?? ()

(gdb) add-symbol-file /root/test/test 0x7f5e37862000+0x0010c0
add symbol table from file "/root/test/test" at
        .text_addr = 0x7f5e378630c0
(y or n) y
Reading symbols from /root/test/test...

(gdb) bt
#0  0x00007f5e37768461 in read () from /root/test/libc.so.6
#1  0x00007f5e376fa670 in _IO_file_underflow () from /root/test/libc.so.6
#2  0x00007f5e376fb7b2 in _IO_default_uflow () from /root/test/libc.so.6
#3  0x00007f5e37863110 in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#4  main () at test.c:24
#5  0x00007f5e376a209b in __libc_start_main () from /root/test/libc.so.6
#6  0x00007f5e3786318a in _start ()

(gdb) q
A debugging session is active.

        Inferior 1 [process 16873] will be detached.

Quit anyway? (y or n) y
Detaching from program: /root/test/ld-linux-x86-64.so.2, process 16873
[Inferior 1 (process 16873) detached]

[1]+  Stopped(SIGTTIN)        /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test

所以我必须回复 Y 到文件不匹配的请求,然后 add-symbol-file 用真正的可执行文件的路径后跟地址(base + offset ) 使用 /proc/pid/maps(第一个结果)和 readelf.

检索

but none worked...

add-symbol-file ... 解决方案 应该 有效。我怀疑您没有提供正确的 .text 地址。

cat /proc/30622/maps | grep "r-xp" | grep "/root/test/test"

这假定 /root/test/test 的第一段具有 RX 权限。

过去是这种情况,但在现代系统中不再是这种情况(参见 )。

您没有提供 readelf -Wl /root/test/test 的输出,但我敢打赌它看起来类似于其他答案中的 4 段示例(第一个 LOAD 段有 R只读权限。

一般需要在内存中找到test可执行文件第一个LOAD段的地址,将.text的地址加到基址。

更新:

根据 /proc/$pid/mapsreadelf 新提供的输出,我们可以看出我的猜测是正确的:这个二进制文件有 4 个 LOAD 段,第一个没有'没有 r-x 权限。

则计算结果为:$address_of_the_first_PT_LOAD + $address_of_.text。即(对于16873进程):

(gdb) add-symbol-file /root/test/test 0x7f5e37862000+0x10c0

objdump -s --section=".text" /root/test/test | grep Contents -A 1 ...

这不必要地令人费解。更简单的方法:

readelf -WS /root/test/test | grep '\.text ' | nawk '{print "0x" }'

更新:

I need to run gdb in the test machine (which doesn't have nawk and other bells

这里的部分困难在于您有一个 PIE 二进制文件(在 运行 时间重新定位)。

如果问题在非 PIE 二进制文件(使用 -fno-pie -no-pie 构建)中重现,则地址可以计算一次(例如在开发机器上)并在测试机器上重复使用。每次 运行 二进制文件时,您都不需要脚本来计算地址。


If I compile the binary to have the interpreter path hardcoded

通常这是最好的方法。为什么不用呢?

如果您希望二进制文件在开发和测试机器上都运行,请在两者上都提供 /root/test 的副本。