是否可以在 linux 中分配大量虚拟内存？

Question

出于某些目的，分配大量虚拟 space 和仅访问的页面中的页面会很有效。分配大量内存是瞬时的，并不实际抢页：

char* p = new char[1024*1024*1024*256];

好的，上面指出的是错误的，因为它是一个 32 位数字。

我预计 new 正在调用调用 sbrk 的 malloc，并且当我访问超出起始位置 4Gb 的位置时，它会尝试将任务内存扩展那么多？

完整程序如下：

#include <cstdint>
int main() {
  constexpr uint64_t GB = 1ULL << 30;
  char* p = new char[256*GB]; // allocate large block of virtual space
  p[0] = 1;
  p[1000000000] = 1;
  p[2000000000] = 1;
}

现在，我在尝试分配大量内存时得到 bad_alloc，显然 malloc 将无法工作。

我的印象是 mmap 会映射到文件，但由于有人建议我正在研究它。

好的，mmap 似乎支持大面积虚拟内存的分配，但它需要一个文件描述符。创建巨大的内存数据结构可能是一个胜利，但如果它们必须由文件支持则不是：

尽管我不喜欢附加到文件的想法，但以下代码使用了 mmap。我不知道在虚拟内存中请求什么数字，并选择了 0x800000000。 mmap returns -1，显然我做错了什么：

#include <cstdint>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>

int main() {
  constexpr uint64_t GB = 1ULL << 30;
  void *addr = (void*)0x8000000000ULL;
  int fd = creat("garbagefile.dat", 0660);
  char* p = (char*)mmap(addr, 256*GB, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
  p[0] = 1;
  p[1000000000] = 1;
  p[2000000000] = 1;
  close(fd);
}

有没有办法分配一大块虚拟内存并稀疏地访问页面，或者这是不可行的？

Answer 1

256*GB 的值不在 32 位整数类型的范围内。尝试 uint64_t 作为 GB 的一种类型：

constexpr uint64_t GB = 1024*1024*1024;

或者，强制 64 位乘法：

char* p = new char[256ULL * GB];

OT：我更喜欢 GB 的定义：

constexpr uint64_t GB = 1ULL << 30;

关于虚拟内存限制，见this answer。

Answer 2

Is it possible to allocate large amount of virtual memory in linux?

可能吧。但您可能需要配置它才能被允许：

The Linux kernel supports the following overcommit handling modes

0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.

1 - Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages.

2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.

Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.

The overcommit policy is set via the sysctl `vm.overcommit_memory'.

因此，如果您想分配比物理内存更多的虚拟内存，那么您需要：

# in shell
sysctl -w vm.overcommit_memory=1

RLIMIT_AS The maximum size of the process's virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

所以，您需要：

setrlimit(RLIMIT_AS, {
    .rlim_cur = RLIM_INFINITY,
    .rlim_max = RLIM_INFINITY,
});

或者，如果您不能授予进程执行此操作的权限，那么您可以在 /etc/security/limits.conf 中持久配置它，这将影响所有进程（user/group）。

Ok, so mmap seems to support ... but it requires a file descriptor. ... could be a win but not if they have to be backed by a file ... I don't like the idea of attaching to a file

您不需要使用文件支持的 mmap。有 MAP_ANONYMOUS。

I did not know what number to put in to request

然后使用null。示例：

mmap(nullptr, 256*GB, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)

就是说，如果您按照描述配置了系统，那么 new 应该和 mmap 一样工作。它可能会使用 malloc，这可能会使用 mmap 来进行这样的大分配。

额外提示：您可能会受益于使用 HugeTLB Pages。

是否可以在 linux 中分配大量虚拟内存？

Is it possible to allocate large amount of virtual memory in linux?

c++

linux

memory-management

virtual-memory