Boost::asio 和 boost::bind:永远不会释放函子内存

Boost::asio and boost::bind: Functor memory is never released

我的代码正在分配内存并且从不释放它,即使它应该(至少在我看来)。

header 看起来像这样:

typedef boost::asio::ssl::stream<boost::asio::ip::tcp::socket> sslSocket_t;

class Object {
    boost::asio::io_service ioService_;
    boost::asio::ip::tcp::acceptor acceptor_;
    boost::asio::ssl::context context_;

    void functionOne();
    void functionTwo(shared_ptr<sslSocket_t>& sslSocket, const boost::system::error_code& error)
}

我的来源是这样的:

void Object::functionOne() {
    for (int i = 0; i < 10; i++) {
        shared_ptr<sslSocket_t> sslSocket(new sslSocket_t(ioService_, context_));
        acceptor_.async_accept(sslSocket->lowest_layer(),
                       boost::bind(&Object::functionTwo, this, sslSocket, boost::asio::placeholders::error));
    }
    acceptor_.cancel();

    boost::asio::io_service::work work(ioService_);
    ioService_.run();
}

void functionTwo(shared_ptr<sslSocket_t>& sslSocket, const boost::system::error_code& err) {
    // Do nothing
}

所以当我调用 Object.functionOne() 时,内存被分配给 Object.ioService_ object,为了能够调用绑定的异步方法。然后在循环之后,接受器上所有未决的异步操作都被取消。一旦 Object.ioService_.运行() 被调用(我一直在测试),相应的处理程序就会被调用。但是由于某种原因,分配的内存没有被释放。那么有人可以解释一下,为什么内存没有被释放并给我一个如何释放它的提示吗?

顺便说一句:我正在研究 debian 并查看 /proc/self/status -> VmRSS 以查看已用内存.

@Vinnie Falco

#include <boost/asio.hpp>
#include <boost/asio/ssl.hpp>
#include <boost/bind.hpp>

#include <iostream>
#include <memory>

typedef boost::asio::ssl::stream<boost::asio::ip::tcp::socket> sslSocket_t;

using namespace std;

struct T  {

    boost::asio::io_service ioService_;
    boost::asio::ip::tcp::acceptor acceptor_;
    boost::asio::ssl::context context_;

    void functionOne() {
        for (int i = 0; i < 10; i++) {
            shared_ptr<sslSocket_t> sslSocket(new sslSocket_t(ioService_, context_));
            acceptor_.async_accept(sslSocket->lowest_layer(),
                               boost::bind(&T::functionTwo, this, sslSocket, boost::asio::placeholders::error));
        }
        acceptor_.cancel();

        boost::asio::io_service::work work(ioService_);
        ioService_.run();
    }

    void functionTwo(shared_ptr<sslSocket_t>& sslSocket, const boost::system::error_code& err) {
        // Do nothing
    }

    T() : acceptor_(ioService_,
                    boost::asio::ip::tcp::endpoint(boost::asio::ip::tcp::v4(), 443)),
          context_(boost::asio::ssl::context::sslv23_server)  {

    }

    ~T()  {

    }
};

int main()  {

    try  {
        T t;

        t.functionOne();
    } catch (std::exception& e) {
        cout << "Exception: " << e.what() << endl;
  }
}

我的问题不是,是否以及为什么调用 T 的析构函数,这按预期工作。但是关于已用内存的行为很奇怪。 因此,如果您增加 for 循环中的限制,您将观察到程序保留了大量内存,即使它应该在所有异步处理程序被调用后释放。但是 sslSocket objects 没有被释放,这就是我的问题:为什么内存(特别是为 sslSocket 分配的内存)绑定到函子functionTwo,未释放,即使在异步方法 fucntionTwo 已被调用且未引用 sslSocket 之后也是如此还剩下吗?

我解释我的担忧的最终方法(4 月 28 日编辑)

好吧,我做了一个运行可行的例子,这表明我的担忧: My Problem in an example

输出:

Before leaking call:     6984 kB
Asynchronous calls of functionTwo: 10000
Memory while ioService is still running:   460244 kB
Memory after ioService is stopped:   460244 kB

更疯狂的是,在我自己的本地实现中,我得到以下输出:

Memory leaking call:     8352 kB
Asynchronous calls of functionTwo: 10000
Memory while ioService is still running:   471932 kB
Memory after ioService is stopped:     8436 kB

因此可以清楚地看到:即使调用了所有异步操作,内存也没有释放。

总结和理解(?)行为(最后编辑)

你们中的一些人可能误解了,我不认为我的代码中存在某种漏洞。我在我的代码示例中将该结构命名为 Leak,这可能让您感到困惑,但我的问题不是在我的示例中是否以及在何处发生内存泄漏。它是关于与 ioService object 结合的内存分配。首先我认为,声称的内存正在无限增加。我采用了最后一种方法来理解这种行为并得出结论,内存管理很好。 OS 没有回收内存,但程序的内存分配收敛到一个限制,这对我来说很好。所以这个问题不适合我。

Example: Converging memory consumption

最让我不安的是我的本地实施表现出略有不同的行为。当 ioService object 完成其作业并重置时,OS 回收了内存,这满足了我的期望。

因此总结所有观察结果:

分配的内存由 C++ 运行时和 OS 管理。直接观察分配过程是相当困难的(如果不是不可能的话?),因为它被优化以减少对新内存页面的请求量,这意味着分配和 freed 内存可能不会由 OS.

立即重新分配

为了向我指出这种行为的关键点,我想描述一下我的程序的用法:我正在开发一个服务器应用程序,这意味着该程序应该 运行 无限量的时间。如果程序在某个时间申请了大量的峰值内存,那完全没问题,但它需要在 运行 时间的某个时间点释放申请的内存,而不是在 运行 时间之后。所以对我来说,只剩下一个问题了:

OS 是否会在某个时候回收已申请(但未使用)的内存?还是我必须在 运行 时间内自己管理内存(使用 newdelete)?

我不确定问题出在哪里,但我认为你做错了什么。你能提供一个独立的例子来展示这个问题吗?本示例程序编译运行,调用析构函数:

#include <boost/asio.hpp>
#include <functional>
#include <iostream>
#include <memory>

struct T
{
    T()
    {
        std::cerr << "T::T()\n";
    }

    ~T()
    {
        std::cerr << "T::~T()\n";
    }
};

void f(std::shared_ptr<T>&)
{
}

int main()
{
    using namespace boost::asio;
    io_service ios;
    ios.post(std::bind(&f, std::make_shared<T>()));
    ios.run();
}

你可以在这里看到输出:http://melpon.org/wandbox/permlink/0fkIAnoMXDOeedx7

输出:

T::T()
T::~T()

我认为你的方法有问题。你永远不应该将异步操作与 asio 交织在一起。如果您这样做,将会发生各种未定义的废话。你通常实现异步接受的方式如下:

void do_accept() {
  shared_ptr<sslSocket_t> socket(new sslSocket_t(ioService_, context_));
  // Queue an async accept operation
  acceptor_.async_accept(socket->lowest_layer(), [this, socket](auto ec) {
    if (!ec) {
      // Handle the socket
    }
    // If not shutting down
    this->do_accept(); // next accept
  });
}

以您的独立示例为例,运行 它在 valgrind 下表明 没有任何内容 被泄露

==14098== 
==14098== HEAP SUMMARY:
==14098==     in use at exit: 73,696 bytes in 7 blocks
==14098==   total heap usage: 163,524 allocs, 163,517 frees, 733,133,505 bytes allocated
==14098== 
==14098== LEAK SUMMARY:
==14098==    definitely lost: 0 bytes in 0 blocks
==14098==    indirectly lost: 0 bytes in 0 blocks
==14098==      possibly lost: 0 bytes in 0 blocks
==14098==    still reachable: 73,696 bytes in 7 blocks
==14098==         suppressed: 0 bytes in 0 blocks
==14098== Rerun with --leak-check=full to see details of leaked memory

如果您提供 valgrind --show-leak-kinds=all --leak-check=full ./test,您会发现 "leaked"(剩余可访问对象)是从 libssl/libcrypto 分配的常见静态内容。即使您只进行 1 次迭代,它们也会被分配。 10000 次迭代没有变化。

==14214== Memcheck, a memory error detector
==14214== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==14214== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==14214== Command: ./test 10000
==14214== 
Before leaking call:    50056 kB
Asynchronous calls of functionTwo: 10000
Memory while ioService is still running:   265592 kB
Memory after ioService is stopped:   265592 kB
==14214== 
==14214== HEAP SUMMARY:
==14214==     in use at exit: 73,696 bytes in 7 blocks
==14214==   total heap usage: 163,524 allocs, 163,517 frees, 733,133,505 bytes allocated
==14214== 
==14214== 24 bytes in 1 blocks are still reachable in loss record 1 of 7
==14214==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14214==    by 0x5307E77: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53BF315: lh_insert (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C1863: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C245D: ERR_get_state (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C25EE: ERR_clear_error (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x40C9CA: context (context.ipp:70)
==14214==    by 0x40C9CA: Leak::Leak() (test.cpp:77)
==14214==    by 0x403E13: main (test.cpp:86)
==14214== 
==14214== 32 bytes in 1 blocks are still reachable in loss record 2 of 7
==14214==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14214==    by 0x5307E77: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53BE7AE: sk_new (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x507FD69: ??? (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
==14214==    by 0x5081E68: SSL_COMP_get_compression_methods (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
==14214==    by 0x5087532: SSL_library_init (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
==14214==    by 0x40B9A8: do_init (openssl_init.ipp:39)
==14214==    by 0x40B9A8: boost::asio::ssl::detail::openssl_init_base::instance() (openssl_init.ipp:131)
==14214==    by 0x403C3C: openssl_init (openssl_init.hpp:60)
==14214==    by 0x403C3C: __static_initialization_and_destruction_0 (openssl_init.hpp:90)
==14214==    by 0x403C3C: _GLOBAL__sub_I_count (test.cpp:96)
==14214==    by 0x40FE1C: __libc_csu_init (in /home/sehe/Projects/Whosebug/test)
==14214==    by 0x5EC09CE: (below main) (libc-start.c:245)
==14214== 
==14214== 32 bytes in 1 blocks are still reachable in loss record 3 of 7
==14214==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14214==    by 0x5307E77: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53BE7CC: sk_new (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x507FD69: ??? (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
==14214==    by 0x5081E68: SSL_COMP_get_compression_methods (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
==14214==    by 0x5087532: SSL_library_init (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
==14214==    by 0x40B9A8: do_init (openssl_init.ipp:39)
==14214==    by 0x40B9A8: boost::asio::ssl::detail::openssl_init_base::instance() (openssl_init.ipp:131)
==14214==    by 0x403C3C: openssl_init (openssl_init.hpp:60)
==14214==    by 0x403C3C: __static_initialization_and_destruction_0 (openssl_init.hpp:90)
==14214==    by 0x403C3C: _GLOBAL__sub_I_count (test.cpp:96)
==14214==    by 0x40FE1C: __libc_csu_init (in /home/sehe/Projects/Whosebug/test)
==14214==    by 0x5EC09CE: (below main) (libc-start.c:245)
==14214== 
==14214== 128 bytes in 1 blocks are still reachable in loss record 4 of 7
==14214==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14214==    by 0x5307E77: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53BEFE1: lh_new (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C1512: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C182F: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C245D: ERR_get_state (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C25EE: ERR_clear_error (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x40C9CA: context (context.ipp:70)
==14214==    by 0x40C9CA: Leak::Leak() (test.cpp:77)
==14214==    by 0x403E13: main (test.cpp:86)
==14214== 
==14214== 176 bytes in 1 blocks are still reachable in loss record 5 of 7
==14214==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14214==    by 0x5307E77: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53BEFBF: lh_new (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C1512: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C182F: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C245D: ERR_get_state (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C25EE: ERR_clear_error (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x40C9CA: context (context.ipp:70)
==14214==    by 0x40C9CA: Leak::Leak() (test.cpp:77)
==14214==    by 0x403E13: main (test.cpp:86)
==14214== 
==14214== 600 bytes in 1 blocks are still reachable in loss record 6 of 7
==14214==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14214==    by 0x5307E77: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C23F5: ERR_get_state (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x53C25EE: ERR_clear_error (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==14214==    by 0x40C9CA: context (context.ipp:70)
==14214==    by 0x40C9CA: Leak::Leak() (test.cpp:77)
==14214==    by 0x403E13: main (test.cpp:86)
==14214== 
==14214== 72,704 bytes in 1 blocks are still reachable in loss record 7 of 7
==14214==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14214==    by 0x57731FF: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==14214==    by 0x4010609: call_init.part.0 (dl-init.c:72)
==14214==    by 0x401071A: call_init (dl-init.c:30)
==14214==    by 0x401071A: _dl_init (dl-init.c:120)
==14214==    by 0x4000D09: ??? (in /lib/x86_64-linux-gnu/ld-2.21.so)
==14214==    by 0x1: ???
==14214==    by 0xFFEFFFF76: ???
==14214==    by 0xFFEFFFF7D: ???
==14214== 
==14214== LEAK SUMMARY:
==14214==    definitely lost: 0 bytes in 0 blocks
==14214==    indirectly lost: 0 bytes in 0 blocks
==14214==      possibly lost: 0 bytes in 0 blocks
==14214==    still reachable: 73,696 bytes in 7 blocks
==14214==         suppressed: 0 bytes in 0 blocks
==14214== 
==14214== For counts of detected and suppressed errors, rerun with: -v
==14214== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

您测量内存使用情况的方法不正确。

我花了一段时间,但我终于自己解决了。

所以为了弄清楚事情,让我们确保了解我的问题的根源: 我正在开发一个服务器应用程序,它意味着 运行 无限长的时间。此应用程序必须能够处理大量并发传入连接。在某个时间点,负载可能会达到峰值,导致我的应用程序占用大量内存。然后过了一段时间,大部分传入的请求都被处理了,导致很多对象在 运行 时间内被释放。由于 OS 不需要内存(我的应用程序是服务器上唯一的大内存消耗者),所有释放的内存都保留在我的应用程序中,可以在另一个时间点重用。这对我来说绝对没问题,但一些客户和管理员可能会将大量不断申请的内存误解为内存泄漏应用程序。为了避免这种情况,我想手动将一些未使用的内存交还给 OS。在我的示例中,绑定到 ioService 的处理程序(例如接受新连接)将在 运行 时间内释放,但适当的内存不会被 [= =31=]。所以要手动执行此操作,我找到了以下解决方案:

释放C/C中Linux下未使用的堆内存++: int malloc_trim(size_t pad)

可以找到文档 here。简而言之,此方法将未使用的内存从堆中释放到 OS,这正是我一直在寻找的。我知道在内存优化方面,手动使用此功能可能很危险,但由于我只想每隔几分钟释放一次内存,所以这个性能问题对我来说是可以接受的。

感谢大家的努力和耐心!