boost::interprocess::managed_mapped_file 字符串双端队列用完 space
boost::interprocess::managed_mapped_file deque of strings runs out of space
我正在使用 boost::interprocess::deque 和 memory_mapped_file 作为文件缓冲区，让数据在重启等情况下仍然存在。
我正在这样创建缓冲区：
typedef boost::interprocess::allocator<char, boost::interprocess::managed_mapped_file::segment_manager> char_allocator_type;
typedef boost::interprocess::basic_string<char, std::char_traits<char>, char_allocator_type> persisted_string_type;
typedef boost::interprocess::allocator<persisted_string_type, boost::interprocess::managed_mapped_file::segment_manager> persisted_string_allocator_type;
typedef boost::interprocess::deque<persisted_string_type , persisted_string_allocator_type> deque_buf;
auto* mmf = new boost::interprocess::managed_mapped_file(boost::interprocess::open_or_create, "file_name", 1000000);
auto* buffer = mmf.find_or_construct<deque_buf>(boost::interprocess::unique_instance)(mmf_->get_segment_manager());
我这样写到缓冲区的后面：
try {
char_allocator_type ca(mmf->get_segment_manager());
persisted_string_type persisted_string(ca);
persisted_string = "some string";
buffer->push_back(persisted_string);
} catch (const boost::interprocess::bad_alloc &e) {
//buffer full, handle it
}
我像这样从缓冲区的前面擦除：
buffer->pop_front();
我像队列一样使用它。因此，每当我无法向背面添加新数据时，我都会从前面擦除数据，直到可以添加新数据为止。
但是，随着它的运行，我必须从前面擦除越来越多的数据，以便在后面添加新数据。最终双端队列几乎不能包含任何元素。增加映射文件大小只会推迟问题。
我做错了什么？
Rgds
克劳斯
首先：为您的代码添加一些平静和安静:)
#include <boost/container/scoped_allocator.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/deque.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
namespace bip = boost::interprocess;
namespace bc = boost::container;
using Segment = bip::managed_mapped_file;
using Mgr = Segment::segment_manager;
template <typename T>
using Alloc = bc::scoped_allocator_adaptor<bip::allocator<T, Mgr>>;
using String =
bip::basic_string<char, std::char_traits<char>, Alloc<char>>;
template <typename T> using Deque = bip::deque<T, Alloc<T>>;
这有助于表达代码。 scoped_allocator_adaptor 减少了您明确（低效地）在分配器实例周围改组的次数：
int main() {
using Buffers = Deque<String>;
bip::managed_mapped_file mmf(bip::open_or_create, "file_name", 1 << 20); // 1 MiB
auto& buffer = *mmf.find_or_construct<Buffers>(bip::unique_instance)(
mmf.get_segment_manager());
try {
buffer.emplace_back("some string");
} catch (const boost::interprocess::bad_alloc &e) {
//buffer full, handle it
}
}
问题
问题可能是碎片化。这里没有真正的解决方案，除非您有能力定期重新填充共享内存段。
另一部分是段管理器开销：
- How much memory should 'managed_shared_memory' allocate? (boost)
- 和我自己的答案根据 containers/allocation 模式衡量不同的开销：Bad alloc is thrown This answer also illustrates the effect of fragmentation:
需要考虑的一件事是为字符串使用一个池。通过使用固定大小的分配器，您将在很大程度上避免碎片化问题。
还要记住使用 reserve() 和 shrink_to_fit() 来指导如何管理实际分配模式的实施。
循环缓冲区
但与您的用例更直接匹配的似乎是 ring buffer。我会画出类似的东西：
using Buffer = boost::container::small_vector<char, 100, Alloc<char>>;
template <typename T> using Ring = boost::circular_buffer<T, Alloc<T>>;
如果您事先知道缓冲区的最大大小，您可以使用 static_vector 代替：
using Buffer = boost::container::static_vector<char, 100>;
静态向量从不分配，所以它甚至不需要分配器。
circular_buffer 与作用域分配器的工作效果不尽如人意，因此使用起来有点痛苦，但可能仍然比手动分配器洗牌更优雅。
#include <boost/circular_buffer.hpp>
#include <boost/container/small_vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
namespace bip = boost::interprocess;
namespace bc = boost::container;
using Segment = bip::managed_mapped_file;
using Mgr = Segment::segment_manager;
template <typename T>
using Alloc = bip::allocator<T, Mgr>;
using Buffer = boost::container::small_vector<char, 100, Alloc<char>>;
using BufferAlloc = Buffer::allocator_type;
template <typename T> using Ring = boost::circular_buffer<T, Alloc<T>>;
static inline std::string_view as_sv(Buffer const& b) {
return {b.data(), b.size()};
}
#include <iomanip>
#include <iostream>
#include <ranges>
namespace v = std::ranges::views;
using std::ranges::subrange;
int main()
{
using Buffers = Ring<Buffer>;
bip::managed_mapped_file mmf(bip::open_or_create, "file_name", 1 << 20); // 1 MiB
auto& sequence = *mmf.find_or_construct<size_t>(bip::unique_instance)(0);
auto& buffers = *mmf.find_or_construct<Buffers>(bip::unique_instance)(
1000, // 1000 buffers capacity
mmf.get_segment_manager());
auto to_buffer =
[a = BufferAlloc(buffers.get_allocator())](std::string_view str) {
return Buffer(str.begin(), str.end(), a);
};
int64_t num = buffers.size(); // signed!
std::cout << "Current size: " << buffers.size() << " ...";
auto last5 = subrange(buffers) | v::drop(std::max(0l, num - 5));
for (auto b : last5 | v::transform(as_sv)) {
std::cout << " " << std::quoted(b);
}
std::cout << std::endl;
try {
buffers.push_back(to_buffer("some string #" + std::to_string(++sequence)));
} catch (const boost::interprocess::bad_alloc& e) {
// buffer full, handle it
std::cerr << e.what() << "\n";
return 1;
}
}
请注意这如何在您不需要任何操作的情况下已经具有 pop_front 行为：
rm file_name; for a in {1..10000}; do if ./sotest; then true; else break; fi; done | nl
版画
1 Current size: 0 ...
2 Current size: 1 ... "some string #1"
3 Current size: 2 ... "some string #1" "some string #2"
4 Current size: 3 ... "some string #1" "some string #2" "some string #3"
5 Current size: 4 ... "some string #1" "some string #2" "some string #3" "some string #4"
6 Current size: 5 ... "some string #1" "some string #2" "some string #3" "some string #4" "some string #5"
7 Current size: 6 ... "some string #2" "some string #3" "some string #4" "some string #5" "some string #6"
8 Current size: 7 ... "some string #3" "some string #4" "some string #5" "some string #6" "some string #7"
9 Current size: 8 ... "some string #4" "some string #5" "some string #6" "some string #7" "some string #8"
10 Current size: 9 ... "some string #5" "some string #6" "some string #7" "some string #8" "some string #9"
11 Current size: 10 ... "some string #6" "some string #7" "some string #8" "some string #9" "some string #10"
12 Current size: 11 ... "some string #7" "some string #8" "some string #9" "some string #10" "some string #11"
13 Current size: 12 ... "some string #8" "some string #9" "some string #10" "some string #11" "some string #12"
14 Current size: 13 ... "some string #9" "some string #10" "some string #11" "some string #12" "some string #13"
...
998 Current size: 997 ... "some string #993" "some string #994" "some string #995" "some string #996" "some string #997"
999 Current size: 998 ... "some string #994" "some string #995" "some string #996" "some string #997" "some string #998"
1000 Current size: 999 ... "some string #995" "some string #996" "some string #997" "some string #998" "some string #999"
1001 Current size: 1000 ... "some string #996" "some string #997" "some string #998" "some string #999" "some string #1000"
1002 Current size: 1000 ... "some string #997" "some string #998" "some string #999" "some string #1000" "some string #1001"
1003 Current size: 1000 ... "some string #998" "some string #999" "some string #1000" "some string #1001" "some string #1002"
1004 Current size: 1000 ... "some string #999" "some string #1000" "some string #1001" "some string #1002" "some string #1003"
1005 Current size: 1000 ... "some string #1000" "some string #1001" "some string #1002" "some string #1003" "some string #1004"
1006 Current size: 1000 ... "some string #1001" "some string #1002" "some string #1003" "some string #1004" "some string #1005"
1007 Current size: 1000 ... "some string #1002" "some string #1003" "some string #1004" "some string #1005" "some string #1006"
...
9998 Current size: 1000 ... "some string #9993" "some string #9994" "some string #9995" "some string #9996" "some string #9997"
9999 Current size: 1000 ... "some string #9994" "some string #9995" "some string #9996" "some string #9997" "some string #9998"
10000 Current size: 1000 ... "some string #9995" "some string #9996" "some string #9997" "some string #9998" "some string #9999"
我正在使用 boost::interprocess::deque 和 memory_mapped_file 作为文件缓冲区，让数据在重启等情况下仍然存在。
我正在这样创建缓冲区：
typedef boost::interprocess::allocator<char, boost::interprocess::managed_mapped_file::segment_manager> char_allocator_type;
typedef boost::interprocess::basic_string<char, std::char_traits<char>, char_allocator_type> persisted_string_type;
typedef boost::interprocess::allocator<persisted_string_type, boost::interprocess::managed_mapped_file::segment_manager> persisted_string_allocator_type;
typedef boost::interprocess::deque<persisted_string_type , persisted_string_allocator_type> deque_buf;
auto* mmf = new boost::interprocess::managed_mapped_file(boost::interprocess::open_or_create, "file_name", 1000000);
auto* buffer = mmf.find_or_construct<deque_buf>(boost::interprocess::unique_instance)(mmf_->get_segment_manager());
我这样写到缓冲区的后面：
try {
char_allocator_type ca(mmf->get_segment_manager());
persisted_string_type persisted_string(ca);
persisted_string = "some string";
buffer->push_back(persisted_string);
} catch (const boost::interprocess::bad_alloc &e) {
//buffer full, handle it
}
我像这样从缓冲区的前面擦除：
buffer->pop_front();
我像队列一样使用它。因此，每当我无法向背面添加新数据时，我都会从前面擦除数据，直到可以添加新数据为止。 但是，随着它的运行，我必须从前面擦除越来越多的数据，以便在后面添加新数据。最终双端队列几乎不能包含任何元素。增加映射文件大小只会推迟问题。
我做错了什么？
Rgds 克劳斯
首先：为您的代码添加一些平静和安静:)
#include <boost/container/scoped_allocator.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/deque.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
namespace bip = boost::interprocess;
namespace bc = boost::container;
using Segment = bip::managed_mapped_file;
using Mgr = Segment::segment_manager;
template <typename T>
using Alloc = bc::scoped_allocator_adaptor<bip::allocator<T, Mgr>>;
using String =
bip::basic_string<char, std::char_traits<char>, Alloc<char>>;
template <typename T> using Deque = bip::deque<T, Alloc<T>>;
这有助于表达代码。 scoped_allocator_adaptor 减少了您明确（低效地）在分配器实例周围改组的次数：
int main() {
using Buffers = Deque<String>;
bip::managed_mapped_file mmf(bip::open_or_create, "file_name", 1 << 20); // 1 MiB
auto& buffer = *mmf.find_or_construct<Buffers>(bip::unique_instance)(
mmf.get_segment_manager());
try {
buffer.emplace_back("some string");
} catch (const boost::interprocess::bad_alloc &e) {
//buffer full, handle it
}
}
问题
问题可能是碎片化。这里没有真正的解决方案，除非您有能力定期重新填充共享内存段。
另一部分是段管理器开销：
- How much memory should 'managed_shared_memory' allocate? (boost)
- 和我自己的答案根据 containers/allocation 模式衡量不同的开销：Bad alloc is thrown This answer also illustrates the effect of fragmentation:
需要考虑的一件事是为字符串使用一个池。通过使用固定大小的分配器，您将在很大程度上避免碎片化问题。
还要记住使用 reserve() 和 shrink_to_fit() 来指导如何管理实际分配模式的实施。
循环缓冲区
但与您的用例更直接匹配的似乎是 ring buffer。我会画出类似的东西：
using Buffer = boost::container::small_vector<char, 100, Alloc<char>>;
template <typename T> using Ring = boost::circular_buffer<T, Alloc<T>>;
如果您事先知道缓冲区的最大大小，您可以使用 static_vector 代替：
using Buffer = boost::container::static_vector<char, 100>;
静态向量从不分配，所以它甚至不需要分配器。
circular_buffer 与作用域分配器的工作效果不尽如人意，因此使用起来有点痛苦，但可能仍然比手动分配器洗牌更优雅。
#include <boost/circular_buffer.hpp>
#include <boost/container/small_vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
namespace bip = boost::interprocess;
namespace bc = boost::container;
using Segment = bip::managed_mapped_file;
using Mgr = Segment::segment_manager;
template <typename T>
using Alloc = bip::allocator<T, Mgr>;
using Buffer = boost::container::small_vector<char, 100, Alloc<char>>;
using BufferAlloc = Buffer::allocator_type;
template <typename T> using Ring = boost::circular_buffer<T, Alloc<T>>;
static inline std::string_view as_sv(Buffer const& b) {
return {b.data(), b.size()};
}
#include <iomanip>
#include <iostream>
#include <ranges>
namespace v = std::ranges::views;
using std::ranges::subrange;
int main()
{
using Buffers = Ring<Buffer>;
bip::managed_mapped_file mmf(bip::open_or_create, "file_name", 1 << 20); // 1 MiB
auto& sequence = *mmf.find_or_construct<size_t>(bip::unique_instance)(0);
auto& buffers = *mmf.find_or_construct<Buffers>(bip::unique_instance)(
1000, // 1000 buffers capacity
mmf.get_segment_manager());
auto to_buffer =
[a = BufferAlloc(buffers.get_allocator())](std::string_view str) {
return Buffer(str.begin(), str.end(), a);
};
int64_t num = buffers.size(); // signed!
std::cout << "Current size: " << buffers.size() << " ...";
auto last5 = subrange(buffers) | v::drop(std::max(0l, num - 5));
for (auto b : last5 | v::transform(as_sv)) {
std::cout << " " << std::quoted(b);
}
std::cout << std::endl;
try {
buffers.push_back(to_buffer("some string #" + std::to_string(++sequence)));
} catch (const boost::interprocess::bad_alloc& e) {
// buffer full, handle it
std::cerr << e.what() << "\n";
return 1;
}
}
请注意这如何在您不需要任何操作的情况下已经具有 pop_front 行为：
rm file_name; for a in {1..10000}; do if ./sotest; then true; else break; fi; done | nl
版画
1 Current size: 0 ...
2 Current size: 1 ... "some string #1"
3 Current size: 2 ... "some string #1" "some string #2"
4 Current size: 3 ... "some string #1" "some string #2" "some string #3"
5 Current size: 4 ... "some string #1" "some string #2" "some string #3" "some string #4"
6 Current size: 5 ... "some string #1" "some string #2" "some string #3" "some string #4" "some string #5"
7 Current size: 6 ... "some string #2" "some string #3" "some string #4" "some string #5" "some string #6"
8 Current size: 7 ... "some string #3" "some string #4" "some string #5" "some string #6" "some string #7"
9 Current size: 8 ... "some string #4" "some string #5" "some string #6" "some string #7" "some string #8"
10 Current size: 9 ... "some string #5" "some string #6" "some string #7" "some string #8" "some string #9"
11 Current size: 10 ... "some string #6" "some string #7" "some string #8" "some string #9" "some string #10"
12 Current size: 11 ... "some string #7" "some string #8" "some string #9" "some string #10" "some string #11"
13 Current size: 12 ... "some string #8" "some string #9" "some string #10" "some string #11" "some string #12"
14 Current size: 13 ... "some string #9" "some string #10" "some string #11" "some string #12" "some string #13"
...
998 Current size: 997 ... "some string #993" "some string #994" "some string #995" "some string #996" "some string #997"
999 Current size: 998 ... "some string #994" "some string #995" "some string #996" "some string #997" "some string #998"
1000 Current size: 999 ... "some string #995" "some string #996" "some string #997" "some string #998" "some string #999"
1001 Current size: 1000 ... "some string #996" "some string #997" "some string #998" "some string #999" "some string #1000"
1002 Current size: 1000 ... "some string #997" "some string #998" "some string #999" "some string #1000" "some string #1001"
1003 Current size: 1000 ... "some string #998" "some string #999" "some string #1000" "some string #1001" "some string #1002"
1004 Current size: 1000 ... "some string #999" "some string #1000" "some string #1001" "some string #1002" "some string #1003"
1005 Current size: 1000 ... "some string #1000" "some string #1001" "some string #1002" "some string #1003" "some string #1004"
1006 Current size: 1000 ... "some string #1001" "some string #1002" "some string #1003" "some string #1004" "some string #1005"
1007 Current size: 1000 ... "some string #1002" "some string #1003" "some string #1004" "some string #1005" "some string #1006"
...
9998 Current size: 1000 ... "some string #9993" "some string #9994" "some string #9995" "some string #9996" "some string #9997"
9999 Current size: 1000 ... "some string #9994" "some string #9995" "some string #9996" "some string #9997" "some string #9998"
10000 Current size: 1000 ... "some string #9995" "some string #9996" "some string #9997" "some string #9998" "some string #9999"