Cassandra Node.js 更新计数器时驱动程序内存泄漏

Question

正在尝试迁移计数器 tables。从源 table 读取每一行并将结果写入新的 table。当用 js 写入数百万行时，堆内存不足运行。同步代码用于确保在尝试写入另一行之前写入行。确定观察堆的问题似乎是在使用 cassandra-driver 执行 UPDATE 查询时堆上升。

const query = `UPDATE keyspace.table_name
  SET counter1 = counter1 + ${x1}, counter2 = counter2 + ${x2}
  WHERE partition_key = ? AND clustering_key = ?`;

await cassandraDestination.execute(query, params, { prepare: true });

这种情况下如何避免内存泄漏？


==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1317672cfc5d]
Security context: 0x192500f1d971 <JSObject>
    1: borrowNextConnection [0x2ea411f96091] [/home/asen.chekov/projects/stats/node_modules/cassandra-driver/lib/request-handler.js:~64] [pc=0x131767db1395](this=0x2ea411f95da1 <JSFunction RequestHandler (sfi = 0x2e8feb1ef231)>,0x21125933f379 <Object map = 0x1ddd59acbdb1>,0x21125933f439 <Object map = 0x1ddd59a82521>,0x110a33490991 <ProfileManager map...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191016.122316.23174.001.json
Node.js report completed
 1: 0x953b10 node::Abort() [node]
 2: 0x9547f4 node::OnFatalError(char const*, char const*) [node]
 3: 0xb32bee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb32e24 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf32452  [node]
 6: 0xf32558 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xf3ec78 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xf3f78b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf424c1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xf0c6f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x11c2b3e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x1317672cfc5d 
Aborted (core dumped)```

Answer 1

请注意，execute() 方法的方法签名是：

execute(query: string, params: Array, options: object)

此外，您不应在 prepare statements 中对参数进行硬编码。

正确的用法是：

const query = `UPDATE keyspace.table_name
  SET counter = counter + 1, counter1 = counter1 + 1
  WHERE partition_key = ? AND clustering_key = ?`;

await cassandraDestination.execute(query, [ key1, key2 ], { prepare: true });

Answer 2

这解决了问题。

正确的查询是：

const query = `UPDATE keyspace.table_name
  SET counter1 = counter1 + ?, counter2 = counter2 + ?
  WHERE partition_key = ? AND clustering_key = ?`;

await cassandraDestination
  .execute(query, [ counter1, counter2, key1, key2 ], {
    prepare: true,
  });

Cassandra Node.js 更新计数器时驱动程序内存泄漏

Cassandra Node.js driver memory leak when UPDATE counter

javascript

cql

cassandra

node.js

cassandra-driver