Cassandra添加节点问题，失败；错误='Cannot allocate memory' (errno=12)
Cassandra add node problem, failed; error='Cannot allocate memory' (errno=12)
我有一个有 14 个节点的 cassandra (v3.11.4) 集群，我想添加一个新节点。机器有 256GB 内存，我将堆大小（最大和最小）设置为 64GB 。但是由于内存错误，我无法添加新节点！确切的问题是什么，我需要做什么？
最后一行日志如下：
INFO [CompactionExecutor:1153] 2020-06-11 10:51:09,719 NoSpamLogger.java:91 - Maximum memory usage reached (20971520000), cannot allocate chunk of 1048576
INFO [Service Thread] 2020-06-11 10:51:10,339 GCInspector.java:284 - G1 Young Generation GC in 239ms. G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 27209478712 -> 25011094408;
INFO [Service Thread] 2020-06-11 10:51:11,217 GCInspector.java:284 - G1 Young Generation GC in 456ms. G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 25011094408 -> 23790284880; G1 Survivor Space: 167772160 -> 268435456;
INFO [Service Thread] 2020-06-11 10:51:12,192 GCInspector.java:284 - G1 Young Generation GC in 486ms. G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 23790284880 -> 23936030504; G1 Survivor Space: 268435456 -> 301989888;
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
5829.747: G1IncCollectionPause [ 772 18 22 ] [ 0 0 1 1 118 ] 0
....
5872.925: G1IncCollectionPause [ 774 17 21 ] [ 0 0 0 1 172 ] 0
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
5876.138: G1IncCollectionPause [ 774 16 24 ] [ 0 0 1 1 148 ] 0
....
6075.988: G1IncCollectionPause [ 772 22 23 ] [ 0 0 1 1 170 ] 0
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
6079.968: G1IncCollectionPause [ 772 0 0 ] [ 0 0 1405 1 135 ] 0
6084.827: G1IncCollectionPause [ 772 0 1 ] [ 0 0 88 1 218 ] 0
.....
6218.542: no vm operation [ 773 13 21 ] [ 0 0 1 1 0 ] 0
6219.252: G1IncCollectionPause [ 773 19 19 ] [ 0 0 0 1 158 ] 0
6219.751: G1CollectFull [ 773 11 20 ] [ 0 1 1 1 75023 ] 0
WARN [Service Thread] 2020-06-11 10:52:32,407 GCInspector.java:282 - G1 Old Generation GC in 67243ms. G1 Eden Space: 2919235584 -> 0; G1 Old Gen: 24289926888 -> 19483266320; G1 Survivor Space: 671088640 -> 0;
INFO [Service Thread] 2020-06-11 10:52:32,408 StatusLogger.java:47 - Pool Name Active Pending Completed Blocked All Time Blocked
WARN [GossipTasks:1] 2020-06-11 10:52:32,506 Gossiper.java:783 - Gossip stage has 9 pending tasks; skipping status check (no nodes will be marked down)
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3160f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO [Service Thread] 2020-06-11 10:52:32,515 StatusLogger.java:51 - MutationStage 576 257 5688240 0 0
INFO [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ViewMutationStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ReadStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - RequestResponseStage 0 0 13 0 0
INFO [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - ReadRepairStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - CounterMutationStage 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d172f6000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243956360960 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - MiscStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - CompactionExecutor 33 33 1068 0 0
INFO [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - MemtableReclaimMemory 0 0 221 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - PendingRangeCalculator 0 0 16 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - GossipStage 0 0 18945 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31652000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691058432 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - HintsDispatcher 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31561000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e3269d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248690071296 also had an error]
[thread 140248708142848 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b35000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696182528 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MigrationStage 0 0 27 0 0
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MemtablePostFlush 1 1 304 0 0
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0 1 1 222 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - ValidationExecutor 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - Sampler 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - MemtableFlushWriter 1 1 184 0 0
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - InternalResponseStage 0 0 25 0 0
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - AntiEntropyStage 0 0 0 0 0
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - CacheCleanupExecutor 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e327a1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248709207808 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.INFO: os::commit_memory(0x00007f8e31af4000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248695916288 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e316de000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691631872 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,530 StatusLogger.java:61 - CompactionManager 33 35
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:73 - MessagingService n/a 0/0
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:83 - Cache Type Size Capacity KeysToSave
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:85 - KeyCache 3211792520 10485760000 all
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:91 - RowCache 0 0 all
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:98 - Table Memtable ops,data
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.parent_repair_history 0,0
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8ceedd1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243279877888 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.repair_history 60,5088
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.view_build_status 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.compaction_history 182,47068
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_aggregates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_triggers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.size_estimates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.paxos 0,0
# An error report file with more information is saved as:
# /home/dev/hs_err_pid11578.log
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.views_builds_in_progress 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.batches 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_keyspaces 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.sstable_activity 6713,123062
INFO [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.batchlog 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e100fb000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248131811072 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3150a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.schema_columns 0,0
[thread 140248689714944 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,534 StatusLogger.java:101 - system.hints 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.IndexInfo 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7caaf9e000, 262144, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248944940800 also had an error]INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_columnfamilies 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_functions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.built_views 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peer_events 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.range_xfers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.transferred_ranges 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.schema_usertypes 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.local 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.available_ranges 4,20360
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.prepared_statements 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.columns 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.types 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.indexes 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.keyspaces 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.dropped_columns 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.aggregates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.triggers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.tables 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.views 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.functions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxxxts 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxet 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxage 7883,699041
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxs 4824,213056
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxlist 705216,33023249
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxs 357,256470
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxxxxxe 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxmembers 6482,260169
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxid 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxcount 84961,1532062
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxe 25736,10620210
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31762000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248692172544 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxes 5511,4021957
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxs 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxd 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxe 1460,1127389
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxs 468,131122
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxd_users 2313247,95919246
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxs 76,36133
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx.users 2287211,201973918
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34df8000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749410048 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxt 16,1040
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxe 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxo 72693,16357850
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxd 26143,1492343
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.roles 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_members 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.resource_role_permissons_index 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_permissions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.sessions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.events 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31693000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691324672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e32599000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248707077888 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34e39000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749676288 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a07e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e315af000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248835942144 also had an error]
[thread 140248690390784 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8430b9f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140205730166528 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a72e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248842954496 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
INFO: os::commit_memory(0x00007f8e11541000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248153069312 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7803456000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140153427945216 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8b5f2c9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140236574201600 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318a9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693511936 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d15901000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243929143040 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318ea000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693778176 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31868000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693245696 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b79000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696461056 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31d40000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248698324736 also had an error]
[thread 140216167200512 also had an error][thread 140248204584704 also had an error][thread 140245662537472 also had an error][thread 140209602766592 also had an error][thread 140230211868416 also had an error]
[thread 140248151602944 also had an error][thread 140210503980800 also had an error][thread 140230032652032 also had an error][thread 140198818285312 also had an error][thread 140248755021568 also had an error][thread 140248765159168 also had an error][thread 140248224515840 also had an error][thread 140230234380032 also had an error][thread 140243956045568 also had an error][thread 140248768407296 also had an error][thread 140248294921984 also had an error][thread 140248768141056 also had an error][thread 140207186040576 also had an error]
[thread 140245662004992 also had an error][thread 140248585135872 also had an error]
可用内存量如下：
感谢您的帮助！
我在开始加入时也看到了这些警告：
INFO [main] 2020-06-11 09:08:21,598 CassandraDaemon.java:493 - JVM Arguments: [-Xloggc:./apache-cassandra-3.11.4/bin/../logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -Xms64G, -Xmx64G, -XX:+UseG1GC, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -XX:+PrintAdaptiveSizePolicy, -XX:+PrintGCTimeStamps, -XX:+PrintSafepointStatistics, -XX:+PrintClassHistogramBeforeFullGC, -XX:+PrintClassHistogramAfterFullGC, -XX:CompileCommandFile=./apache-cassandra-3.11.4/bin/../conf/hotspot_compiler, -javaagent:./apache-cassandra-3.11.4/bin/../lib/jamm-0.3.0.jar, -Dcassandra.jmx.remote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.authenticate=true, -Dcom.sun.management.jmxremote.password.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.password, -Dcom.sun.management.jmxremote.access.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.access, -Djava.library.path=./apache-cassandra-3.11.4/bin/../lib/sigar-bin, -Djava.rmi.server.hostname=172.17.41.172, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=./apache-cassandra-3.11.4/bin/../logs, -Dcassandra.storagedir=./apache-cassandra-3.11.4/bin/../data]
WARN [main] 2020-06-11 09:08:21,659 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN [main] 2020-06-11 09:08:21,660 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations
INFO [main] 2020-06-11 09:08:21,661 StartupChecks.java:176 - JMX is enabled to receive remote connections on port: 7199
INFO [main] 2020-06-11 09:08:21,663 SigarLibrary.java:44 - Initializing SIGAR library
INFO [main] 2020-06-11 09:08:21,677 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN [main] 2020-06-11 09:08:21,681 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
错误消息说 JVM 提交内存失败，因为 mmap 系统调用返回错误代码 12 (ENOMEM)。
这通常发生在进程达到 OS 内存限制之一时：
内存映射的数量超出了 vm.max_map_count sysctl 限制。这是 Cassandra 的一个很常见的问题，因为 Cassandra 倾向于映射数千个文件，并且默认限制相当低 - 大约 65K。
如何检查：wc -l /proc/<pid>/maps 其中 <pid> 是 Java 进程 ID。
如何修复：sudo sysctl vm.max_map_count=1000000
进程的虚拟内存总量超出RLIMIT_AS
检查方法：ulimit -v、cat /proc/<pid>/status | grep Vm
如何修复：ulimit -v unlimited 在同一 shell 中启动 Cassandra 之前。
Overcommit被禁用，过载率太低
如何检查：sysctl vm.overcommit_memory、sysctl vm.overcommit_ratio、cat /proc/meminfo
如何修复：sudo sysctl vm.overcommit_memory=0
我有一个有 14 个节点的 cassandra (v3.11.4) 集群，我想添加一个新节点。机器有 256GB 内存，我将堆大小（最大和最小）设置为 64GB 。但是由于内存错误，我无法添加新节点！确切的问题是什么，我需要做什么？
最后一行日志如下：
INFO [CompactionExecutor:1153] 2020-06-11 10:51:09,719 NoSpamLogger.java:91 - Maximum memory usage reached (20971520000), cannot allocate chunk of 1048576
INFO [Service Thread] 2020-06-11 10:51:10,339 GCInspector.java:284 - G1 Young Generation GC in 239ms. G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 27209478712 -> 25011094408;
INFO [Service Thread] 2020-06-11 10:51:11,217 GCInspector.java:284 - G1 Young Generation GC in 456ms. G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 25011094408 -> 23790284880; G1 Survivor Space: 167772160 -> 268435456;
INFO [Service Thread] 2020-06-11 10:51:12,192 GCInspector.java:284 - G1 Young Generation GC in 486ms. G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 23790284880 -> 23936030504; G1 Survivor Space: 268435456 -> 301989888;
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
5829.747: G1IncCollectionPause [ 772 18 22 ] [ 0 0 1 1 118 ] 0
....
5872.925: G1IncCollectionPause [ 774 17 21 ] [ 0 0 0 1 172 ] 0
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
5876.138: G1IncCollectionPause [ 774 16 24 ] [ 0 0 1 1 148 ] 0
....
6075.988: G1IncCollectionPause [ 772 22 23 ] [ 0 0 1 1 170 ] 0
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
6079.968: G1IncCollectionPause [ 772 0 0 ] [ 0 0 1405 1 135 ] 0
6084.827: G1IncCollectionPause [ 772 0 1 ] [ 0 0 88 1 218 ] 0
.....
6218.542: no vm operation [ 773 13 21 ] [ 0 0 1 1 0 ] 0
6219.252: G1IncCollectionPause [ 773 19 19 ] [ 0 0 0 1 158 ] 0
6219.751: G1CollectFull [ 773 11 20 ] [ 0 1 1 1 75023 ] 0
WARN [Service Thread] 2020-06-11 10:52:32,407 GCInspector.java:282 - G1 Old Generation GC in 67243ms. G1 Eden Space: 2919235584 -> 0; G1 Old Gen: 24289926888 -> 19483266320; G1 Survivor Space: 671088640 -> 0;
INFO [Service Thread] 2020-06-11 10:52:32,408 StatusLogger.java:47 - Pool Name Active Pending Completed Blocked All Time Blocked
WARN [GossipTasks:1] 2020-06-11 10:52:32,506 Gossiper.java:783 - Gossip stage has 9 pending tasks; skipping status check (no nodes will be marked down)
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3160f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO [Service Thread] 2020-06-11 10:52:32,515 StatusLogger.java:51 - MutationStage 576 257 5688240 0 0
INFO [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ViewMutationStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ReadStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - RequestResponseStage 0 0 13 0 0
INFO [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - ReadRepairStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - CounterMutationStage 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d172f6000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243956360960 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - MiscStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - CompactionExecutor 33 33 1068 0 0
INFO [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - MemtableReclaimMemory 0 0 221 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - PendingRangeCalculator 0 0 16 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - GossipStage 0 0 18945 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31652000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691058432 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - HintsDispatcher 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31561000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e3269d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248690071296 also had an error]
[thread 140248708142848 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b35000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696182528 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MigrationStage 0 0 27 0 0
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MemtablePostFlush 1 1 304 0 0
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0 1 1 222 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - ValidationExecutor 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - Sampler 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - MemtableFlushWriter 1 1 184 0 0
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - InternalResponseStage 0 0 25 0 0
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - AntiEntropyStage 0 0 0 0 0
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - CacheCleanupExecutor 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e327a1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248709207808 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.INFO: os::commit_memory(0x00007f8e31af4000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248695916288 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e316de000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691631872 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,530 StatusLogger.java:61 - CompactionManager 33 35
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:73 - MessagingService n/a 0/0
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:83 - Cache Type Size Capacity KeysToSave
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:85 - KeyCache 3211792520 10485760000 all
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:91 - RowCache 0 0 all
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:98 - Table Memtable ops,data
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.parent_repair_history 0,0
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8ceedd1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243279877888 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.repair_history 60,5088
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.view_build_status 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.compaction_history 182,47068
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_aggregates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_triggers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.size_estimates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.paxos 0,0
# An error report file with more information is saved as:
# /home/dev/hs_err_pid11578.log
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.views_builds_in_progress 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.batches 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_keyspaces 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.sstable_activity 6713,123062
INFO [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.batchlog 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e100fb000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248131811072 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3150a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.schema_columns 0,0
[thread 140248689714944 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,534 StatusLogger.java:101 - system.hints 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.IndexInfo 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7caaf9e000, 262144, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248944940800 also had an error]INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_columnfamilies 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_functions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.built_views 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peer_events 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.range_xfers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.transferred_ranges 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.schema_usertypes 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.local 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.available_ranges 4,20360
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.prepared_statements 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.columns 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.types 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.indexes 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.keyspaces 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.dropped_columns 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.aggregates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.triggers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.tables 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.views 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.functions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxxxts 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxet 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxage 7883,699041
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxs 4824,213056
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxlist 705216,33023249
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxs 357,256470
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxxxxxe 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxmembers 6482,260169
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxid 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxcount 84961,1532062
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxe 25736,10620210
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31762000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248692172544 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxes 5511,4021957
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxs 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxd 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxe 1460,1127389
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxs 468,131122
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxd_users 2313247,95919246
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxs 76,36133
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx.users 2287211,201973918
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34df8000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749410048 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxt 16,1040
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxe 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxo 72693,16357850
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxd 26143,1492343
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.roles 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_members 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.resource_role_permissons_index 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_permissions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.sessions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.events 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31693000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691324672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e32599000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248707077888 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34e39000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749676288 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a07e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e315af000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248835942144 also had an error]
[thread 140248690390784 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8430b9f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140205730166528 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a72e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248842954496 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
INFO: os::commit_memory(0x00007f8e11541000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248153069312 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7803456000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140153427945216 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8b5f2c9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140236574201600 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318a9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693511936 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d15901000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243929143040 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318ea000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693778176 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31868000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693245696 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b79000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696461056 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31d40000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248698324736 also had an error]
[thread 140216167200512 also had an error][thread 140248204584704 also had an error][thread 140245662537472 also had an error][thread 140209602766592 also had an error][thread 140230211868416 also had an error]
[thread 140248151602944 also had an error][thread 140210503980800 also had an error][thread 140230032652032 also had an error][thread 140198818285312 also had an error][thread 140248755021568 also had an error][thread 140248765159168 also had an error][thread 140248224515840 also had an error][thread 140230234380032 also had an error][thread 140243956045568 also had an error][thread 140248768407296 also had an error][thread 140248294921984 also had an error][thread 140248768141056 also had an error][thread 140207186040576 also had an error]
[thread 140245662004992 also had an error][thread 140248585135872 also had an error]
可用内存量如下：
我在开始加入时也看到了这些警告：
INFO [main] 2020-06-11 09:08:21,598 CassandraDaemon.java:493 - JVM Arguments: [-Xloggc:./apache-cassandra-3.11.4/bin/../logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -Xms64G, -Xmx64G, -XX:+UseG1GC, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -XX:+PrintAdaptiveSizePolicy, -XX:+PrintGCTimeStamps, -XX:+PrintSafepointStatistics, -XX:+PrintClassHistogramBeforeFullGC, -XX:+PrintClassHistogramAfterFullGC, -XX:CompileCommandFile=./apache-cassandra-3.11.4/bin/../conf/hotspot_compiler, -javaagent:./apache-cassandra-3.11.4/bin/../lib/jamm-0.3.0.jar, -Dcassandra.jmx.remote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.authenticate=true, -Dcom.sun.management.jmxremote.password.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.password, -Dcom.sun.management.jmxremote.access.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.access, -Djava.library.path=./apache-cassandra-3.11.4/bin/../lib/sigar-bin, -Djava.rmi.server.hostname=172.17.41.172, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=./apache-cassandra-3.11.4/bin/../logs, -Dcassandra.storagedir=./apache-cassandra-3.11.4/bin/../data]
WARN [main] 2020-06-11 09:08:21,659 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN [main] 2020-06-11 09:08:21,660 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations
INFO [main] 2020-06-11 09:08:21,661 StartupChecks.java:176 - JMX is enabled to receive remote connections on port: 7199
INFO [main] 2020-06-11 09:08:21,663 SigarLibrary.java:44 - Initializing SIGAR library
INFO [main] 2020-06-11 09:08:21,677 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN [main] 2020-06-11 09:08:21,681 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
错误消息说 JVM 提交内存失败，因为 mmap 系统调用返回错误代码 12 (ENOMEM)。
这通常发生在进程达到 OS 内存限制之一时：
内存映射的数量超出了
vm.max_map_countsysctl 限制。这是 Cassandra 的一个很常见的问题，因为 Cassandra 倾向于映射数千个文件，并且默认限制相当低 - 大约 65K。如何检查：
wc -l /proc/<pid>/maps其中<pid>是 Java 进程 ID。如何修复：
sudo sysctl vm.max_map_count=1000000进程的虚拟内存总量超出
RLIMIT_AS检查方法：
ulimit -v、cat /proc/<pid>/status | grep Vm如何修复：
ulimit -v unlimited在同一 shell 中启动 Cassandra 之前。Overcommit被禁用，过载率太低
如何检查：
sysctl vm.overcommit_memory、sysctl vm.overcommit_ratio、cat /proc/meminfo如何修复：
sudo sysctl vm.overcommit_memory=0