Cassandra添加节点问题,失败;错误='Cannot allocate memory' (errno=12)

Cassandra add node problem, failed; error='Cannot allocate memory' (errno=12)

我有一个有 14 个节点的 cassandra (v3.11.4) 集群,我想添加一个新节点。机器有 256GB 内存,我将堆大小(最大和最小)设置为 64GB 。但是由于内存错误,我无法添加新节点!确切的问题是什么,我需要做什么?

最后一行日志如下:

INFO  [CompactionExecutor:1153] 2020-06-11 10:51:09,719 NoSpamLogger.java:91 - Maximum memory usage reached (20971520000), cannot allocate chunk of 1048576
INFO  [Service Thread] 2020-06-11 10:51:10,339 GCInspector.java:284 - G1 Young Generation GC in 239ms.  G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 27209478712 -> 25011094408; 
INFO  [Service Thread] 2020-06-11 10:51:11,217 GCInspector.java:284 - G1 Young Generation GC in 456ms.  G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 25011094408 -> 23790284880; G1 Survivor Space: 167772160 -> 268435456; 
INFO  [Service Thread] 2020-06-11 10:51:12,192 GCInspector.java:284 - G1 Young Generation GC in 486ms.  G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 23790284880 -> 23936030504; G1 Survivor Space: 268435456 -> 301989888; 
         vmop                    [threads: total initially_running wait_to_block]    [time: spin block sync cleanup vmop] page_trap_count
5829.747: G1IncCollectionPause             [     772         18             22    ]      [     0     0     1     1   118    ]  0   
....   
5872.925: G1IncCollectionPause             [     774         17             21    ]      [     0     0     0     1   172    ]  0   
         vmop                    [threads: total initially_running wait_to_block]    [time: spin block sync cleanup vmop] page_trap_count
5876.138: G1IncCollectionPause             [     774         16             24    ]      [     0     0     1     1   148    ]  0   
....  
6075.988: G1IncCollectionPause             [     772         22             23    ]      [     0     0     1     1   170    ]  0   
         vmop                    [threads: total initially_running wait_to_block]    [time: spin block sync cleanup vmop] page_trap_count
6079.968: G1IncCollectionPause             [     772          0              0    ]      [     0     0  1405     1   135    ]  0   
6084.827: G1IncCollectionPause             [     772          0              1    ]      [     0     0    88     1   218    ]  0   
.....  
6218.542: no vm operation                  [     773         13             21    ]      [     0     0     1     1     0    ]  0   
6219.252: G1IncCollectionPause             [     773         19             19    ]      [     0     0     0     1   158    ]  0   
6219.751: G1CollectFull                    [     773         11             20    ]      [     0     1     1     1 75023    ]  0   
WARN  [Service Thread] 2020-06-11 10:52:32,407 GCInspector.java:282 - G1 Old Generation GC in 67243ms.  G1 Eden Space: 2919235584 -> 0; G1 Old Gen: 24289926888 -> 19483266320; G1 Survivor Space: 671088640 -> 0; 
INFO  [Service Thread] 2020-06-11 10:52:32,408 StatusLogger.java:47 - Pool Name                    Active   Pending      Completed   Blocked  All Time Blocked
WARN  [GossipTasks:1] 2020-06-11 10:52:32,506 Gossiper.java:783 - Gossip stage has 9 pending tasks; skipping status check (no nodes will be marked down)
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3160f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO  [Service Thread] 2020-06-11 10:52:32,515 StatusLogger.java:51 - MutationStage                   576       257        5688240         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ViewMutationStage                 0         0              0         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ReadStage                         0         0              0         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - RequestResponseStage              0         0             13         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - ReadRepairStage                   0         0              0         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - CounterMutationStage              0         0              0         0                 0

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d172f6000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243956360960 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - MiscStage                         0         0              0         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - CompactionExecutor               33        33           1068         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - MemtableReclaimMemory             0         0            221         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - PendingRangeCalculator            0         0             16         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - GossipStage                       0         0          18945         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - SecondaryIndexManagement          0         0              0         0                 0

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31652000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691058432 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - HintsDispatcher                   0         0              0         0                 0

Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31561000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e3269d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248690071296 also had an error]

[thread 140248708142848 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b35000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696182528 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MigrationStage                    0         0             27         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MemtablePostFlush                 1         1            304         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0         1         1            222         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - ValidationExecutor                0         0              0         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - Sampler                           0         0              0         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - MemtableFlushWriter               1         1            184         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - InternalResponseStage             0         0             25         0                 0

INFO  [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - AntiEntropyStage                  0         0              0         0                 0

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
INFO  [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - CacheCleanupExecutor              0         0              0         0                 0

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e327a1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248709207808 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.INFO: os::commit_memory(0x00007f8e31af4000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248695916288 also had an error]

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e316de000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691631872 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,530 StatusLogger.java:61 - CompactionManager                33        35
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:73 - MessagingService                n/a       0/0
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:83 - Cache Type                     Size                 Capacity               KeysToSave
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:85 - KeyCache                 3211792520              10485760000                      all
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:91 - RowCache                          0                        0                      all
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:98 - Table                       Memtable ops,data
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.parent_repair_history                 0,0
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8ceedd1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243279877888 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.repair_history             60,5088
INFO  [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.view_build_status                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.compaction_history           182,47068
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_aggregates                  0,0
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_triggers                    0,0
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.size_estimates                     0,0
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.paxos                              0,0
# An error report file with more information is saved as:
# /home/dev/hs_err_pid11578.log
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.views_builds_in_progress                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.batches                            0,0
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_keyspaces                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.sstable_activity           6713,123062
INFO  [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.batchlog                           0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e100fb000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248131811072 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3150a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO  [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.schema_columns                     0,0
[thread 140248689714944 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,534 StatusLogger.java:101 - system.hints                              0,0
INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.IndexInfo                          0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7caaf9e000, 262144, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248944940800 also had an error]INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_columnfamilies                 0,0

INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_functions                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.built_views                        0,0
INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peer_events                        0,0
INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.range_xfers                        0,0
INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peers                              0,0
INFO  [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.transferred_ranges                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.schema_usertypes                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.local                              0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.available_ranges               4,20360
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.prepared_statements                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.columns                     0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.types                       0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.indexes                     0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.keyspaces                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.dropped_columns                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.aggregates                  0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.triggers                    0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.tables                      0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.views                       0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.functions                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxxxts                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxet                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxage         7883,699041
INFO  [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxs            4824,213056
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxlist     705216,33023249
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxs          357,256470
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxxxxxe                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxmembers          6482,260169
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxid                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxcount       84961,1532062
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxe         25736,10620210
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31762000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248692172544 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxes        5511,4021957
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxs                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxd                   0,0
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxe         1460,1127389
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxs              468,131122
INFO  [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxd_users    2313247,95919246
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxs                76,36133
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx.users      2287211,201973918
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34df8000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749410048 also had an error]
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxt             16,1040
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx                  0,0
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxe                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxo          72693,16357850
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxd           26143,1492343
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.roles                         0,0
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_members                  0,0
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.resource_role_permissons_index                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_permissions                 0,0
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.sessions                    0,0
INFO  [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.events                      0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31693000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691324672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e32599000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248707077888 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34e39000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749676288 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a07e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e315af000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248835942144 also had an error]

[thread 140248690390784 also had an error]Java HotSpot(TM) 64-Bit Server VM warning: 
Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8430b9f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140205730166528 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a72e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248842954496 also had an error]Java HotSpot(TM) 64-Bit Server VM warning: 
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
INFO: os::commit_memory(0x00007f8e11541000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248153069312 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7803456000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140153427945216 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8b5f2c9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140236574201600 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318a9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693511936 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d15901000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243929143040 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318ea000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693778176 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31868000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693245696 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b79000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696461056 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31d40000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248698324736 also had an error]
[thread 140216167200512 also had an error][thread 140248204584704 also had an error][thread 140245662537472 also had an error][thread 140209602766592 also had an error][thread 140230211868416 also had an error]




[thread 140248151602944 also had an error][thread 140210503980800 also had an error][thread 140230032652032 also had an error][thread 140198818285312 also had an error][thread 140248755021568 also had an error][thread 140248765159168 also had an error][thread 140248224515840 also had an error][thread 140230234380032 also had an error][thread 140243956045568 also had an error][thread 140248768407296 also had an error][thread 140248294921984 also had an error][thread 140248768141056 also had an error][thread 140207186040576 also had an error]
[thread 140245662004992 also had an error][thread 140248585135872 also had an error]

可用内存量如下:

感谢您的帮助!

我在开始加入时也看到了这些警告:

 INFO  [main] 2020-06-11 09:08:21,598 CassandraDaemon.java:493 - JVM Arguments: [-Xloggc:./apache-cassandra-3.11.4/bin/../logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -Xms64G, -Xmx64G, -XX:+UseG1GC, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -XX:+PrintAdaptiveSizePolicy, -XX:+PrintGCTimeStamps, -XX:+PrintSafepointStatistics, -XX:+PrintClassHistogramBeforeFullGC, -XX:+PrintClassHistogramAfterFullGC, -XX:CompileCommandFile=./apache-cassandra-3.11.4/bin/../conf/hotspot_compiler, -javaagent:./apache-cassandra-3.11.4/bin/../lib/jamm-0.3.0.jar, -Dcassandra.jmx.remote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.authenticate=true, -Dcom.sun.management.jmxremote.password.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.password, -Dcom.sun.management.jmxremote.access.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.access, -Djava.library.path=./apache-cassandra-3.11.4/bin/../lib/sigar-bin, -Djava.rmi.server.hostname=172.17.41.172, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=./apache-cassandra-3.11.4/bin/../logs, -Dcassandra.storagedir=./apache-cassandra-3.11.4/bin/../data]
WARN  [main] 2020-06-11 09:08:21,659 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  [main] 2020-06-11 09:08:21,660 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations
INFO  [main] 2020-06-11 09:08:21,661 StartupChecks.java:176 - JMX is enabled to receive remote connections on port: 7199
INFO  [main] 2020-06-11 09:08:21,663 SigarLibrary.java:44 - Initializing SIGAR library
INFO  [main] 2020-06-11 09:08:21,677 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN  [main] 2020-06-11 09:08:21,681 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.

错误消息说 JVM 提交内存失败,因为 mmap 系统调用返回错误代码 12 (ENOMEM)。

这通常发生在进程达到 OS 内存限制之一时:

  • 内存映射的数量超出了 vm.max_map_count sysctl 限制。这是 Cassandra 的一个很常见的问题,因为 Cassandra 倾向于映射数千个文件,并且默认限制相当低 - 大约 65K。

    如何检查:wc -l /proc/<pid>/maps 其中 <pid> 是 Java 进程 ID。

    如何修复:sudo sysctl vm.max_map_count=1000000

  • 进程的虚拟内存总量超出RLIMIT_AS

    检查方法:ulimit -vcat /proc/<pid>/status | grep Vm

    如何修复:ulimit -v unlimited 在同一 shell 中启动 Cassandra 之前。

  • Overcommit被禁用,过载率太低

    如何检查:sysctl vm.overcommit_memorysysctl vm.overcommit_ratiocat /proc/meminfo

    如何修复:sudo sysctl vm.overcommit_memory=0