如何将第三个节点添加到 2 节点 Cassandra 集群？
How do I add a third node to a 2-node Cassandra Cluster?
我正在使用 Cassandra v3.9。我如下所示更新了配置文件，但 25.80.44.50 不会加入集群（它与 25.80.44.51 一起正式成为 2 节点集群的一部分）。
当我恢复到 2 节点集群时，数据库已损坏，我不得不从备份中恢复。
节点 1
配置：
[root@program-node01-vm conf]# egrep -v "^$|^#|^ *#" /var/lib/cassandra/conf/cassandra.yaml
cluster_name: 'My Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "25.80.44.51,25.80.44.52"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 25.80.44.50
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: 25.80.44.50
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
require_client_auth: false
require_endpoint_verification: false
client_encryption_options:
enabled: false
require_client_auth: false
internode_compression: none
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10
compaction_large_partition_warning_threshold_mb: 100
gc_warn_threshold_in_ms: 1000
低频:
[root@program-node01 conf]# lsof -P -n -p 13328 | grep -i tcp
java 13328 cassandra 93u IPv4 7854361 0t0 TCP 127.0.0.1:46464 (LISTEN)
java 13328 cassandra 94u IPv4 7854362 0t0 TCP *:7199 (LISTEN)
java 13328 cassandra 97u IPv4 7803257 0t0 TCP 25.80.44.50:7000->25.80.44.51:53828 (ESTABLISHED)
java 13328 cassandra 99u IPv4 7866406 0t0 TCP 25.80.44.50:37347->25.80.44.51:7000 (ESTABLISHED)
java 13328 cassandra 101u IPv4 7828918 0t0 TCP 25.80.44.50:51688->25.80.44.52:7000 (ESTABLISHED)
java 13328 cassandra 104u IPv4 7866413 0t0 TCP 25.80.44.50:7000->25.80.44.51:41813 (ESTABLISHED)
java 13328 cassandra 105u IPv4 7752003 0t0 TCP 25.80.44.50:7000->25.80.44.52:50307 (ESTABLISHED)
java 13328 cassandra 113u IPv4 7869505 0t0 TCP 25.80.44.50:34373->25.80.44.52:7000 (ESTABLISHED)
java 13328 cassandra 194u IPv4 7826676 0t0 TCP 25.80.44.50:60348->25.80.44.51:7000 (ESTABLISHED)
java 13328 cassandra 220u IPv4 7803243 0t0 TCP 25.80.44.50:7000 (LISTEN)
java 13328 cassandra 336u IPv4 7803259 0t0 TCP 25.80.44.50:9042 (LISTEN)
java 13328 cassandra 337u IPv4 7752004 0t0 TCP 25.80.44.50:7000->25.80.44.50:37007 (ESTABLISHED)
java 13328 cassandra 340u IPv4 7884819 0t0 TCP 25.80.44.50:37007->25.80.44.50:7000 (ESTABLISHED)
这证明此节点 (25.80.44.50) 已连接（通过 tcp）到其他两个节点（25.80.44.51 和 25.80.44.52）。但是，我没有看到类似于 7000 的端口 9042 的连接。我认为 9042 是节点到节点的通信端口。
网络统计
[root@program-node01 conf]# netstat -anp | grep 7000
tcp 0 0 25.80.44.50:7000 0.0.0.0:* LISTEN 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.51:53828 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.50:37007 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:51688 25.80.44.52:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.52:50307 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:37007 25.80.44.50:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.51:41813 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:34373 25.80.44.52:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:37347 25.80.44.51:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:60348 25.80.44.51:7000 ESTABLISHED 13328/java
[root@program-node01 conf]# netstat -anp | grep 9042
tcp 0 0 25.80.44.50:9042 0.0.0.0:* LISTEN 13328/java
[root@program-node01 conf]#
节点工具信息：
[root@program-node01 conf]# /var/lib/cassandra/bin/nodetool info
ID : 5db50d6d-3eae-4a47-b7c9-4743cd80ec99
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 50.69 MiB
Generation No : 1616531552
Uptime (seconds) : 5663
Heap Memory (MB) : 80.06 / 7987.25
Off Heap Memory (MB) : 0.07
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 132, size 24.41 KiB, capacity 100 MiB, 94 hits, 161 requests, 0.584 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 10, size 1.17 KiB, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 14, size 896 KiB, capacity 480 MiB, 70 misses, 267 requests, 0.738 recent hit rate, NaN microseconds miss latency
Percent Repaired : 91.24266615492786%
Token : (node is not joined to the cluster)
[root@program-node01 conf]#
我认为这是问题所在！
(node is not joined to the cluster)为Token:
节点 2
配置：
[root@program-node02-vm conf]# egrep -v "^$|^#|^ *#" /var/lib/cassandra/conf/cassandra.yaml
cluster_name: 'My Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "25.80.44.50,25.80.44.52"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 25.80.44.51
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: 25.80.44.51
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
require_client_auth: false
require_endpoint_verification: false
client_encryption_options:
enabled: false
require_client_auth: false
internode_compression: none
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10
compaction_large_partition_warning_threshold_mb: 100
gc_warn_threshold_in_ms: 1000
低频:
[root@program-node02 conf]# lsof -P -n -p 12100 | grep -i tcp
java 12100 cassandra 72u IPv4 7457886 0t0 TCP 25.80.44.51:7000->25.80.44.52:39067 (ESTABLISHED)
java 12100 cassandra 73u IPv4 7441813 0t0 TCP 25.80.44.51:55223->25.80.44.52:7000 (ESTABLISHED)
java 12100 cassandra 76u IPv4 7452718 0t0 TCP 25.80.44.51:41813->25.80.44.50:7000 (ESTABLISHED)
java 12100 cassandra 89u IPv4 7475374 0t0 TCP 127.0.0.1:39020 (LISTEN)
java 12100 cassandra 90u IPv4 7475375 0t0 TCP *:7199 (LISTEN)
java 12100 cassandra 92u IPv4 7429183 0t0 TCP 25.80.44.51:44473->25.80.44.52:7000 (ESTABLISHED)
java 12100 cassandra 93u IPv4 6972054 0t0 TCP 25.80.44.51:33449->25.80.44.52:7000 (ESTABLISHED)
java 12100 cassandra 94u IPv4 7451798 0t0 TCP 25.80.44.51:7000->25.80.44.50:60348 (ESTABLISHED)
java 12100 cassandra 97u IPv4 7441814 0t0 TCP 25.80.44.51:7000->25.80.44.51:33562 (ESTABLISHED)
java 12100 cassandra 235u IPv4 7464346 0t0 TCP 25.80.44.51:7000 (LISTEN)
java 12100 cassandra 236u IPv4 7291404 0t0 TCP 25.80.44.51:7000->25.80.44.50:37347 (ESTABLISHED)
java 12100 cassandra 237u IPv4 7451793 0t0 TCP 25.80.44.51:7000->25.80.44.52:59547 (ESTABLISHED)
java 12100 cassandra 325u IPv4 7429185 0t0 TCP 25.80.44.51:9042 (LISTEN)
java 12100 cassandra 326u IPv4 6972044 0t0 TCP 25.80.44.51:53828->25.80.44.50:7000 (ESTABLISHED)
java 12100 cassandra 329u IPv4 7291409 0t0 TCP 25.80.44.51:33562->25.80.44.51:7000 (ESTABLISHED)
网络统计：
[root@program-node02 conf]# netstat -anp | grep 7000
tcp 0 0 25.80.44.51:7000 0.0.0.0:* LISTEN 12100/java
tcp 0 0 25.80.44.51:33562 25.80.44.51:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.50:37347 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.51:33562 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.52:59547 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.52:39067 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:41813 25.80.44.50:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:53828 25.80.44.50:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.50:60348 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:44473 25.80.44.52:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:55223 25.80.44.52:7000 ESTABLISHED 12100/java
[root@program-node02 conf]# netstat -anp | grep 9042
tcp 0 0 25.80.44.51:9042 0.0.0.0:* LISTEN 12100/java
[root@program-node02 conf]#
节点工具信息：
[root@program-node02 conf]# /var/lib/cassandra/bin/nodetool info
ID : fa92ebd8-32f8-4f4f-bc85-37afd86a20c2
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 50.71 MiB
Generation No : 1616531553
Uptime (seconds) : 5673
Heap Memory (MB) : 1122.51 / 7987.25
Off Heap Memory (MB) : 0.07
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 110, size 22.55 KiB, capacity 100 MiB, 108 hits, 173 requests, 0.624 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 4, size 480 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 493, size 30.81 MiB, capacity 480 MiB, 567 misses, 796 requests, 0.288 recent hit rate, NaN microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/--tokens to see all 256 tokens)
[root@program-node02 conf]#
节点 3
配置：
[root@program-node03 conf]# egrep -v "^$|^#|^ *#" /var/lib/cassandra/conf/cassandra.yaml
cluster_name: 'My Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "25.80.44.50,25.80.44.51"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 25.80.44.52
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: 25.80.44.52
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
require_client_auth: false
require_endpoint_verification: false
client_encryption_options:
enabled: false
require_client_auth: false
internode_compression: none
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10
compaction_large_partition_warning_threshold_mb: 100
低频:
[root@program-node03 conf]# lsof -P -n -p 30557 | grep -i tcp
java 30557 cassandra 79u IPv4 32883385 0t0 TCP 25.80.44.52:7000->25.80.44.52:35735 (ESTABLISHED)
java 30557 cassandra 80u IPv4 32848102 0t0 TCP 25.80.44.52:50307->25.80.44.50:7000 (ESTABLISHED)
java 30557 cassandra 84u IPv4 32833187 0t0 TCP 25.80.44.52:7000->25.80.44.51:44473 (ESTABLISHED)
java 30557 cassandra 87u IPv4 32848098 0t0 TCP 25.80.44.52:7000->25.80.44.50:51688 (ESTABLISHED)
java 30557 cassandra 89u IPv4 32848099 0t0 TCP 25.80.44.52:39067->25.80.44.51:7000 (ESTABLISHED)
java 30557 cassandra 95u IPv4 32844289 0t0 TCP 127.0.0.1:33308 (LISTEN)
java 30557 cassandra 96u IPv4 32844290 0t0 TCP *:7199 (LISTEN)
java 30557 cassandra 97u IPv4 32848100 0t0 TCP 25.80.44.52:7000->25.80.44.50:34373 (ESTABLISHED)
java 30557 cassandra 98u IPv4 32887275 0t0 TCP 25.80.44.52:35735->25.80.44.52:7000 (ESTABLISHED)
java 30557 cassandra 100u IPv4 32848101 0t0 TCP 25.80.44.52:7000->25.80.44.52:37285 (ESTABLISHED)
java 30557 cassandra 212u IPv4 32833190 0t0 TCP 25.80.44.52:9042 (LISTEN)
java 30557 cassandra 215u IPv4 32853056 0t0 TCP 25.80.44.52:37285->25.80.44.52:7000 (ESTABLISHED)
java 30557 cassandra 216u IPv4 32851005 0t0 TCP 25.80.44.52:7000->25.80.44.51:33449 (ESTABLISHED)
java 30557 cassandra 226u IPv4 32833179 0t0 TCP 25.80.44.52:7000 (LISTEN)
java 30557 cassandra 228u IPv4 32833185 0t0 TCP 25.80.44.52:59547->25.80.44.51:7000 (ESTABLISHED)
java 30557 cassandra 229u IPv4 32833186 0t0 TCP 25.80.44.52:7000->25.80.44.51:55223 (ESTABLISHED)
网络统计：
[root@program-node03 conf]# netstat -anp | grep 7000
tcp 0 0 25.80.44.52:7000 0.0.0.0:* LISTEN 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.51:44473 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:37285 25.80.44.52:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.50:34373 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.50:51688 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:50307 25.80.44.50:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:59547 25.80.44.51:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.52:37285 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:39067 25.80.44.51:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.51:55223 ESTABLISHED 30557/java
[root@program-node03 conf]# netstat -anp | grep 9042
tcp 0 0 25.80.44.52:9042 0.0.0.0:* LISTEN 30557/java
[root@program-node03 conf]#
节点工具信息：
[root@program-node03 conf]# /var/lib/cassandra/bin/nodetool info
ID : 5db50d6d-3eae-4a47-b7c9-4743cd80ec99
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 47.3 MiB
Generation No : 1616531553
Uptime (seconds) : 5674
Heap Memory (MB) : 81.91 / 1916.00
Off Heap Memory (MB) : 0.07
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 93, size 12.27 KiB, capacity 95 MiB, 164 hits, 221 requests, 0.742 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 8, size 960 bytes, capacity 47 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 420, size 26.25 MiB, capacity 447 MiB, 559 misses, 866 requests, 0.355 recent hit rate, NaN microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/--tokens to see all 256 tokens)
[root@program-node03 conf]#
尝试将您的 seeds 列表设置为在所有 3 个节点中都相同。
现在，每个节点都不同...只有当种子列表中的一个节点已经是 运行 时才有效。集群中的第一个节点需要在该种子列表中找到自己。
在像这样的小型集群中，我将从您的第一个节点获取 seeds 设置，并确保所有其他节点的 seeds 属性与其匹配：
- seeds: "25.80.44.51,25.80.44.52"
这将是一个好的开始。
已解决：
问题出在数据库数据中。使用上面的配置，我：
- 停止了 Cassandra 的所有 3 个节点。
- 从节点 2 和 3 删除了数据库
- 从我在更改任何内容之前所做的备份恢复了节点 1 数据库
- 在节点 1 上启动 Cassandra（并让它“安定下来”）
- 在节点 2 上启动 Cassandra（并让它复制所有数据）
- 在节点 3 上启动 Cassandra（并让它复制所有数据）
我唯一剩下的问题是每个节点只有 68%（ish）的数据。由于我正在寻找容错能力，我相信我需要将模式中所有数据的 replication_factor 更改为 3。
[root@program-node01 ~]# /var/lib/cassandra/bin/nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 25.80.44.52 45.77 MiB 256 67.2% bcee7571-a2eb-402e-b86f-c7e3327a8b7d rack1
UN 25.80.44.50 51.93 MiB 256 68.0% cc96b4c7-d55e-4b32-80aa-a114bf8af7d1 rack1
UN 25.80.44.51 51.89 MiB 256 64.8% fa92ebd8-32f8-4f4f-bc85-37afd86a20c2 rack1
我正在使用 Cassandra v3.9。我如下所示更新了配置文件，但 25.80.44.50 不会加入集群（它与 25.80.44.51 一起正式成为 2 节点集群的一部分）。
当我恢复到 2 节点集群时，数据库已损坏，我不得不从备份中恢复。
节点 1
配置：
[root@program-node01-vm conf]# egrep -v "^$|^#|^ *#" /var/lib/cassandra/conf/cassandra.yaml
cluster_name: 'My Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "25.80.44.51,25.80.44.52"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 25.80.44.50
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: 25.80.44.50
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
require_client_auth: false
require_endpoint_verification: false
client_encryption_options:
enabled: false
require_client_auth: false
internode_compression: none
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10
compaction_large_partition_warning_threshold_mb: 100
gc_warn_threshold_in_ms: 1000
低频:
[root@program-node01 conf]# lsof -P -n -p 13328 | grep -i tcp
java 13328 cassandra 93u IPv4 7854361 0t0 TCP 127.0.0.1:46464 (LISTEN)
java 13328 cassandra 94u IPv4 7854362 0t0 TCP *:7199 (LISTEN)
java 13328 cassandra 97u IPv4 7803257 0t0 TCP 25.80.44.50:7000->25.80.44.51:53828 (ESTABLISHED)
java 13328 cassandra 99u IPv4 7866406 0t0 TCP 25.80.44.50:37347->25.80.44.51:7000 (ESTABLISHED)
java 13328 cassandra 101u IPv4 7828918 0t0 TCP 25.80.44.50:51688->25.80.44.52:7000 (ESTABLISHED)
java 13328 cassandra 104u IPv4 7866413 0t0 TCP 25.80.44.50:7000->25.80.44.51:41813 (ESTABLISHED)
java 13328 cassandra 105u IPv4 7752003 0t0 TCP 25.80.44.50:7000->25.80.44.52:50307 (ESTABLISHED)
java 13328 cassandra 113u IPv4 7869505 0t0 TCP 25.80.44.50:34373->25.80.44.52:7000 (ESTABLISHED)
java 13328 cassandra 194u IPv4 7826676 0t0 TCP 25.80.44.50:60348->25.80.44.51:7000 (ESTABLISHED)
java 13328 cassandra 220u IPv4 7803243 0t0 TCP 25.80.44.50:7000 (LISTEN)
java 13328 cassandra 336u IPv4 7803259 0t0 TCP 25.80.44.50:9042 (LISTEN)
java 13328 cassandra 337u IPv4 7752004 0t0 TCP 25.80.44.50:7000->25.80.44.50:37007 (ESTABLISHED)
java 13328 cassandra 340u IPv4 7884819 0t0 TCP 25.80.44.50:37007->25.80.44.50:7000 (ESTABLISHED)
这证明此节点 (25.80.44.50) 已连接（通过 tcp）到其他两个节点（25.80.44.51 和 25.80.44.52）。但是，我没有看到类似于 7000 的端口 9042 的连接。我认为 9042 是节点到节点的通信端口。
网络统计
[root@program-node01 conf]# netstat -anp | grep 7000
tcp 0 0 25.80.44.50:7000 0.0.0.0:* LISTEN 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.51:53828 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.50:37007 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:51688 25.80.44.52:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.52:50307 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:37007 25.80.44.50:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:7000 25.80.44.51:41813 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:34373 25.80.44.52:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:37347 25.80.44.51:7000 ESTABLISHED 13328/java
tcp 0 0 25.80.44.50:60348 25.80.44.51:7000 ESTABLISHED 13328/java
[root@program-node01 conf]# netstat -anp | grep 9042
tcp 0 0 25.80.44.50:9042 0.0.0.0:* LISTEN 13328/java
[root@program-node01 conf]#
节点工具信息：
[root@program-node01 conf]# /var/lib/cassandra/bin/nodetool info
ID : 5db50d6d-3eae-4a47-b7c9-4743cd80ec99
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 50.69 MiB
Generation No : 1616531552
Uptime (seconds) : 5663
Heap Memory (MB) : 80.06 / 7987.25
Off Heap Memory (MB) : 0.07
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 132, size 24.41 KiB, capacity 100 MiB, 94 hits, 161 requests, 0.584 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 10, size 1.17 KiB, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 14, size 896 KiB, capacity 480 MiB, 70 misses, 267 requests, 0.738 recent hit rate, NaN microseconds miss latency
Percent Repaired : 91.24266615492786%
Token : (node is not joined to the cluster)
[root@program-node01 conf]#
我认为这是问题所在！
(node is not joined to the cluster)为Token:
节点 2
配置：
[root@program-node02-vm conf]# egrep -v "^$|^#|^ *#" /var/lib/cassandra/conf/cassandra.yaml
cluster_name: 'My Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "25.80.44.50,25.80.44.52"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 25.80.44.51
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: 25.80.44.51
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
require_client_auth: false
require_endpoint_verification: false
client_encryption_options:
enabled: false
require_client_auth: false
internode_compression: none
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10
compaction_large_partition_warning_threshold_mb: 100
gc_warn_threshold_in_ms: 1000
低频:
[root@program-node02 conf]# lsof -P -n -p 12100 | grep -i tcp
java 12100 cassandra 72u IPv4 7457886 0t0 TCP 25.80.44.51:7000->25.80.44.52:39067 (ESTABLISHED)
java 12100 cassandra 73u IPv4 7441813 0t0 TCP 25.80.44.51:55223->25.80.44.52:7000 (ESTABLISHED)
java 12100 cassandra 76u IPv4 7452718 0t0 TCP 25.80.44.51:41813->25.80.44.50:7000 (ESTABLISHED)
java 12100 cassandra 89u IPv4 7475374 0t0 TCP 127.0.0.1:39020 (LISTEN)
java 12100 cassandra 90u IPv4 7475375 0t0 TCP *:7199 (LISTEN)
java 12100 cassandra 92u IPv4 7429183 0t0 TCP 25.80.44.51:44473->25.80.44.52:7000 (ESTABLISHED)
java 12100 cassandra 93u IPv4 6972054 0t0 TCP 25.80.44.51:33449->25.80.44.52:7000 (ESTABLISHED)
java 12100 cassandra 94u IPv4 7451798 0t0 TCP 25.80.44.51:7000->25.80.44.50:60348 (ESTABLISHED)
java 12100 cassandra 97u IPv4 7441814 0t0 TCP 25.80.44.51:7000->25.80.44.51:33562 (ESTABLISHED)
java 12100 cassandra 235u IPv4 7464346 0t0 TCP 25.80.44.51:7000 (LISTEN)
java 12100 cassandra 236u IPv4 7291404 0t0 TCP 25.80.44.51:7000->25.80.44.50:37347 (ESTABLISHED)
java 12100 cassandra 237u IPv4 7451793 0t0 TCP 25.80.44.51:7000->25.80.44.52:59547 (ESTABLISHED)
java 12100 cassandra 325u IPv4 7429185 0t0 TCP 25.80.44.51:9042 (LISTEN)
java 12100 cassandra 326u IPv4 6972044 0t0 TCP 25.80.44.51:53828->25.80.44.50:7000 (ESTABLISHED)
java 12100 cassandra 329u IPv4 7291409 0t0 TCP 25.80.44.51:33562->25.80.44.51:7000 (ESTABLISHED)
网络统计：
[root@program-node02 conf]# netstat -anp | grep 7000
tcp 0 0 25.80.44.51:7000 0.0.0.0:* LISTEN 12100/java
tcp 0 0 25.80.44.51:33562 25.80.44.51:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.50:37347 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.51:33562 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.52:59547 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.52:39067 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:41813 25.80.44.50:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:53828 25.80.44.50:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:7000 25.80.44.50:60348 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:44473 25.80.44.52:7000 ESTABLISHED 12100/java
tcp 0 0 25.80.44.51:55223 25.80.44.52:7000 ESTABLISHED 12100/java
[root@program-node02 conf]# netstat -anp | grep 9042
tcp 0 0 25.80.44.51:9042 0.0.0.0:* LISTEN 12100/java
[root@program-node02 conf]#
节点工具信息：
[root@program-node02 conf]# /var/lib/cassandra/bin/nodetool info
ID : fa92ebd8-32f8-4f4f-bc85-37afd86a20c2
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 50.71 MiB
Generation No : 1616531553
Uptime (seconds) : 5673
Heap Memory (MB) : 1122.51 / 7987.25
Off Heap Memory (MB) : 0.07
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 110, size 22.55 KiB, capacity 100 MiB, 108 hits, 173 requests, 0.624 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 4, size 480 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 493, size 30.81 MiB, capacity 480 MiB, 567 misses, 796 requests, 0.288 recent hit rate, NaN microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/--tokens to see all 256 tokens)
[root@program-node02 conf]#
节点 3
配置：
[root@program-node03 conf]# egrep -v "^$|^#|^ *#" /var/lib/cassandra/conf/cassandra.yaml
cluster_name: 'My Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "25.80.44.50,25.80.44.51"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 25.80.44.52
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: 25.80.44.52
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
require_client_auth: false
require_endpoint_verification: false
client_encryption_options:
enabled: false
require_client_auth: false
internode_compression: none
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10
compaction_large_partition_warning_threshold_mb: 100
低频:
[root@program-node03 conf]# lsof -P -n -p 30557 | grep -i tcp
java 30557 cassandra 79u IPv4 32883385 0t0 TCP 25.80.44.52:7000->25.80.44.52:35735 (ESTABLISHED)
java 30557 cassandra 80u IPv4 32848102 0t0 TCP 25.80.44.52:50307->25.80.44.50:7000 (ESTABLISHED)
java 30557 cassandra 84u IPv4 32833187 0t0 TCP 25.80.44.52:7000->25.80.44.51:44473 (ESTABLISHED)
java 30557 cassandra 87u IPv4 32848098 0t0 TCP 25.80.44.52:7000->25.80.44.50:51688 (ESTABLISHED)
java 30557 cassandra 89u IPv4 32848099 0t0 TCP 25.80.44.52:39067->25.80.44.51:7000 (ESTABLISHED)
java 30557 cassandra 95u IPv4 32844289 0t0 TCP 127.0.0.1:33308 (LISTEN)
java 30557 cassandra 96u IPv4 32844290 0t0 TCP *:7199 (LISTEN)
java 30557 cassandra 97u IPv4 32848100 0t0 TCP 25.80.44.52:7000->25.80.44.50:34373 (ESTABLISHED)
java 30557 cassandra 98u IPv4 32887275 0t0 TCP 25.80.44.52:35735->25.80.44.52:7000 (ESTABLISHED)
java 30557 cassandra 100u IPv4 32848101 0t0 TCP 25.80.44.52:7000->25.80.44.52:37285 (ESTABLISHED)
java 30557 cassandra 212u IPv4 32833190 0t0 TCP 25.80.44.52:9042 (LISTEN)
java 30557 cassandra 215u IPv4 32853056 0t0 TCP 25.80.44.52:37285->25.80.44.52:7000 (ESTABLISHED)
java 30557 cassandra 216u IPv4 32851005 0t0 TCP 25.80.44.52:7000->25.80.44.51:33449 (ESTABLISHED)
java 30557 cassandra 226u IPv4 32833179 0t0 TCP 25.80.44.52:7000 (LISTEN)
java 30557 cassandra 228u IPv4 32833185 0t0 TCP 25.80.44.52:59547->25.80.44.51:7000 (ESTABLISHED)
java 30557 cassandra 229u IPv4 32833186 0t0 TCP 25.80.44.52:7000->25.80.44.51:55223 (ESTABLISHED)
网络统计：
[root@program-node03 conf]# netstat -anp | grep 7000
tcp 0 0 25.80.44.52:7000 0.0.0.0:* LISTEN 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.51:44473 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:37285 25.80.44.52:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.50:34373 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.50:51688 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:50307 25.80.44.50:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:59547 25.80.44.51:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.52:37285 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:39067 25.80.44.51:7000 ESTABLISHED 30557/java
tcp 0 0 25.80.44.52:7000 25.80.44.51:55223 ESTABLISHED 30557/java
[root@program-node03 conf]# netstat -anp | grep 9042
tcp 0 0 25.80.44.52:9042 0.0.0.0:* LISTEN 30557/java
[root@program-node03 conf]#
节点工具信息：
[root@program-node03 conf]# /var/lib/cassandra/bin/nodetool info
ID : 5db50d6d-3eae-4a47-b7c9-4743cd80ec99
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 47.3 MiB
Generation No : 1616531553
Uptime (seconds) : 5674
Heap Memory (MB) : 81.91 / 1916.00
Off Heap Memory (MB) : 0.07
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 93, size 12.27 KiB, capacity 95 MiB, 164 hits, 221 requests, 0.742 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 8, size 960 bytes, capacity 47 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 420, size 26.25 MiB, capacity 447 MiB, 559 misses, 866 requests, 0.355 recent hit rate, NaN microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/--tokens to see all 256 tokens)
[root@program-node03 conf]#
尝试将您的 seeds 列表设置为在所有 3 个节点中都相同。
现在，每个节点都不同...只有当种子列表中的一个节点已经是 运行 时才有效。集群中的第一个节点需要在该种子列表中找到自己。
在像这样的小型集群中，我将从您的第一个节点获取 seeds 设置，并确保所有其他节点的 seeds 属性与其匹配：
- seeds: "25.80.44.51,25.80.44.52"
这将是一个好的开始。
已解决： 问题出在数据库数据中。使用上面的配置，我：
- 停止了 Cassandra 的所有 3 个节点。
- 从节点 2 和 3 删除了数据库
- 从我在更改任何内容之前所做的备份恢复了节点 1 数据库
- 在节点 1 上启动 Cassandra（并让它“安定下来”）
- 在节点 2 上启动 Cassandra（并让它复制所有数据）
- 在节点 3 上启动 Cassandra（并让它复制所有数据）
我唯一剩下的问题是每个节点只有 68%（ish）的数据。由于我正在寻找容错能力，我相信我需要将模式中所有数据的 replication_factor 更改为 3。
[root@program-node01 ~]# /var/lib/cassandra/bin/nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 25.80.44.52 45.77 MiB 256 67.2% bcee7571-a2eb-402e-b86f-c7e3327a8b7d rack1
UN 25.80.44.50 51.93 MiB 256 68.0% cc96b4c7-d55e-4b32-80aa-a114bf8af7d1 rack1
UN 25.80.44.51 51.89 MiB 256 64.8% fa92ebd8-32f8-4f4f-bc85-37afd86a20c2 rack1