Cassandra 3.3 - 集群节点 - 不规则行为
Cassandra 3.3 - cluster node - irregular behavior
我有一个符合节点
的 Cassandra 集群
{ 192.168.120.57, 192.168.120.58, 192.168.120.59 }, 复制因子为 2
我检测到节点 192.168.120.59 行为异常
-> 服务是 运行 (sudo service cassandra status)
-> nodetool 还告诉我节点是 运行
XXXX@cassandra-prod03:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.120.57 4.05 GB 256 ? 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb rack1
UN 192.168.120.58 4.4 GB 256 ? 53bc77ff-50b3-4bf0-bd55-17e2bb7c8208 rack1
UN 192.168.120.59 2.77 GB 256 ? 03d8d7fd-1d86-4034-a537-9915adb0d4b3 rack1
但是 -> 我无法通过 DevCenter 进行连接（我上次尝试可以并且 cassandra.yml 没有改变）
我查看了这个节点（192.168.120.59）的日志，显示：
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,827 HintsDispatchExecutor.java:224 - Failed to dispatch hints file 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb-1465184315371-1.hints: file is corrupted ({})
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,830 CassandraDaemon.java:195 - Exception in thread Thread[HintsDispatcher:1,1,main]
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,834 StorageService.java:470 - Stopping gossiper
ERROR [HintsDispatcher:1] 2016-06-23 16:57:03,842 StorageService.java:480 - Stopping native transport
在其他节点，日志多次显示：
DEBUG [GossipTasks:1] 2016-06-24 07:56:21,551 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:22,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:23,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:24,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:25,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:26,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:27,554 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
这些是我做过但没有奏效的事情：
1) 重启机器。
2) 节点工具修复。
XXXXs@cassandra-prod03:~$ nodetool repair
[2016-06-23 17:07:37,582] Starting repair command #1, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
Jun 23, 2016 5:53:57 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException
error: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
at org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at javax.management.NotificationBroadcasterSupport.execute(NotificationBroadcasterSupport.java:337)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.access00(RMIConnector.java:121)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
3) 节点工具擦洗。
之后我关注了这个页面https://docs.datastax.com/en/cassandra/1.2/cassandra/operations/ops_backup_noderestart_t.html
在出现异常行为的节点上：
1) 删除了所有表
2) 复制并粘贴快照子目录中的表
3) 重启cassandra服务(sudo service cassandra start)
现在开发中心可以连接了
4) 我又运行 nodetool repair了，好像是没整理好。已经运行3个多小时了，还是0%（不过暂时没抛出其他错误），但我觉得不行。
XXXX@cassandra-prod03:/var/lib/cassandra/data/recommendersjobs$ nodetool repair
[2016-06-24 12:24:13,901] Starting repair command #2, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
[2016-06-24 12:24:17,235] Repair session 7990e900-3a30-11e6-9861-e998758604bf for range [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]] failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser, [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]]] Validation failed in /192.168.120.58 (progress: 0%)
在此日志中出现：
1）failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser（靠近中间）
2）Validation failed in /192.168.120.58 (progress: 0%)（在最后一部分）
虽然这次修复是运行（我认为是运行），但是在这个节点的日志和集群的其他节点中，多次出现：
DEBUG [ReadRepairStage:4] 2016-06-24 14:08:18,318 ReadCallback.java:235 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-3297346077674417080, 42555f31313131303131333538) (d41d8cd98f00b204e9800998ecf8427e vs 9e7f1588ffcd53d13ab3f8bb2be0f05e)
at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) ~[apache-cassandra-3.3.jar:3.3]
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226) ~[apache-cassandra-3.3.jar:3.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
注意 1：由于复制因子 2，cassandra 过去和现在都在工作。
注2：cassandra Cluster正在生产中使用，应用程序正在插入和读取行。
我该如何纠正这种不正常的行为？
我想创建一个新节点并将其与不规则节点交换，这样做的正确步骤是什么？
当节点没有响应/出现故障时
如果您可以创建一个新的虚拟机来替换故障节点。
参见：https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html
如果您不想创建新的虚拟机并使用现有的故障节点。
重要提示：需要有一个允许您删除受影响节点的所有数据的复制因子。
nodetool decommission (On that node)
参见：https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsDecommission.html
sudo rm -rf /var/lib/cassandra/*
clear/delete节点中的所有数据
参见：https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceClearCpkgData.html
sudo service cassandra start
一段时间后，
nodetool repair
如果nodetool修复失败：
对于键空间中的 tables：
nodetool scrub <keyspace> <table>
在每个节点中→查看节点日志并等待完成
run nodetool repair <keyspace> <table>
仅在一个节点中（任何人）
注意：没有必要为每个table运行，你可以运行：
nodetool scrub <keyspace>
在每个节点中→查看节点日志并等待完成
nodetool repair <keyspace>
我首先建议尝试 1 table 以便仅查看 table 的日志并更好地理解正在发生的事情
注意：
将登录级别设置为 INFO 以查看重要内容
nodetool setlogginglevel org.apache.cassandra INFO
我有一个符合节点
的 Cassandra 集群
{ 192.168.120.57, 192.168.120.58, 192.168.120.59 }, 复制因子为 2
我检测到节点 192.168.120.59 行为异常
-> 服务是 运行 (sudo service cassandra status)
-> nodetool 还告诉我节点是 运行
XXXX@cassandra-prod03:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.120.57 4.05 GB 256 ? 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb rack1
UN 192.168.120.58 4.4 GB 256 ? 53bc77ff-50b3-4bf0-bd55-17e2bb7c8208 rack1
UN 192.168.120.59 2.77 GB 256 ? 03d8d7fd-1d86-4034-a537-9915adb0d4b3 rack1
但是 -> 我无法通过 DevCenter 进行连接（我上次尝试可以并且 cassandra.yml 没有改变）
我查看了这个节点（192.168.120.59）的日志，显示：
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,827 HintsDispatchExecutor.java:224 - Failed to dispatch hints file 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb-1465184315371-1.hints: file is corrupted ({})
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,830 CassandraDaemon.java:195 - Exception in thread Thread[HintsDispatcher:1,1,main]
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,834 StorageService.java:470 - Stopping gossiper
ERROR [HintsDispatcher:1] 2016-06-23 16:57:03,842 StorageService.java:480 - Stopping native transport
在其他节点，日志多次显示：
DEBUG [GossipTasks:1] 2016-06-24 07:56:21,551 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:22,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:23,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:24,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:25,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:26,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:27,554 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
这些是我做过但没有奏效的事情：
1) 重启机器。
2) 节点工具修复。
XXXXs@cassandra-prod03:~$ nodetool repair
[2016-06-23 17:07:37,582] Starting repair command #1, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
Jun 23, 2016 5:53:57 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException
error: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
at org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at javax.management.NotificationBroadcasterSupport.execute(NotificationBroadcasterSupport.java:337)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.access00(RMIConnector.java:121)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
3) 节点工具擦洗。
之后我关注了这个页面https://docs.datastax.com/en/cassandra/1.2/cassandra/operations/ops_backup_noderestart_t.html
在出现异常行为的节点上：
1) 删除了所有表
2) 复制并粘贴快照子目录中的表
3) 重启cassandra服务(sudo service cassandra start)
现在开发中心可以连接了
4) 我又运行 nodetool repair了，好像是没整理好。已经运行3个多小时了，还是0%（不过暂时没抛出其他错误），但我觉得不行。
XXXX@cassandra-prod03:/var/lib/cassandra/data/recommendersjobs$ nodetool repair
[2016-06-24 12:24:13,901] Starting repair command #2, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
[2016-06-24 12:24:17,235] Repair session 7990e900-3a30-11e6-9861-e998758604bf for range [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]] failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser, [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]]] Validation failed in /192.168.120.58 (progress: 0%)
在此日志中出现：
1）failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser（靠近中间）
2）Validation failed in /192.168.120.58 (progress: 0%)（在最后一部分）
虽然这次修复是运行（我认为是运行），但是在这个节点的日志和集群的其他节点中，多次出现：
DEBUG [ReadRepairStage:4] 2016-06-24 14:08:18,318 ReadCallback.java:235 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-3297346077674417080, 42555f31313131303131333538) (d41d8cd98f00b204e9800998ecf8427e vs 9e7f1588ffcd53d13ab3f8bb2be0f05e)
at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) ~[apache-cassandra-3.3.jar:3.3]
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226) ~[apache-cassandra-3.3.jar:3.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
注意 1：由于复制因子 2，cassandra 过去和现在都在工作。
注2：cassandra Cluster正在生产中使用，应用程序正在插入和读取行。
我该如何纠正这种不正常的行为？
我想创建一个新节点并将其与不规则节点交换，这样做的正确步骤是什么？
当节点没有响应/出现故障时
如果您可以创建一个新的虚拟机来替换故障节点。
参见：https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html
如果您不想创建新的虚拟机并使用现有的故障节点。
重要提示：需要有一个允许您删除受影响节点的所有数据的复制因子。
nodetool decommission (On that node)
参见：https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsDecommission.html
sudo rm -rf /var/lib/cassandra/*
clear/delete节点中的所有数据
参见：https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceClearCpkgData.html
sudo service cassandra start
一段时间后，
nodetool repair
如果nodetool修复失败：
对于键空间中的 tables：
nodetool scrub <keyspace> <table>
在每个节点中→查看节点日志并等待完成
run nodetool repair <keyspace> <table>
仅在一个节点中（任何人）
注意：没有必要为每个table运行，你可以运行：
nodetool scrub <keyspace>
在每个节点中→查看节点日志并等待完成
nodetool repair <keyspace>
我首先建议尝试 1 table 以便仅查看 table 的日志并更好地理解正在发生的事情
注意：
将登录级别设置为 INFO 以查看重要内容
nodetool setlogginglevel org.apache.cassandra INFO