如何使用 Zookeeper 正确配置 HDFS 高可用性？
How to properly configure HDFS high availability using Zookeeper?
我是学大数据的。今天来找大家提问关于HDFS使用Zookeeper的高可用问题。我知道已经有 bcp of topic 处理这个主题，我已经阅读了很多。已经 15 天了，我一直在浏览论坛而没有找到我要找的东西（也许我也没有找对地方 ;-) ）
这里的流程我已经三遍了：https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html.
我可能做对了所有事情，但是当我杀死我的一个名称节点时，none 其中一个接管了。
我的架构如下：
- 5 个虚拟机
- VM 1,3 和 5 是名称节点
- VM 1 到 5 是数据节点。
我启动了我的日志节点，我启动了我的 DFSZKFailoverController，我格式化了我的第一个名称节点，我用 -bootstrapStandby 将我的第一个名称节点的配置复制到另外两个，然后我启动了我的集群。
尽管如此并且 ZKFC 和名称节点日志中没有明显的问题，但我无法让名称节点接管垂死的名称节点。
有人知道如何帮助我吗？
非常感谢您的帮助:)
zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=5
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=2
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/zookeeper/zoo
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true
admin.serverPort=7979
server.1=10.10.10.15:2888:3888
server.2=10.10.10.16:2888:3888
server.3=10.10.10.17:2888:3888
server.4=10.10.10.18:2888:3888
server.5=10.10.10.19:2888:3888
核心-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- default configuration -->
<property>
<name>fs.default.name</name>
<value>hdfs://my-cluster</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<!-- zookeeper configuration -->
<property>
<name>ha.zookeeper.quorum</name>
<value>10.10.10.15:2181,10.10.10.16:2181,10.10.10.17:2181,10.10.10.18:2181,10.10.10.19:2181</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- cluster configuration -->
<property>
<name>dfs.nameservices</name>
<value>my-cluster</value>
</property>
<!-- namenode configuration -->
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hdfs/data/nameNode</value>
</property>
<!-- datanode configuration -->
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hdfs/data/dataNode</value>
</property>
<!-- secondary namenode configuration -->
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/home/hdfs/data/secondaryNameNode</value>
</property>
<!-- replication factor -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- webhdfs connector -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<!-- high-availability configuration -->
<property>
<name>dfs.ha.namenodes.my-cluster</name>
<value>nn1,nn2,nn3</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn1</name>
<value>10.10.10.15:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn2</name>
<value>10.10.10.19:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn3</name>
<value>10.10.10.17:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn1</name>
<value>10.10.10.15:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn2</name>
<value>10.10.10.19:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn3</name>
<value>10.10.10.17:9870</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://10.10.10.15:8485;10.10.10.19:8485;10.10.10.17:8485/my-cluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hdfs/data/journalNode</value>
</property>
<!-- failover configuration -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.my-cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hdfsuser/.ssh/id_rsa</value>
</property>
</configuration>
dfs.service
[Unit]
Description=Hadoop DFS namenode and datanode
After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target
Requires=network-online.target
[Service]
User=hdfsuser
Group=hdfsgroup
Type=simple
ExecStart=/apps/hadoop/sbin/start-dfs.sh
ExecStop=/apps/hadoop/sbin/stop-dfs.sh
RemainAfterExit=yes
Restart=on-failure
StartLimitInterval=350
StartLimitBurst=10
[Install]
WantedBy=multi-user.target
hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log
（在我使名称节点崩溃之前）
2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.2.1
STARTUP_MSG: classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hado$STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG: java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoo20-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 wat20-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.20-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log
（在我崩溃了一个名称节点之后）
2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.2.1
STARTUP_MSG: classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/commons-net-$STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG: java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/20-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.20-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, i20-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
2020-04-09 13:38:59,910 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.io.EOFException: End of File Exception between local host is: "node15-hdfs-spark-master/10.10.10.15"; destination host is: "hdfs-0":9000; : java.io.EOFException; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:791)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access0(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1850)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1183)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079)
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,938 WARN org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Can't get local NN thread dump due to Connexion refusée (Connection refused)
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hdfs-0/10.10.10.15:9000 and marking that fencing is necessary
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ZooKeeper: Session: 0x50000a3038f0000 closed
2020-04-09 13:38:59,947 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x50000a3038f0000
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x50000a3038f0000
2020-04-09 13:39:01,951 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1020-04-09 13:39:01,953 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access0(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access00(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 8 more
2020-04-09 13:39:03,956 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1020-04-09 13:39:03,958 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access0(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access00(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 8 more
...
...
...
我的配置问题最终还是出在我安装hadoop集群时没有安装的两条命令：
- 首先是 nc 命令：通过从 yum 安装 nmap 包修复
- 然后命令 fuser: 通过从 yum 安装 psmisc 包修复
yum install -y nmap.x86_64
yum install -y psmisc.x86_64
希望它能很快对你们中的其他人有所帮助。
我是学大数据的。今天来找大家提问关于HDFS使用Zookeeper的高可用问题。我知道已经有 bcp of topic 处理这个主题，我已经阅读了很多。已经 15 天了，我一直在浏览论坛而没有找到我要找的东西（也许我也没有找对地方 ;-) ）
这里的流程我已经三遍了：https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html.
我可能做对了所有事情，但是当我杀死我的一个名称节点时，none 其中一个接管了。
我的架构如下： - 5 个虚拟机 - VM 1,3 和 5 是名称节点 - VM 1 到 5 是数据节点。
我启动了我的日志节点，我启动了我的 DFSZKFailoverController，我格式化了我的第一个名称节点，我用 -bootstrapStandby 将我的第一个名称节点的配置复制到另外两个，然后我启动了我的集群。
尽管如此并且 ZKFC 和名称节点日志中没有明显的问题，但我无法让名称节点接管垂死的名称节点。
有人知道如何帮助我吗？
非常感谢您的帮助:)
zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=5
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=2
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/zookeeper/zoo
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true
admin.serverPort=7979
server.1=10.10.10.15:2888:3888
server.2=10.10.10.16:2888:3888
server.3=10.10.10.17:2888:3888
server.4=10.10.10.18:2888:3888
server.5=10.10.10.19:2888:3888
核心-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- default configuration -->
<property>
<name>fs.default.name</name>
<value>hdfs://my-cluster</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<!-- zookeeper configuration -->
<property>
<name>ha.zookeeper.quorum</name>
<value>10.10.10.15:2181,10.10.10.16:2181,10.10.10.17:2181,10.10.10.18:2181,10.10.10.19:2181</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- cluster configuration -->
<property>
<name>dfs.nameservices</name>
<value>my-cluster</value>
</property>
<!-- namenode configuration -->
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hdfs/data/nameNode</value>
</property>
<!-- datanode configuration -->
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hdfs/data/dataNode</value>
</property>
<!-- secondary namenode configuration -->
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/home/hdfs/data/secondaryNameNode</value>
</property>
<!-- replication factor -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- webhdfs connector -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<!-- high-availability configuration -->
<property>
<name>dfs.ha.namenodes.my-cluster</name>
<value>nn1,nn2,nn3</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn1</name>
<value>10.10.10.15:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn2</name>
<value>10.10.10.19:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.my-cluster.nn3</name>
<value>10.10.10.17:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn1</name>
<value>10.10.10.15:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn2</name>
<value>10.10.10.19:9870</value>
</property>
<property>
<name>dfs.namenode.http-address.my-cluster.nn3</name>
<value>10.10.10.17:9870</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://10.10.10.15:8485;10.10.10.19:8485;10.10.10.17:8485/my-cluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hdfs/data/journalNode</value>
</property>
<!-- failover configuration -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.my-cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hdfsuser/.ssh/id_rsa</value>
</property>
</configuration>
dfs.service
[Unit]
Description=Hadoop DFS namenode and datanode
After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target
Requires=network-online.target
[Service]
User=hdfsuser
Group=hdfsgroup
Type=simple
ExecStart=/apps/hadoop/sbin/start-dfs.sh
ExecStop=/apps/hadoop/sbin/stop-dfs.sh
RemainAfterExit=yes
Restart=on-failure
StartLimitInterval=350
StartLimitBurst=10
[Install]
WantedBy=multi-user.target
hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log （在我使名称节点崩溃之前）
2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.2.1
STARTUP_MSG: classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hado$STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG: java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoo20-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 wat20-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.20-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
hadoop-hdfsuser-zkfc-node15-hdfs-spark-master.log （在我崩溃了一个名称节点之后）
2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.2.1
STARTUP_MSG: classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/commons-net-$STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG: java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/20-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.20-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, i20-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
2020-04-09 13:38:59,910 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.io.EOFException: End of File Exception between local host is: "node15-hdfs-spark-master/10.10.10.15"; destination host is: "hdfs-0":9000; : java.io.EOFException; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:791)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access0(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1850)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1183)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079)
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,938 WARN org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Can't get local NN thread dump due to Connexion refusée (Connection refused)
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hdfs-0/10.10.10.15:9000 and marking that fencing is necessary
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ZooKeeper: Session: 0x50000a3038f0000 closed
2020-04-09 13:38:59,947 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x50000a3038f0000
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x50000a3038f0000
2020-04-09 13:39:01,951 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1020-04-09 13:39:01,953 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access0(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access00(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 8 more
2020-04-09 13:39:03,956 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1020-04-09 13:39:03,958 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/ha$
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
at org.apache.hadoop.ha.HealthMonitor.access0(HealthMonitor.java:49)
at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access00(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 8 more
...
...
...
我的配置问题最终还是出在我安装hadoop集群时没有安装的两条命令：
- 首先是 nc 命令：通过从 yum 安装 nmap 包修复
- 然后命令 fuser: 通过从 yum 安装 psmisc 包修复
yum install -y nmap.x86_64
yum install -y psmisc.x86_64
希望它能很快对你们中的其他人有所帮助。