Как правильно настроить высокую доступность HDFS с помощью Zookeeper? - PullRequest
Новая
       23

Как правильно настроить высокую доступность HDFS с помощью Zookeeper?

1 голос
/ 06 апреля 2020

Я студент больших данных. Сегодня я приду к вам с вопросом о высокой доступности HDFS с использованием Zookeeper. Я знаю, что там уже был bcp topi c, занимающийся этой темой, я уже прочитал многие из них. Уже 15 дней я просматриваю форумы, не находя того, что ищу (может быть, я тоже не в нужном месте ;-))

Я трижды выполнял процедуру здесь: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html.

Возможно, я все сделал правильно, но когда я убиваю один из моих наменодов, ни один из них не вступает во владение.

Моя архитектура такая следующим образом: - 5 ВМ - ВМ 1,3 и 5 - это нанододы - ВМ от 1 до 5 - это датододы.

Я запустил свои журнальные узлы, я запустил свой DFSZKFailoverController, я отформатировал свой первый наменод, я скопировал с -bootstrapStandby Конфигурирование моего первого наменода для двух других, и я запустил свой кластер.

Несмотря на все это и никаких очевидных проблем в журналах ZKF C и namenode, я не могу заставить namenode взять на себя умирающую namenode.

Кто-нибудь знает, как мне помочь?

Большое спасибо за вашу помощь:)

zoo.cfg 

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=5
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=2
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/home/zookeeper/zoo
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true

admin.serverPort=7979

server.1=10.10.10.15:2888:3888
server.2=10.10.10.16:2888:3888
server.3=10.10.10.17:2888:3888
server.4=10.10.10.18:2888:3888
server.5=10.10.10.19:2888:3888

core-site. xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <!-- default configuration -->
    <property>
        <name>fs.default.name</name>
        <value>hdfs://my-cluster</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>

    <!-- zookeeper configuration -->
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>10.10.10.15:2181,10.10.10.16:2181,10.10.10.17:2181,10.10.10.18:2181,10.10.10.19:2181</value>
    </property>

</configuration>
* 10 29 * hdfs-site. xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

    <!-- cluster configuration -->
    <property>
        <name>dfs.nameservices</name>
        <value>my-cluster</value>
    </property>

    <!-- namenode configuration -->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/hdfs/data/nameNode</value>
    </property>

    <!-- datanode configuration -->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/home/hdfs/data/dataNode</value>
    </property>

    <!-- secondary namenode configuration -->
    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>/home/hdfs/data/secondaryNameNode</value>
    </property>

    <!-- replication factor -->
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>

    <!-- webhdfs connector -->
    <property> 
        <name>dfs.webhdfs.enabled</name> 
        <value>true</value> 
    </property> 

    <!-- high-availability configuration -->
    <property>
        <name>dfs.ha.namenodes.my-cluster</name>
        <value>nn1,nn2,nn3</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.my-cluster.nn1</name>
        <value>10.10.10.15:9000</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.my-cluster.nn2</name>
        <value>10.10.10.19:9000</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.my-cluster.nn3</name>
        <value>10.10.10.17:9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.my-cluster.nn1</name>
        <value>10.10.10.15:9870</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.my-cluster.nn2</name>
        <value>10.10.10.19:9870</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.my-cluster.nn3</name>
        <value>10.10.10.17:9870</value>
    </property>

    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://10.10.10.15:8485;10.10.10.19:8485;10.10.10.17:8485/my-cluster</value>
    </property>

    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/home/hdfs/data/journalNode</value>
    </property>

    <!-- failover configuration -->
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>dfs.client.failover.proxy.provider.my-cluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>

    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/hdfsuser/.ssh/id_rsa</value>
    </property>

</configuration>

dfs.service 

[Unit]
Description=Hadoop DFS namenode and datanode
After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target
Requires=network-online.target

[Service]
User=hdfsuser
Group=hdfsgroup
Type=simple
ExecStart=/apps/hadoop/sbin/start-dfs.sh
ExecStop=/apps/hadoop/sbin/stop-dfs.sh
RemainAfterExit=yes
Restart=on-failure
StartLimitInterval=350
StartLimitBurst=10

[Install]
WantedBy=multi-user.target

has oop - hdfsuser-zkf c -node15-hdfs-spark-master.log (до того, как я сделаю sh имя) 

2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG:   host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 3.2.1
STARTUP_MSG:   classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hado$STARTUP_MSG:   build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG:   java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoo$2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 wat$2020-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.$2020-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state

имел oop -hdfsuser-zkf c -node15-hdfs-spark-master.log (после того, как я sh наменоде) 

2020-04-09 13:32:22,216 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG:   host = node15-hdfs-spark-master/10.10.10.15
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 3.2.1
STARTUP_MSG:   classpath = /apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/commons-net-$STARTUP_MSG:   build = https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842; compiled by 'rohithsharmaks' on 2019-09-10T15:56Z
STARTUP_MSG:   java = 1.8.0_242
************************************************************/
2020-04-09 13:32:22,229 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-09 13:32:22,628 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at hdfs-0/10.10.10.15:9000
2020-04-09 13:32:22,751 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=node15
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_242
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre
2020-04-09 13:32:22,752 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/apps/hadoop/etc/hadoop:/apps/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/apps/hadoop/share/hadoop/common/lib/$2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/apps/hadoop/lib/native
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-09 13:32:22,753 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-09 13:32:22,756 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfsuser
2020-04-09 13:32:22,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=node15:2181,node16:2181,node17:2181,node18:2181,node19:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.$2020-04-09 13:32:22,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node19/10.10.10.19:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-09 13:32:22,784 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node19/10.10.10.19:2181, initiating session
2020-04-09 13:32:22,817 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server node19/10.10.10.19:2181, sessionid = 0x50000a3038f0000, negotiated timeout = 10000
2020-04-09 13:32:22,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected.
2020-04-09 13:32:22,864 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 300, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, i$2020-04-09 13:32:22,888 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-04-09 13:32:22,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY
2020-04-09 13:32:23,049 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_HEALTHY
2020-04-09 13:32:23,074 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced...
2020-04-09 13:32:23,085 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0a6d792d636c757374657212036e6e321a06686466732d3420a84628d33e
2020-04-09 13:32:23,088 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hdfs-4/10.10.10.19:9000
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-4/10.10.10.19:9000 to standby state without fencing
2020-04-09 13:32:23,102 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/my-cluster/ActiveBreadCrumb to indicate that the local node is the most recent active...
2020-04-09 13:32:23,110 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hdfs-0/10.10.10.15:9000 active...
2020-04-09 13:32:23,759 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hdfs-0/10.10.10.15:9000 to active state
2020-04-09 13:38:59,910 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.io.EOFException: End of File Exception between local host is: "node15-hdfs-spark-master/10.10.10.15"; destination host is: "hdfs-0":9000; : java.io.EOFException; For more details see:  http://wiki.apache.org/ha$
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:791)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
        at org.apache.hadoop.ipc.Client.call(Client.java:1491)
        at org.apache.hadoop.ipc.Client.call(Client.java:1388)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
        at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
        at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
        at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
        at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
        at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1850)
        at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1183)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079)
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,913 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hdfs-0/10.10.10.15:9000 entered state: SERVICE_NOT_RESPONDING
2020-04-09 13:38:59,938 WARN org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Can't get local NN thread dump due to Connexion refusée (Connection refused)
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hdfs-0/10.10.10.15:9000 and marking that fencing is necessary
2020-04-09 13:38:59,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ZooKeeper: Session: 0x50000a3038f0000 closed
2020-04-09 13:38:59,947 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x50000a3038f0000
2020-04-09 13:38:59,947 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x50000a3038f0000
2020-04-09 13:39:01,951 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=10$2020-04-09 13:39:01,953 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see:  http://wiki.apache.org/ha$
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
        at org.apache.hadoop.ipc.Client.call(Client.java:1491)
        at org.apache.hadoop.ipc.Client.call(Client.java:1388)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
        at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
        at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
        at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
        at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
        at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
        at org.apache.hadoop.ipc.Client.call(Client.java:1435)
        ... 8 more
2020-04-09 13:39:03,956 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hdfs-0/10.10.10.15:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=10$2020-04-09 13:39:03,958 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hdfs-0/10.10.10.15:9000
java.net.ConnectException: Call From node15-hdfs-spark-master/10.10.10.15 to hdfs-0:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see:  http://wiki.apache.org/ha$
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
        at org.apache.hadoop.ipc.Client.call(Client.java:1491)
        at org.apache.hadoop.ipc.Client.call(Client.java:1388)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
        at com.sun.proxy.$Proxy9.getServiceStatus(Unknown Source)
        at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:136)
        at org.apache.hadoop.ha.HealthMonitor.doHealthChecks(HealthMonitor.java:202)
        at org.apache.hadoop.ha.HealthMonitor.access$600(HealthMonitor.java:49)
        at org.apache.hadoop.ha.HealthMonitor$MonitorDaemon.run(HealthMonitor.java:296)
Caused by: java.net.ConnectException: Connexion refusée
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
        at org.apache.hadoop.ipc.Client.call(Client.java:1435)
        ... 8 more
...
...
...

1 Ответ

0 голосов
/ 14 апреля 2020

Проблема с моей конфигурацией, наконец, возникла из-за двух команд, которые не были установлены при установке кластера oop:

  • , во-первых, команда n c: исправлена ​​путем установки пакет nmap из yum
  • , затем команда fuser: исправлено путем установки пакета psmis c из yum 
yum install -y nmap.x86_64
yum install -y psmisc.x86_64

Надеюсь, это скоро поможет другим из вас.

...