使用 WEBHDFS 配置 Apache KNOX 网关的问题

Question

我已经使用 Apache Ambari 安装了 Apache KNOX。按照以下 link 使用 KNOX.KNOX 配置 WEBHDFS 安装在 address1 中。 https://knox.apache.org/books/knox-0-13-0/user-guide.html#WebHDFS

使用 curl 命令调用 webhdfs 时 curl -i -k -u guest:guest-password -X GET 'https://address1:8443/gateway/Andromd/webhdfs/v1/?op=LISTSTATUS'

它抛出以下错误

<title> 404 Not Found</title>
<p> Problem accessing /gateway//Andromd/webhdfs/v1/build-version.Reason:
<pre> Not Found </pre></p>

Andromd为集群名，添加在{GATEWAY_HOME}/conf/topologies/Andromd.xml下。配置如下

<service>
    <role>NAMENODE</role>
    <url>hdfs://address2:8020</url>
</service>
<service>
    <role>WEBHDFS</role>
    <url>http://address2:50070/webhdfs</url>
</service

/etc/hadoop/conf/hdfs-site.xml内容如下

<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>
<property>
    <name>dfs.namenode.rpc-address</name>
    <value>address2:8020</value>
</property>
<property>
    <name>dfs.namenode.http-address</name>
    <value>address2:50070</value>
</property>
<property>
    <name>dfs.https.namenode.https-address</name>
    <value>address2:50470</value>
</property>

请告诉我，关于配置，我这边是否遗漏了什么。

Gateway.log内容：

2017-10-13 18:24:24,586 ERROR digester3.Digester (Digester.java:parse(1652)) - An error occurred while parsing XML from '(already loaded from stream)', see nested exceptions
org.xml.sax.SAXParseException; lineNumber: 88; columnNumber: 18; The content of elements must consist of well-formed character data or markup.
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
        at org.apache.commons.digester3.Digester.parse(Digester.java:1642)
        at org.apache.commons.digester3.Digester.parse(Digester.java:1701)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.loadTopologyAttempt(DefaultTopologyService.java:124)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.loadTopology(DefaultTopologyService.java:100)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.loadTopologies(DefaultTopologyService.java:233)
        at org.apache.hadoop.gateway.services.topology.impl.DefaultTopologyService.reloadTopologies(DefaultTopologyService.java:318)
        at org.apache.hadoop.gateway.GatewayServer.start(GatewayServer.java:312)
        at org.apache.hadoop.gateway.GatewayServer.startGateway(GatewayServer.java:231)
        at org.apache.hadoop.gateway.GatewayServer.main(GatewayServer.java:114)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70)
        at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39)
        at org.apache.hadoop.gateway.launcher.Command.run(Command.java:101)
        at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69)
        at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46)
2017-10-13 18:24:24,587 ERROR hadoop.gateway (DefaultTopologyService.java:loadTopologies(250)) - Failed to load topology /usr/hdp/2.4.2.0-258/knox/bin/../conf/topologies/Andromeda.xml: org.xml.sax.SAXParseException; lineNumber: 88; columnNumber: 18; The content of elements must consist of well-formed character data or markup.
2017-10-13 18:24:24,588 INFO  hadoop.gateway (GatewayServer.java:handleCreateDeployment(450)) - Loading topology admin from /usr/hdp/2.4.2.0-258/knox/bin/../data/deployments/admin.war.15f15c171c0
2017-10-13 18:24:24,793 INFO  hadoop.gateway (GatewayServer.java:start(315)) - Monitoring topologies in directory: /usr/hdp/2.4.2.0-258/knox/bin/../conf/topologies
2017-10-13 18:24:24,795 INFO  hadoop.gateway (GatewayServer.java:startGateway(232)) - Started gateway on port 8,443.

添加到上述查询中，有必要在 Andromd.xml 中包含以下标签，方法是将 sandbox.hortonworks.com 替换为我们的集群特定信息。

<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>
<property>
    <name>dfs.namenode.rpc-address</name>
    <value>sandbox.hortonworks.com:8020</value>
</property>
<property>
    <name>dfs.namenode.http-address</name>
    <value>sandbox.hortonworks.com:50070</value>
</property>
<property>
    <name>dfs.https.namenode.https-address</name>
    <value>sandbox.hortonworks.com:50470</value>
</property>

Answer 1

2017-10-13 18:24:24,587 ERROR hadoop.gateway (DefaultTopologyService.java:loadTopologies(250)) - Failed to load topology /usr/hdp/2.4.2.0-258/knox/bin/../conf/topologies/Andromeda.xml: org.xml.sax.SAXParseException; lineNumber: 88; columnNumber: 18; The content of elements must consist of well-formed character data or markup.

查看此行，您的拓扑文件 Andromeda.xml 格式不正确（缺少结束标记或类似标记），行号：88。

这就是您收到 404 的原因，因为未部署拓扑。修复拓扑文件后检查日志，确保没有启动错误。

使用 WEBHDFS 配置 Apache KNOX 网关的问题

Issue configuring Apache KNOX gateway with WEBHDFS

hadoop

knox-gateway