Hadoop

Question

当我导出到 jar 文件并在 Hadoop 节点上运行时，我在 eclipse 项目中有以下代码

        String hdrsUrl = "hdfs://servername:8020";
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", hdrsUrl);
        FileSystem fs = FileSystem.get(conf);

我收到以下错误

 15/04/17 08:21:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation
        at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access0(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
        at com.express.BrierleyProcessor.App.main(App.java:33)
        ... 5 more

虽然，当我在节点上创建一个 java 文件时，编译它，设置 class 路径，然后执行它没有问题。

我错过了什么？

Answer 1

当运行命令时：

[user@server bin]$  hadoop version

Hadoop 2.6.0.2.2.0.0-2041
Subversion git@github.com:hortonworks/hadoop.git -r 7d56f02902b436d46efba030651a2fbe7c1cf1e9
Compiled by jenkins on 2014-11-19T19:42Z
Compiled with protoc 2.5.0
From source with checksum f0c0406cc910a79f206d2ee4c2a68773
This command was run using /usr/hdp/2.2.0.0-2041/hadoop/hadoop-common-2.6.0.2.2.0.0-2041.jar

显示的版本号（2.6.0.2.2.0.0-2041）含义如下：

2.6.0 - hadoop 版本

2.2.0.0-2041 - HDP 版本。

因此，您应该使用 2.2 版的 hadoop jar 文件，而不是 2.6 版

Hadoop - UnsupportedOperationException：分布式文件系统未实现

Hadoop - UnsupportedOperationException: Not implemented by the DistributedFileSystem

java

hortonworks-data-platform