使用kafka和HBase引起的冲突
Conflicts caused by the use of kafka and HBase
当我使用"spark streaming"读取"kafka"（需要sasl验证）然后将数据存储到"HBase"时，"HBase"给出了以下错误
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:181)
at com.xueersi.datamining.ups.database.implement.HbaseClient.connect(HbaseClient.scala:91)
at com.xueersi.datamining.ups.stream.start.BaseInfoLogAnalysisStart$$anonfun$main$$anonfun$apply.apply(BaseInfoLogAnalysisStart.scala:78)
at com.xueersi.datamining.ups.stream.start.BaseInfoLogAnalysisStart$$anonfun$main$$anonfun$apply.apply(BaseInfoLogAnalysisStart.scala:75)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$$anonfun$apply.apply(RDD.scala:925)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$$anonfun$apply.apply(RDD.scala:925)
at org.apache.spark.SparkContext$$anonfun$runJob.apply(SparkContext.scala:1956)
at org.apache.spark.SparkContext$$anonfun$runJob.apply(SparkContext.scala:1956)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
... 15 more
Caused by: java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:931)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:658)
... 20 more
Caused by: java.lang.NullPointerException
at org.apache.kafka.common.security.plain.PlainSaslServer$PlainSaslServerFactory.getMechanismNames(PlainSaslServer.java:163)
at org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.(SaslRpcServer.java:381)
at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:186)
at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:570)
at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:418)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:314)
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:68)
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:152)
at org.apache.hadoop.io.retry.RetryInvocationHandler.(RetryInvocationHandler.java:75)
at org.apache.hadoop.io.retry.RetryInvocationHandler.(RetryInvocationHandler.java:66)
at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:58)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:181)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:762)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:693)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:158)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2816)
at org.apache.hadoop.fs.FileSystem.access0(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:186)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:120)
at org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.(ProtobufUtil.java:246)
... 25 more
但是当我读另一个"Kafka"（没有sasl验证）时，“HBase.
另外，"kerberos"认证需要"HBase"
我觉得kafka的sasl认证和hbase的kerberos认证有冲突
有没有人可以给我一些建议？
我好像找到答案了：https://issues.apache.org/jira/browse/KAFKA-5294
然后我手动指定依赖（我以前用的版本是0.10.2.1）
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.11.0.0</version>
<scope>compile</scope>
</dependency>
有效
当我使用"spark streaming"读取"kafka"（需要sasl验证）然后将数据存储到"HBase"时，"HBase"给出了以下错误
java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:181) at com.xueersi.datamining.ups.database.implement.HbaseClient.connect(HbaseClient.scala:91) at com.xueersi.datamining.ups.stream.start.BaseInfoLogAnalysisStart$$anonfun$main$$anonfun$apply.apply(BaseInfoLogAnalysisStart.scala:78) at com.xueersi.datamining.ups.stream.start.BaseInfoLogAnalysisStart$$anonfun$main$$anonfun$apply.apply(BaseInfoLogAnalysisStart.scala:75) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$$anonfun$apply.apply(RDD.scala:925) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$$anonfun$apply.apply(RDD.scala:925) at org.apache.spark.SparkContext$$anonfun$runJob.apply(SparkContext.scala:1956) at org.apache.spark.SparkContext$$anonfun$runJob.apply(SparkContext.scala:1956) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ... 15 more Caused by: java.lang.ExceptionInInitializerError at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64) at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:931) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:658) ... 20 more Caused by: java.lang.NullPointerException at org.apache.kafka.common.security.plain.PlainSaslServer$PlainSaslServerFactory.getMechanismNames(PlainSaslServer.java:163) at org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.(SaslRpcServer.java:381) at org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:186) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:570) at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:418) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:314) at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider$DefaultProxyFactory.createProxy(ConfiguredFailoverProxyProvider.java:68) at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:152) at org.apache.hadoop.io.retry.RetryInvocationHandler.(RetryInvocationHandler.java:75) at org.apache.hadoop.io.retry.RetryInvocationHandler.(RetryInvocationHandler.java:66) at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:58) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:181) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:762) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:693) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:158) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2816) at org.apache.hadoop.fs.FileSystem.access0(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:186) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:120) at org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.(ProtobufUtil.java:246) ... 25 more
但是当我读另一个"Kafka"（没有sasl验证）时，“HBase. 另外，"kerberos"认证需要"HBase" 我觉得kafka的sasl认证和hbase的kerberos认证有冲突 有没有人可以给我一些建议？
我好像找到答案了：https://issues.apache.org/jira/browse/KAFKA-5294
然后我手动指定依赖（我以前用的版本是0.10.2.1）
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.11.0.0</version>
<scope>compile</scope>
</dependency>
有效