Fiware - Cygnus 到 Cosmos，我无法将数据上传到 HDFS
Fiware - Cygnus to Cosmos, I can not upload data to HDFS
我有一个带 Cygnus 的 Orion 实例，订阅和通知工作正常，但我无法从我的实例发送文件到 cosmos.lab.fi-ware.org。
[错误 - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] http://130.206.80.46:14000 的连接被拒绝
我的cygnus.conf：
# APACHE_FLUME_HOME/conf/cygnus.conf
# The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
# Regarding sinks, you can use multiple ones at the same time; the only requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sinks at the same time).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default organization (organization semantic depend on the persistence sink)
cygnusagent.sources.http-source.handler.default_organization = org42
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type =
es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# The FQDN/IP address of the Cosmos deployment where the notification events will be persisted
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = myUsername
# default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = **********
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix =
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
错误日志：
2015-02-04 22:52:28,627 (lifecycleSupervisor-1-1)
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)]
HttpFS operation: PUT 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/?op=mkdirs&user.name=maxime.mularz HTTP/1.1
2015-02-04 22:53:31,690 (lifecycleSupervisor-1-1)
[ERROR -.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)]
Connection to http://130.206.80.46:14000 refused
2015-02-04 22:56:02,182 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.persist(OrionHDFSSink.java:212)]
Persisting data. File: Room1-Room-temperature-float.txt, Data: 2015- 02-04T22:56:02.182|1423086962|Room1|Room|temperature|float|90)
2015-02-04 22:56:02,183 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.exists(HttpFSBackend.java:158)]
HttpFS operation: GET 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/Room1-Room-temperature-float.txt?op=getfilestatus&user.name=maxime.mularz HTTP/1.1
提前致谢。
在启用从 Lanion 节点到西班牙节点的访问后已修复。
我有一个带 Cygnus 的 Orion 实例，订阅和通知工作正常，但我无法从我的实例发送文件到 cosmos.lab.fi-ware.org。
[错误 - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] http://130.206.80.46:14000 的连接被拒绝
我的cygnus.conf：
# APACHE_FLUME_HOME/conf/cygnus.conf
# The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
# Regarding sinks, you can use multiple ones at the same time; the only requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sinks at the same time).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default organization (organization semantic depend on the persistence sink)
cygnusagent.sources.http-source.handler.default_organization = org42
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type =
es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# The FQDN/IP address of the Cosmos deployment where the notification events will be persisted
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = myUsername
# default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = **********
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix =
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
错误日志：
2015-02-04 22:52:28,627 (lifecycleSupervisor-1-1)
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)]
HttpFS operation: PUT 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/?op=mkdirs&user.name=maxime.mularz HTTP/1.1
2015-02-04 22:53:31,690 (lifecycleSupervisor-1-1)
[ERROR -.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)]
Connection to http://130.206.80.46:14000 refused
2015-02-04 22:56:02,182 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.persist(OrionHDFSSink.java:212)]
Persisting data. File: Room1-Room-temperature-float.txt, Data: 2015- 02-04T22:56:02.182|1423086962|Room1|Room|temperature|float|90)
2015-02-04 22:56:02,183 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.exists(HttpFSBackend.java:158)]
HttpFS operation: GET 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/Room1-Room-temperature-float.txt?op=getfilestatus&user.name=maxime.mularz HTTP/1.1
提前致谢。
在启用从 Lanion 节点到西班牙节点的访问后已修复。