使用 logstash 将 csv 文件传输到服务器
using logstash to transport csv file to server
我目前正在学习使用大数据，
客户端：Windows 10
服务器端：CentOS7
服务器端已经安装了elasticsearch、logstash和kibana，
作为客户端，我可以通过 http://10.64.2.246:5601/ 访问 kibana，一切都很酷。现在我想了解如何将数据从我的电脑传输到服务器端的 elasticsearch。我已经在客户端下载了 logstash。
我准备了来自 kaggle 的 2 列数据。
我制作了我的 conf 文件
配置文件
input {
file {
path => "C:\Users\Charles\Desktop\DATA"
start_position => "beginning"
sincedb_path=>"/dev/null"
}
}
filter {
csv {
separator=>","
columns=>["Date","Price"]
}
mutate {convert =>["Price","float"]}
}
output{
elasticsearch{
host => "10.64.2.246"
index=>"oil"
document_type=>"minyaks"
}
stdout{}
}
这是我在客户端执行的 conf 文件的代码。
bin/logstash -f /Users/Charles/Desktop/DATA/logstash_oil.conf
这是显示的错误
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logs which is now configured via log4j2.properties
[2019-11-07T20:33:11,077][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-11-07T20:33:11,090][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-11-07T20:33:12,416][INFO ][org.reflections.Reflections] Reflections took 29 ms to scan 1 urls, producing 20 keys and 40 values
[2019-11-07T20:33:12,917][ERROR][logstash.outputs.elasticsearch] Unknown setting 'host' for elasticsearch
[2019-11-07T20:33:12,925][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:100)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i[=11=]$initialize.call(JavaBasePipelineExt$INVOKER$i[=11=]$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1156)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1143)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize[=11=](C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/java_pipeline.rb:26)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute[=11=](C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute[=11=]$__VARARGS__(C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state(C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Unknown Source)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
create at org/logstash/execution/ConvergeResultExt.java:109
add at org/logstash/execution/ConvergeResultExt.java:37
converge_state at C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:339
[2019-11-07T20:33:12,932][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[2019-11-07T20:33:12,954][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[2019-11-07T20:33:13,004][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
需要有人向我解释一下。我按照 youtube 的说明进行操作，但讲师在本地进行。
谢谢。
问题出在这一行：
[2019-11-07T20:33:12,917][ERROR][logstash.outputs.elasticsearch] Unknown setting 'host' for elasticsearch
所以你需要改变
host => "10.64.2.246"
到
hosts => "10.64.2.246"
此外，由于您使用的似乎是 ES 7.4.2，因此您可以删除以下行作为 document/mapping types are deprecated
document_type=>"minyaks"
我目前正在学习使用大数据， 客户端：Windows 10 服务器端：CentOS7 服务器端已经安装了elasticsearch、logstash和kibana， 作为客户端，我可以通过 http://10.64.2.246:5601/ 访问 kibana，一切都很酷。现在我想了解如何将数据从我的电脑传输到服务器端的 elasticsearch。我已经在客户端下载了 logstash。
我准备了来自 kaggle 的 2 列数据。 我制作了我的 conf 文件
配置文件
input {
file {
path => "C:\Users\Charles\Desktop\DATA"
start_position => "beginning"
sincedb_path=>"/dev/null"
}
}
filter {
csv {
separator=>","
columns=>["Date","Price"]
}
mutate {convert =>["Price","float"]}
}
output{
elasticsearch{
host => "10.64.2.246"
index=>"oil"
document_type=>"minyaks"
}
stdout{}
}
这是我在客户端执行的 conf 文件的代码。
bin/logstash -f /Users/Charles/Desktop/DATA/logstash_oil.conf
这是显示的错误
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logs which is now configured via log4j2.properties
[2019-11-07T20:33:11,077][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-11-07T20:33:11,090][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-11-07T20:33:12,416][INFO ][org.reflections.Reflections] Reflections took 29 ms to scan 1 urls, producing 20 keys and 40 values
[2019-11-07T20:33:12,917][ERROR][logstash.outputs.elasticsearch] Unknown setting 'host' for elasticsearch
[2019-11-07T20:33:12,925][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:100)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i[=11=]$initialize.call(JavaBasePipelineExt$INVOKER$i[=11=]$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1156)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1143)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize[=11=](C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/java_pipeline.rb:26)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute[=11=](C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute[=11=]$__VARARGS__(C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "C_3a_.Users.Charles.Desktop.DATA.logstash_minus_7_dot_4_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state(C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Unknown Source)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
create at org/logstash/execution/ConvergeResultExt.java:109
add at org/logstash/execution/ConvergeResultExt.java:37
converge_state at C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:339
[2019-11-07T20:33:12,932][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[2019-11-07T20:33:12,954][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "C:/Users/Charles/Desktop/DATA/logstash-7.4.2/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[2019-11-07T20:33:13,004][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
需要有人向我解释一下。我按照 youtube 的说明进行操作，但讲师在本地进行。
谢谢。
问题出在这一行：
[2019-11-07T20:33:12,917][ERROR][logstash.outputs.elasticsearch] Unknown setting 'host' for elasticsearch
所以你需要改变
host => "10.64.2.246"
到
hosts => "10.64.2.246"
此外，由于您使用的似乎是 ES 7.4.2，因此您可以删除以下行作为 document/mapping types are deprecated
document_type=>"minyaks"