AWS glue 端点和端口转发以及 spark 程序抛出错误
AWS glue endpoint and port forwarding and spark program throws error
我一直在努力寻找 运行ning spark program in zeppelin notebook 问题的解决方案。不确定出了什么问题
我正在使用 zeppelin zeppelin-0.7.3-bin-all
我已经创建了 AWS glue 端点和端口转发。
点击这些链接对我没有任何帮助
https://gist.github.com/codspire/7b0955b9e67fe73f6118dad9539cbaa2
https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-local-notebook.html
当我运行一段spark代码在http://localhost:8080/
%pyspark
a=5*4
print("value = %i" % (a))
sc.version
出现以下错误
org.apache.thrift.transport.TTransportException at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:266)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:250)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:373)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:406)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) at java.util.concurrent.FutureTask.run(Unknown Source) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access1(Unknown
Source) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) at java.lang.Thread.run(Unknown Source)
请帮忙！
当您创建端点时，您需要使用与您的 Zeppelin 兼容的 Glue 版本来创建它。在你的情况下 (Zeppelin 0.7.3) 你需要 Glue Version 0.9.
我一直在努力寻找 运行ning spark program in zeppelin notebook 问题的解决方案。不确定出了什么问题
我正在使用 zeppelin zeppelin-0.7.3-bin-all 我已经创建了 AWS glue 端点和端口转发。
点击这些链接对我没有任何帮助 https://gist.github.com/codspire/7b0955b9e67fe73f6118dad9539cbaa2 https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-local-notebook.html
当我运行一段spark代码在http://localhost:8080/
%pyspark
a=5*4
print("value = %i" % (a))
sc.version
出现以下错误
org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:266) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:250) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:373) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:406) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access1(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
请帮忙！
当您创建端点时，您需要使用与您的 Zeppelin 兼容的 Glue 版本来创建它。在你的情况下 (Zeppelin 0.7.3) 你需要 Glue Version 0.9.