为什么在presto的Custom UDAF中没有找到return类型
Why did not found return type in Custom UDAF of presto
我制作的自定义函数在 EMR 上无法正常工作。
我想创建一个简单的 UDAF return 42。
首先，我的自定义函数我写了一个简单的函数，但在 presto 中没有工作。
presto-cli 中出现错误：
presto> select answer_to_life('the universe');
Query 20180324_120433_00000_7n6s6 failed: answer_to_life(varchar):bigint not found
com.facebook.presto.spi.PrestoException: answer_to_life(varchar):bigint not found
at com.facebook.presto.metadata.FunctionRegistry.doGetSpecializedFunctionKey(FunctionRegistry.java:972)
at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:146)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5153)
at com.facebook.presto.metadata.FunctionRegistry.getSpecializedFunctionKey(FunctionRegistry.java:898)
at com.facebook.presto.metadata.FunctionRegistry.getAggregateFunctionImplementation(FunctionRegistry.java:875)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.buildAccumulatorFactory(LocalExecutionPlanner.java:1973)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.planGlobalAggregation(LocalExecutionPlanner.java:1984)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:955)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:1919)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:952)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:638)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:393)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:324)
at com.facebook.presto.execution.SqlTaskExecution.<init>(SqlTaskExecution.java:161)
at com.facebook.presto.execution.SqlTaskExecution.createSqlTaskExecution(SqlTaskExecution.java:121)
at com.facebook.presto.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:71)
at com.facebook.presto.execution.SqlTask.updateTask(SqlTask.java:340)
at com.facebook.presto.execution.SqlTaskManager.updateTask(SqlTaskManager.java:321)
at com.facebook.presto.server.TaskResource.createOrUpdateTask(TaskResource.java:128)
at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static[=11=](ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:454)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
at org.eclipse.jetty.util.thread.QueuedThreadPool.run(QueuedThreadPool.java:591)
at java.lang.Thread.run(Thread.java:748)
AggregationFunction 代码如下。
我参考了这个presto code
package sample;
import com.facebook.presto.spi.block.BlockBuilder;
import com.facebook.presto.spi.function.*;
import com.facebook.presto.spi.type.BigintType;
import com.facebook.presto.spi.type.StandardTypes;
import io.airlift.slice.Slice;
@AggregationFunction("answer_to_life")
public final class AnswerToLife {
private AnswerToLife() {
}
@InputFunction
public static void input(@AggregationState NullState state, @SqlType(StandardTypes.VARCHAR) Slice value) {
}
@CombineFunction
public static void combine(@AggregationState NullState state, @AggregationState NullState other) {
}
@OutputFunction(StandardTypes.BIGINT)
public static void output(@AggregationState NullState state, BlockBuilder out) {
BigintType.BIGINT.writeLong(out, 42);
}
}
详细代码在这里 (https://github.com/asari-mtr/presto-udaf/tree/Whosebug)。
我部署的结构如下：
$ ls -1 /usr/lib/presto/plugin/my-udaf/
commons-codec-1.4.jar
guava-21.0.jar
hive-udf-1.0-SNAPSHOT.jar
presto-array-0.197.jar
stats-0.155.jar
presto-udaf-1.0-SNAPSHOT.jar
我用emr-5.12.0(Presto 0.188)
感谢您的宝贵时间。
已编辑1
jar 文件中的列表。
% jar -tf target/presto-udaf-1.0-SNAPSHOT.jar
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
sample/
META-INF/services/com.facebook.presto.spi.Plugin
sample/AnswerToLife.class
sample/AnswerToLifePlugin.class
sample/NullState.class
META-INF/maven/
META-INF/maven/sample/
META-INF/maven/sample/presto-udaf/
META-INF/maven/sample/presto-udaf/pom.xml
META-INF/maven/sample/presto-udaf/pom.properties
sever.log
$ grep answer -i /mnt/var/log/presto/server.log
2018-03-26T06:37:14.213Z INFO main com.facebook.presto.server.PluginManager Installing sample.AnswerToLifePlugin
2018-03-26T06:37:14.214Z INFO main com.facebook.presto.server.PluginManager Registering functions from sample.AnswerToLife
并执行show funcstions
presto> show functions;
Function | Return Type | Argument Types
---------------------------------+-----------------------------------------------------+------------------------------------------------------------------------------
ST_Area | double | Geometry
ST_AsText | varchar | Geometry
...
answer_to_life | bigint | varchar
...
已编辑2
删除src/main/resources/META-INF/services/com.facebook.presto.spi.Plugin
将<packaging>presto-plugin</packaging>添加到pom.xml
https://github.com/asari-mtr/presto-udaf/commit/f2a2ddbf0339e08f418b378a7ead511020e98a3b
我在 /usr/lib/presto/plugin/.
下部署了 Maven 制作的 zip
但是，错误的内容没有改变。
已编辑3
我在 Mac 上从 github（分支 0.188）获取源代码并构建了 presto。
当我们将上面的 UDAF 放在它的 presto 上时，它运行得很好。
可能是EMR上presto的安装过程有误
从您写的内容来看，缺少两件事。
- 确保您在 pom.xml
中使用了 maven 打包 presto-plugin
- 还实现了
com.facebook.presto.spi.Plugin，在 getFunctions 方法中 return 您的函数。
查看 presto-geospatial Presto 模块并查看 com.facebook.presto.plugin.geospatial.GeoPlugin 及其 pom.xml 作为参考。
解决了！！
自定义插件没有工作，因为它不是插件创建，但在 EMR 上部署的方式是错误的。
由于我没有部署到我执行的过程中的所有节点，我通过将以下 shell 注册为 bootstrap 操作来确认操作。
#/bin/bash
aws s3 cp s3://hogehoge/presto-udaf-1.0-SNAPSHOT.zip /tmp/
sudo mkdir -p /usr/lib/presto/plugin/
sudo unzip -d /usr/lib/presto/plugin/ /tmp/presto-udaf-1.0-SNAPSHOT.zip
Plugins must be installed on all nodes in the Presto cluster
(coordinator and workers).
You can use a bootstrap action to copy objects from Amazon S3 to each
node in a cluster before your applications are installed. The AWS CLI
is installed on each node of a cluster, so your bootstrap action can
call AWS CLI commands.
我制作的自定义函数在 EMR 上无法正常工作。
我想创建一个简单的 UDAF return 42。
首先，我的自定义函数我写了一个简单的函数，但在 presto 中没有工作。
presto-cli 中出现错误：
presto> select answer_to_life('the universe');
Query 20180324_120433_00000_7n6s6 failed: answer_to_life(varchar):bigint not found
com.facebook.presto.spi.PrestoException: answer_to_life(varchar):bigint not found
at com.facebook.presto.metadata.FunctionRegistry.doGetSpecializedFunctionKey(FunctionRegistry.java:972)
at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:146)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5153)
at com.facebook.presto.metadata.FunctionRegistry.getSpecializedFunctionKey(FunctionRegistry.java:898)
at com.facebook.presto.metadata.FunctionRegistry.getAggregateFunctionImplementation(FunctionRegistry.java:875)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.buildAccumulatorFactory(LocalExecutionPlanner.java:1973)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.planGlobalAggregation(LocalExecutionPlanner.java:1984)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:955)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:1919)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:952)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:638)
at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:596)
at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:393)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:324)
at com.facebook.presto.execution.SqlTaskExecution.<init>(SqlTaskExecution.java:161)
at com.facebook.presto.execution.SqlTaskExecution.createSqlTaskExecution(SqlTaskExecution.java:121)
at com.facebook.presto.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:71)
at com.facebook.presto.execution.SqlTask.updateTask(SqlTask.java:340)
at com.facebook.presto.execution.SqlTaskManager.updateTask(SqlTaskManager.java:321)
at com.facebook.presto.server.TaskResource.createOrUpdateTask(TaskResource.java:128)
at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static[=11=](ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:454)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
at org.eclipse.jetty.util.thread.QueuedThreadPool.run(QueuedThreadPool.java:591)
at java.lang.Thread.run(Thread.java:748)
AggregationFunction 代码如下。
我参考了这个presto code
package sample;
import com.facebook.presto.spi.block.BlockBuilder;
import com.facebook.presto.spi.function.*;
import com.facebook.presto.spi.type.BigintType;
import com.facebook.presto.spi.type.StandardTypes;
import io.airlift.slice.Slice;
@AggregationFunction("answer_to_life")
public final class AnswerToLife {
private AnswerToLife() {
}
@InputFunction
public static void input(@AggregationState NullState state, @SqlType(StandardTypes.VARCHAR) Slice value) {
}
@CombineFunction
public static void combine(@AggregationState NullState state, @AggregationState NullState other) {
}
@OutputFunction(StandardTypes.BIGINT)
public static void output(@AggregationState NullState state, BlockBuilder out) {
BigintType.BIGINT.writeLong(out, 42);
}
}
详细代码在这里 (https://github.com/asari-mtr/presto-udaf/tree/Whosebug)。
我部署的结构如下：
$ ls -1 /usr/lib/presto/plugin/my-udaf/
commons-codec-1.4.jar
guava-21.0.jar
hive-udf-1.0-SNAPSHOT.jar
presto-array-0.197.jar
stats-0.155.jar
presto-udaf-1.0-SNAPSHOT.jar
我用emr-5.12.0(Presto 0.188)
感谢您的宝贵时间。
已编辑1
jar 文件中的列表。
% jar -tf target/presto-udaf-1.0-SNAPSHOT.jar
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
sample/
META-INF/services/com.facebook.presto.spi.Plugin
sample/AnswerToLife.class
sample/AnswerToLifePlugin.class
sample/NullState.class
META-INF/maven/
META-INF/maven/sample/
META-INF/maven/sample/presto-udaf/
META-INF/maven/sample/presto-udaf/pom.xml
META-INF/maven/sample/presto-udaf/pom.properties
sever.log
$ grep answer -i /mnt/var/log/presto/server.log
2018-03-26T06:37:14.213Z INFO main com.facebook.presto.server.PluginManager Installing sample.AnswerToLifePlugin
2018-03-26T06:37:14.214Z INFO main com.facebook.presto.server.PluginManager Registering functions from sample.AnswerToLife
并执行show funcstions
presto> show functions;
Function | Return Type | Argument Types
---------------------------------+-----------------------------------------------------+------------------------------------------------------------------------------
ST_Area | double | Geometry
ST_AsText | varchar | Geometry
...
answer_to_life | bigint | varchar
...
已编辑2
删除src/main/resources/META-INF/services/com.facebook.presto.spi.Plugin
将<packaging>presto-plugin</packaging>添加到pom.xml
https://github.com/asari-mtr/presto-udaf/commit/f2a2ddbf0339e08f418b378a7ead511020e98a3b
我在 /usr/lib/presto/plugin/.
但是，错误的内容没有改变。
已编辑3
我在 Mac 上从 github（分支 0.188）获取源代码并构建了 presto。
当我们将上面的 UDAF 放在它的 presto 上时，它运行得很好。
可能是EMR上presto的安装过程有误
从您写的内容来看，缺少两件事。
- 确保您在 pom.xml 中使用了 maven 打包
- 还实现了
com.facebook.presto.spi.Plugin，在getFunctions方法中 return 您的函数。
presto-plugin
查看 presto-geospatial Presto 模块并查看 com.facebook.presto.plugin.geospatial.GeoPlugin 及其 pom.xml 作为参考。
解决了！！
自定义插件没有工作，因为它不是插件创建，但在 EMR 上部署的方式是错误的。
由于我没有部署到我执行的过程中的所有节点，我通过将以下 shell 注册为 bootstrap 操作来确认操作。
#/bin/bash
aws s3 cp s3://hogehoge/presto-udaf-1.0-SNAPSHOT.zip /tmp/
sudo mkdir -p /usr/lib/presto/plugin/
sudo unzip -d /usr/lib/presto/plugin/ /tmp/presto-udaf-1.0-SNAPSHOT.zip
Plugins must be installed on all nodes in the Presto cluster (coordinator and workers).
You can use a bootstrap action to copy objects from Amazon S3 to each node in a cluster before your applications are installed. The AWS CLI is installed on each node of a cluster, so your bootstrap action can call AWS CLI commands.