Presto-Glue-EMR 集成：presto-cli 给出 NullPointerException

Question

我正在尝试将我的 Glue 目录连接到 EMR 中的 Presto 和 Hive。在 presto-cli 中运行查询时，我得到 NullPointerException 而 相同的查询在 hive-cli 中成功。

像下面这样启动cli

presto-cli --catalog hive

执行查询时出现异常：

Query 20180814_174636_00003_iika5 failed: java.lang.NullPointerException: parameters is null

EMR 配置如下所示：

[
    {
        "classification": "presto-connector-hive",
        "properties": {
            "hive.metastore": "glue"
        },
        "configurations": []
    },
    {
        "classification": "hive-site",
        "properties": {
            "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
        },
        "configurations": []
    }
]

EMR 版本： 5.16.0 Presto 版本： 0.203

参考文档：https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto-glue.html

调试日志

Query 20180816_060942_00001_m9i52 failed: java.lang.NullPointerException: parameters is null
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException: parameters is null
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2052)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3943)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3967)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4952)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4958)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.get(CachingHiveMetastore.java:207)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.getPartitionNamesByParts(CachingHiveMetastore.java:499)
    at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.doGetPartitionNames(SemiTransactionalHiveMetastore.java:467)
    at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.getPartitionNamesByParts(SemiTransactionalHiveMetastore.java:445)
    at com.facebook.presto.hive.HivePartitionManager.getFilteredPartitionNames(HivePartitionManager.java:284)
    at com.facebook.presto.hive.HivePartitionManager.getPartitions(HivePartitionManager.java:146)
    at com.facebook.presto.hive.HiveMetadata.getTableLayouts(HiveMetadata.java:1305)
    at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.getTableLayouts(ClassLoaderSafeConnectorMetadata.java:73)
    at com.facebook.presto.metadata.MetadataManager.getLayouts(MetadataManager.java:346)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout.planTableScan(PickTableLayout.java:203)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout.access0(PickTableLayout.java:61)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout$PickTableLayoutWithoutPredicate.apply(PickTableLayout.java:186)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout$PickTableLayoutWithoutPredicate.apply(PickTableLayout.java:153)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.transform(IterativeOptimizer.java:168)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreNode(IterativeOptimizer.java:141)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:104)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:193)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:106)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:193)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:106)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.optimize(IterativeOptimizer.java:95)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:140)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:129)
    at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:327)
    at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:312)
    at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:268)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: parameters is null
    at java.util.Objects.requireNonNull(Objects.java:228)
    at com.facebook.presto.hive.metastore.Partition.<init>(Partition.java:54)
    at com.facebook.presto.hive.metastore.Partition$Builder.build(Partition.java:180)
    at com.facebook.presto.hive.metastore.glue.converter.GlueToPrestoConverter.convertPartition(GlueToPrestoConverter.java:141)
    at com.facebook.presto.hive.metastore.glue.GlueHiveMetastore.lambda$getPartitions(GlueHiveMetastore.java:558)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at com.facebook.presto.hive.metastore.glue.GlueHiveMetastore.getPartitions(GlueHiveMetastore.java:558)
    at com.facebook.presto.hive.metastore.glue.GlueHiveMetastore.getPartitionNamesByParts(GlueHiveMetastore.java:541)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadPartitionNamesByParts(CachingHiveMetastore.java:504)
    at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
    at com.google.common.cache.CacheLoader.load(CacheLoader.java:188)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2273)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2156)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2046)
    ... 33 more

Answer 1

presto 0.203 似乎有这个错误，我也遇到过，我切换到更新的版本并且它有效。

在我写这个答案的时候，EMR 5.17 已经发布，它的 presto 0.206 已经解决了这个问题。

Presto-Glue-EMR 集成：presto-cli 给出 NullPointerException

Presto-Glue-EMR integration: presto-cli giving NullPointerException

amazon-emr

presto

aws-glue

trino