使用Hcatalog导入sqoop时遇到的权限问题

Question

我正在尝试使用带有 HCatalog 集成的 sqoop 导入将数据从 Teradata 提取到 Hive。下面是我的 sqoop 导入命令：

sqoop import -libjars /path/tdgssconfig.jar \
         -Dmapreduce.job.queuename=${queue} \
         -Dmapreduce.map.java.opts=-Xmx16g \
         -Dmapreduce.map.memory.mb=20480 \
         --driver com.teradata.jdbc.TeraDriver \
         --connect jdbc:teradata:<db-url>,charset=ASCII,LOGMECH=LDAP \
         --username ${srcDbUsr} \
         --password-file ${srcDbPassFile} \
         --verbose \
         --query "${query} AND $CONDITIONS" \
         --split-by ${splitBy} \
         --fetch-size ${fetchSize} \
         --null-string '\N' \
         --null-non-string '\N' \
         --fields-terminated-by , \
         --hcatalog-database ${tgtDbName} \
         --hcatalog-table ${tgtTblName} \
         --hcatalog-partition-keys ${partitionKey} \
         --hcatalog-partition-values "${partitionValue}"

我遇到了以下错误 - 将分区添加到 Metastore 时出错。权限被拒绝。:

18/07/03 12:14:02 INFO mapreduce.Job: Job job_1530241180113_6487 failed with state FAILED due to: Job commit failed: org.apache.hive.hcatalog.common.HCatException : 2006 : Error adding partition to metastore. Cause : org.apache.hadoop.security.AccessControlException: Permission denied. user=<usr-name> is not the owner of inode=<partition-key=partition-value>
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:195)
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:181)
    at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:178)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3560)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3543)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:3508)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6559)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermissionInt(FSNamesystem.java:1807)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1787)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:654)
    at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setPermission(AuthorizationProviderProxyClientProtocol.java:174)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2141)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2137)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2135)

    at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:969)
    at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:249)
    at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
    at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

如何解决这个权限问题？

Answer 1

解决了问题。 sqoop hcatalog 无法将文件添加到 Hive 内部 table，因为它驻留在 Hive 目录中并且所有者是 hive，而不是特定用户。解决方案是创建一个外部 table，以便底层目录将用户（而非配置单元）作为所有者。

使用Hcatalog导入sqoop时遇到的权限问题

Permission issue encountered during sqoop import using Hcatalog

import

hive

sqoop

hcatalog