运行 sqoop 导入时出错 - 错误 manager.OracleManager：无法回滚事务

Question

我正在使用 sqoop 将数据从 oracle db 加载到 hive

规格：

CDH-5.5.1 Sqoop 1.4.2

初始化：

    export SQOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop
    export HIVE_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive
    export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc7.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive/lib/*
    export HADOOP_USER_NAME=hdfs
    export PATH=$PATH:$HIVE_HOME/bin

Sqoop 命令：

sqoop import --connect jdbc:oracle:thin:@<host>/<pluggable DB SID> --username username --password password --table tablename -m 1 --hive-import --hive-table hivetablename

执行时出现如下错误

    16/06/01 16:11:37 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM VEHICLEDETAILS t WHERE 1=0
    16/06/01 16:11:37 WARN manager.SqlManager: SQLException closing ResultSet: java.sql.SQLException: Could not commit with auto-commit set on
    16/06/01 16:11:37 ERROR manager.OracleManager: Failed to rollback transaction
    java.sql.SQLException: Could not rollback with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2423)
        at org.apache.sqoop.manager.OracleManager.getColumnNames(OracleManager.java:767)
        at org.apache.sqoop.orm.ClassWriter.getColumnNames(ClassWriter.java:1207)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1062)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:37 ERROR manager.OracleManager: Failed to list columns
    java.sql.SQLException: Could not commit with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2356)
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2403)
        at org.apache.sqoop.manager.OracleManager.getColumnNames(OracleManager.java:764)
        at org.apache.sqoop.orm.ClassWriter.getColumnNames(ClassWriter.java:1207)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1062)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:37 INFO orm.CompilationManager: HADOOP_HOME is /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/bin/../lib/sqoop/../hadoop
    Note: /tmp/sqoop-root/compile/1d122ab0d74da045fd45da7a765f595d/VEHICLEDETAILS.java uses or overrides a deprecated API.
    Note: Recompile with -Xlint:deprecation for details.
    16/06/01 16:11:38 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/1d122ab0d74da045fd45da7a765f595d/VEHICLEDETAILS.jar
    16/06/01 16:11:38 ERROR manager.OracleManager: Failed to rollback transaction
    java.lang.NullPointerException
        at org.apache.sqoop.manager.OracleManager.getPrimaryKey(OracleManager.java:828)
        at org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:408)
        at org.apache.sqoop.manager.SqlManager.checkTableImportOptions(SqlManager.java:427)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:462)
        at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:38 ERROR manager.OracleManager: Failed to list columns
    java.sql.SQLException: Could not rollback with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2423)
        at org.apache.sqoop.manager.OracleManager$ConnCache.getConnection(OracleManager.java:190)
        at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:283)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.OracleManager.getPrimaryKey(OracleManager.java:814)
        at org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:408)
        at org.apache.sqoop.manager.SqlManager.checkTableImportOptions(SqlManager.java:427)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:462)
        at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:39 INFO manager.OracleManager: Time zone has been set to GMT
    16/06/01 16:11:39 ERROR manager.OracleManager: Failed to rollback transaction
    java.sql.SQLException: Could not rollback with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2423)
        at org.apache.sqoop.manager.OracleManager.getPrimaryKey(OracleManager.java:828)
        at org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:408)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:464)
        at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:39 ERROR manager.OracleManager: Failed to list columns
    java.sql.SQLException: Could not commit with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2356)
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2403)
        at org.apache.sqoop.manager.OracleManager.getPrimaryKey(OracleManager.java:825)
        at org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:408)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:464)
        at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:39 INFO mapreduce.ImportJobBase: Beginning import of VEHICLEDETAILS
    16/06/01 16:11:39 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
    16/06/01 16:11:39 ERROR manager.OracleManager: Failed to rollback transaction
    java.lang.NullPointerException
        at org.apache.sqoop.manager.OracleManager.getColumnNames(OracleManager.java:767)
        at org.apache.sqoop.mapreduce.DataDrivenImportJob.configureInputFormat(DataDrivenImportJob.java:164)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:194)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
        at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:39 ERROR manager.OracleManager: Failed to list columns
    java.sql.SQLException: Could not rollback with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2423)
        at org.apache.sqoop.manager.OracleManager$ConnCache.getConnection(OracleManager.java:190)
        at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:283)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.OracleManager.getColumnNames(OracleManager.java:751)
        at org.apache.sqoop.mapreduce.DataDrivenImportJob.configureInputFormat(DataDrivenImportJob.java:164)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:194)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
        at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
    16/06/01 16:11:39 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    16/06/01 16:11:39 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
    16/06/01 16:11:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    16/06/01 16:11:40 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/tmp/hadoop-root/mapred/staging/hdfs1761993336/.staging/job_local1761993336_0001
    Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
        at org.apache.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:53)
        at com.cloudera.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:36)
        at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:121)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:304)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:321)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:199)
        at org.apache.hadoop.mapreduce.Job.run(Job.java:1307)
        at org.apache.hadoop.mapreduce.Job.run(Job.java:1304)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
        at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
        at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

我正在使用 ojdbc 7 jar 和 oracle 12c 数据库和 jdk 8

Answer 1

使用"--query select * from VEHICLEDETAILS" instead of "--table VEHICLEDETAILS"

看到这个Posthttps://discuss.zendesk.com/hc/en-us/articles/204206528-Sqoop-import-fails-to-import-oracle-table-with-Could-not-commit-with-auto-commit-set-on-

Answer 2

我使用以下查询将数据从 mysql 导入配置单元 table。它也应该适用于 Oracle。只需要修改jdbcurl

sqoop import --connect jdbc:mysql://localhost:3306/retail_db --username retail_dba -P --table customers --target-dir /temp1 --hive-table Whosebug --hive-import

如果有帮助请告诉我

运行 sqoop 导入时出错 - 错误 manager.OracleManager：无法回滚事务

Error while running sqoop import - ERROR manager.OracleManager: Failed to rollback transaction

hadoop

hive

sqoop