SQOOP 导出：java.sql.BatchUpdateException：参数 #1 尚未设置

Question

我正在尝试将 sqoop 导出到 sql 服务器并使用以下命令：

sqoop-export -D mapred.job.name=SQOOP:TEST:UPSERT -Dsqoop.export.records.per.statement=1 --connect $CONN --username $USER --password $PWD --table table_test_1 --hcatalog-database default --hcatalog-table test_1 --input-fields-terminated-by '[=12=]1' --input-null-string '\N' --update-key ID --update-mode allowinsert --verbose --columns "name,id,age"

我收到以下错误。不知道为什么它说 parameter# 1 is not set。如果通过参数1，sqoop引用了update-key，这里是我的id，在我的hivetable中不为null。我正在使用 sqoop 1.4

16/08/12 11:43:06 INFO mapreduce.Job: Task Id : attempt_1470577803876_1072_m_000000_2, Status : FAILED
Error: java.io.IOException: java.sql.BatchUpdateException: Parameter #1 has not been set.
        at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
        at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
        at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658)
        at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
        at org.apache.sqoop.mapreduce.hcat.SqoopHCatExportMapper.map(SqoopHCatExportMapper.java:56)
        at org.apache.sqoop.mapreduce.hcat.SqoopHCatExportMapper.map(SqoopHCatExportMapper.java:35)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
        at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.sql.BatchUpdateException: Parameter #1 has not been set.
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeBatch(JtdsStatement.java:1099)
        at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:231)

16/08/12 11:43:20 INFO mapreduce.Job:  map 100% reduce 0%
16/08/12 11:43:20 INFO mapreduce.Job: Job job_1470577803876_1072 failed with state FAILED due to: Task failed task_1470577803876_1072_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/08/12 11:43:20 INFO mapreduce.Job: Counters: 9
        Job Counters
                Failed map tasks=4
                Launched map tasks=4
                Other local map tasks=3
                Rack-local map tasks=1

任何帮助。

Answer 1

问题已解决。这是因为 sqoop export 语句中没有指定驱动程序和连接管理器信息。 Sqoop 使用的 jar 也不支持 upsert to target 。

使用支持 upsert 的自定义 jar 到目标并且它有效。

sqoop-export -D mapred.job.name=SQOOP:TEST:UPSERT -Dsqoop.export.records.per.statement=1 --connect $CONN --username $USER --password $PWD --table table_test_1 --hcatalog-database default --hcatalog-table test_1 --input-fields-terminated-by '[=10=]1' --input-null-string '\N' --update-key ID --update-mode allowinsert --verbose --columns "name,id,age" --driver net.sourceforge.jtds.jdbc.Driver --connection-manager com.custom.sqoop.manager.SQLServerManager

SQOOP 导出：java.sql.BatchUpdateException：参数 #1 尚未设置

SQOOP EXPORT : java.sql.BatchUpdateException: Parameter #1 has not been set

sql-server

hadoop

hive

sqoop