shell 脚本中的 SQOOP 导出失败

Question

我在 shell 的帮助下将 table 从配置单元导出到 mysql script.The 下面是 sqoop 导出命令

sqoop export --connect jdbc:mysql://192.168.154.129:3306/ey -username root --table call_detail_records --export-dir /apps/hive/warehouse/xademo.db/call_detail_records --fields-terminated-by '|' --lines-terminated-by '\n' --m 4 --batch

上述命令在 CLI 中运行良好。但它在 shell 脚本中不起作用，它会生成以下警告和错误。

警告：

15/05/05 13:30:06 WARN sqoop.SqoopOptions: Character argument '|' has multiple characters; only the first will be used.
15/05/05 13:30:06 WARN sqoop.SqoopOptions: Character argument '\n' has multiple characters; only the first will be used.

错误：

15/05/05 13:30:50 INFO mapreduce.Job:  map 0% reduce 0%
15/05/05 13:31:56 INFO mapreduce.Job: Task Id : attempt_1430805361424_0046_m_000001_0, Status : FAILED
Error: java.io.IOException: Can't export data, please check failed map task logs
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.RuntimeException: Can't parse input data: 'PHONE_NUM|PLAN|DATE|STAUS|BALANCE|IMEI|REGION'
    at customer_details.__loadFromFields(customer_details.java:464)
    at customer_details.parse(customer_details.java:382)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
    ... 10 more
Caused by: java.util.NoSuchElementException
    at java.util.ArrayList$Itr.next(ArrayList.java:834)
    at customer_details.__loadFromFields(customer_details.java:434)
    ... 12 more

我在 shell 脚本中的 Sqoop 命令将包含将被扩展的变量。

nohup sqoop export --connect jdbc:mysql://192.168.154.129:3306/ey -username root --table $TBL_NAME --export-dir $HIVE_DIR --fields-terminated-by "$FIELD_SEP" --lines-terminated-by "'"'\'"$LINE_SEP""'" --m $NUM_MAPPERS --batch > $sqoop_outs/$TBL_NAME.out 2>&1 &

非常感谢任何帮助。我为此苦苦挣扎了很长时间...

Answer 1

最后我找到了原因，当我从 CLI 运行和 Shell 脚本时，SQOOP 命令中 " 和 ' 的不同处理。

解决方案：我必须按如下方式更改 shell 脚本

nohup sqoop export --connect jdbc:mysql://192.168.154.129:3306/ey -username root --table $TBL_NAME --export-dir $HIVE_DIR --fields-terminated-by "$FIELD_SEP" --lines-terminated-by '\'"$LINE_SEP" --m $NUM_MAPPERS --batch > $sqoop_outs/$TBL_NAME.out 2>&1 &

它将发出如下 SQOOP 命令，但它运行良好

sqoop export --connect jdbc:mysql://192.168.154.129:3306/ey -username root --table call_detail_records --export-dir /apps/hive/warehouse/xademo.db/call_detail_records --fields-terminated-by | --lines-terminated-by \n --m 4 --batch

Answer 2

这是为了导入

当你从 cli 运行 sqoop 命令时，选项的参数应该有 '，另一方面当你从 oozie 运行它不应包含在单个引号 '.

中

我正在使用带有以下参数的 sqoop fro、oozie：

<arg>--fields-terminated-by</arg>
<arg>'[=10=]1'</arg>
<arg>--null-string</arg>
<arg>'\N'</arg>
<arg>--null-non-string</arg>
<arg>'\N'</arg>

上面的代码没有按预期工作，但是下面的代码可以

<arg>--fields-terminated-by</arg>
<arg>[=11=]1</arg>
<arg>--null-string</arg>
<arg>\N</arg>
<arg>--null-non-string</arg>
<arg>\N</arg>

shell 脚本中的 SQOOP 导出失败

SQOOP export in shell script fails

shell

hadoop

sqoop

sqoop2