如何将带有空格的列名传递给 sqoop --map-column-java

Question

我必须使用 sqoop 导入数据，我的源列名称之间有空格，所以当我在 --map-column-java 参数中添加它时出现错误。

示例 Sqoop 导入：

sqoop import --connect jdbc-con --username "user1" --query "select * from table where $CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java data col1=String, data col2=String, data col3=String --as-avrodatafile

列名称：

data col1,
data col2,
data col3

错误：

19/03/07 07:31:55 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should be the form key=value[,key=value]* java.lang.IllegalArgumentException: Malformed mapping. Column mapping should be the form key=value[,key=value]* at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1355) at org.apache.sqoop.SqoopOptions.setMapColumnJava(SqoopOptions.java:1375) at org.apache.sqoop.tool.BaseSqoopTool.applyCodeGenOptions(BaseSqoopTool.java:1363) at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) at org.apache.sqoop.Sqoop.run(Sqoop.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Malformed mapping. Column mapping should be the form key=value[,key=value]*

Answer 1

能够解决这个问题：

1.空格问题： sqoop import --connect jdbc-con --username "user1" --query "select * from table where $CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile

2。错误 tool.ImportTool：导入失败：无法转换 SQL 类型 2005: 源中的 3 列有 2005 并且 nvarchar 在 --map-column-java 中添加了它们解决了这个问题

3。 org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: 不在联合 ["null","long"]: 1****

这是由于在 select 查询中使用了 * 造成的，因此将 sqoop 查询修改为：

sqoop import --connect jdbc-con --username "user1" --query "select [col1,data col2,data col3] from table where $CONDITIONS" --target-dir /target/path/ -m 1 --map -列-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile

Answer 2

你可以用这个方法代替我用过，效果很好

这里我将列转换为字符串，以便时间戳无法更改为 int

keep note of that point It will help you to make your string properly
address = <localhost/server-ip-address/>

port = put your database port number
Sqoop is expecting the comma-separated list of mapping in form 'name of column'='new type'
columns-name =  give your database column name of timestamp or date time to date


database-name = give your datbase name 

database-user-name = put your user name 

password = put your password 

演示正确理解代码

sqoop import  --map-column-java "columns-name=String"    --connect jdbc:postgresql://address:port/database-name     --username user-name     --password  database-password --query "select * from demo where $CONDITIONS;"  -m 1 --target-dir /jdbc/star --as-parquetfile  --enclosed-by '\"'

单列代码演示

sqoop import  --map-column-java "date_of_birth=String"    --connect jdbc:postgresql://192.168.0.1:1928/alpha     --username postgres     --password  mysecretpass --query "select * from demo where $CONDITIONS;"  -m 1 --target-dir /jdbc/star --as-parquetfile  --enclosed-by '\"'

处理多列的代码演示

sqoop import  --map-column-java "date_of_birth=String,create_date=String"    --connect jdbc:postgresql://192.168.0.1:1928/alpha     --username postgres     --password  mysecretpass --query "select * from demo where $CONDITIONS;"  -m 1 --target-dir /jdbc/star --as-parquetfile  --enclosed-by '\"'

如何将带有空格的列名传递给 sqoop --map-column-java

How to pass column names having spaces to sqoop --map-column-java

hadoop

sqoop

avro