将 csv 文件导入 greenplum 外部 table 时如何转义换行符？

Question

我正在尝试从 csv 创建外部 table，如下所示：

CREATE EXTERNAL TABLE hctest.ex_abs
(
a text,
b text,
c text,
d text,
e text,
f text,
g text
)
LOCATION ('gpfdist://192.168.56.111:10000/absdatasample.csv')
FORMAT 'CSV' (DELIMITER '|' HEADER);

csv 由竖线 (|) 分隔，如下所示：

Employee ID|Time Type|Start Date|End Date|Number Of Days|Comment|Ration of Leave
90007507|Leave|11/27/2020|11/27/2020|1|dear mas Andria,

seek for approval for 1 day off. Thank you.|8
90007507|Leave|05/08/2020|05/08/2020|1|dear mas Andria, kindly approve 1 day leave at 8th May. Thank you.|5
90006391|Leave|04/27/2020|04/30/2020|4|Requesting leave days for new baby born|7
90006988|Leave|04/20/2020|04/21/2020|2|Dear Mas Tommy,
Herewith I would like to ask your approval for my leave which will be taken on 20 - 21 April 2020 (2 days of leave). I take this leave because of I need to attend the family wedding out of town along with visiting my extended family before Ramadhan in my hometown. 

Your approval will be highly appreciated.

Thank you,
Andrian Indrawan|2
90005573|Leave|04/09/2020|04/09/2020|1||4
90007088|Leave|04/08/2020|04/09/2020|2||9
90004055|Leave|04/08/2020|04/09/2020|2|Leave for family's reason|6

我发现错误：

ERROR:  missing data for column "g"  (seg0 slice1 192.168.56.111:6000 pid=4486)
DETAIL:  External table ex_absdata, line 2 of gpfdist://192.168.56.111:10000/absdatasample.csv: "90007507|Leave|11/27/2020|11/27/2020|1|dear mas Andria,"

我该如何解决这个问题？

Answer 1

你有几个选择。

您可以在外部 Table 上使用 LOG ERRORS 选项，这样您就可以加载好数据并拒绝坏数据。但是文件中嵌入了很多换行符。
修复文件。

我找到了 this 个示例。

然后我拿了那个例子和你的样本文件。

awk -F\| '{ while (NF < 7 || $NF == "") { brokenline=[=10=]; getline; [=10=] = brokenline [=10=]}; print }' load.txt

Employee ID|Time Type|Start Date|End Date|Number Of Days|Comment|Ration of Leave
90007507|Leave|11/27/2020|11/27/2020|1|dear mas Andria,seek for approval for 1 day off. Thank you.|8
90007507|Leave|05/08/2020|05/08/2020|1|dear mas Andria, kindly approve 1 day leave at 8th May. Thank you.|5
90006391|Leave|04/27/2020|04/30/2020|4|Requesting leave days for new baby born|7
90006988|Leave|04/20/2020|04/21/2020|2|Dear Mas Tommy,Herewith I would like to ask your approval for my leave which will be taken on 20 - 21 April 2020 (2 days of leave). I take this leave because of I need to attend the family wedding out of town along with visiting my extended family before Ramadhan in my hometown. Your approval will be highly appreciated.Thank you,Andrian Indrawan|2
90005573|Leave|04/09/2020|04/09/2020|1||4
90007088|Leave|04/08/2020|04/09/2020|2||9
90004055|Leave|04/08/2020|04/09/2020|2|Leave for family's reason|6

另一种可能的修复方法是创建一个 perl、awk、sed、python 等脚本（如上）来遍历文件，使其成为带双引号的真正 CSV。如果这样做，您可以保留嵌入的换行符。我认为这太过分了，因为无论您从非结构化数据中获得什么价值，实际上都不需要换行符。

将 csv 文件导入 greenplum 外部 table 时如何转义换行符？

how to escape newline when importing csv file into greenplum external table?

linux

greenplum