MonetDB 复制到 table "unexpected end of file"
MonetDB COPY INTO table "unexpected end of file"
我正在使用 COPY INTO 语句将一个相当大的 csv 文件 (300GB) 加载到 tables 中。经过漫长的等待时间后，我得到一个 "unexpected end of file" 异常和 table if if empty after that.
这是我的导入查询：
COPY d1 FROM '/home/d1_data/d1.csv'
还有我的csv数据文件：
2019-02-01T00:00:10,st0,0.839071,0.179288,0.585304,0.679371,0.492911,0.056175,0.498442,0.938126,0.668068,0.929086,0.081897,0.843644,0.974037,0.159324,0.142218,0.140207,0.625254,0.425917,0.771387,0.096174,0.120735,0.725770,0.139911,0.310633,0.382543,0.896953,0.445951,0.119868,0.424562,0.181185,0.379519,0.105958,0.845021,0.533097,0.723558,0.944910,0.036968,0.112205,0.799767,0.728473,0.968308,0.111421,0.905472,0.980631,0.865119,0.293025,0.973192,0.408123,0.272021,0.125133,0.763793,0.819480,0.600016,0.178615,0.777532,0.081147,0.652687,0.458067,0.767267,0.711449,0.957630,0.115871,0.569370,0.517578,0.093003,0.682874,0.679829,0.485540,0.926170,0.080369,0.570393,0.484541,0.568747,0.626574,0.117149,0.715187,0.655418,0.276893,0.841691,0.173985,0.805234,0.241210,0.858166,0.021120,0.224665,0.238334,0.864353,0.103404,0.868038,0.992483,0.624129,0.755107,0.620674,0.763600,0.199850,0.396798,0.612075,0.515486,0.961466,0.434988
2019-02-01T00:00:20,st0,0.322934,0.755268,0.061692,0.212437,0.231739,0.826009,0.402892,0.546866,0.748315,0.428897,0.634761,0.384299,0.192479,0.391302,0.920955,0.526497,0.150713,0.338057,0.933859,0.137499,0.875741,0.228530,0.297205,0.266878,0.288009,0.060985,0.882594,0.490286,0.870628,0.317989,0.476885,0.132587,0.459073,0.457800,0.380606,0.978631,0.687570,0.353860,0.224363,0.931935,0.272906,0.443753,0.908269,0.173270,0.567581,0.705271,0.659782,0.530196,0.615158,0.107020,0.337759,0.287402,0.113100,0.750601,0.380647,0.338062,0.470644,0.560054,0.916784,0.102615,0.653475,0.234832,0.241591,0.092253,0.984721,0.061122,0.418502,0.268967,0.170532,0.623880,0.505132,0.659034,0.752930,0.888594,0.871888,0.676820,0.938585,0.050625,0.063221,0.559219,0.451311,0.844238,0.915815,0.935894,0.918915,0.271461,0.099396,0.661230,0.405390,0.608056,0.919490,0.483303,0.240281,0.329818,0.181569,0.511471,0.432861,0.463347,0.560382,0.855283
2019-02-01T00:00:30,st0,0.692054,0.538778,0.764992,0.656943,0.006166,0.610429,0.479586,0.639454,0.107885,0.338176,0.535457,0.871265,0.291767,0.955159,0.271295,0.421824,0.772407,0.531340,0.419594,0.776071,0.452270,0.281994,0.479907,0.745093,0.627713,0.774344,0.699013,0.587567,0.878019,0.153955,0.986209,0.704153,0.783832,0.704486,0.200587,0.630304,0.235955,0.429266,0.752330,0.484207,0.394956,0.518921,0.688756,0.720469,0.056679,0.160093,0.502845,0.915870,0.359901,0.744948,0.005774,0.194809,0.180417,0.100580,0.428749,0.621978,0.782535,0.834345,0.960411,0.703126,0.681373,0.894144,0.943699,0.037323,0.294162,0.047351,0.940178,0.396505,0.243780,0.410479,0.257793,0.581372,0.235662,0.441054,0.536284,0.588570,0.946028,0.466676,0.124679,0.803133,0.713820,0.810444,0.810953,0.259700,0.450738,0.995637,0.339662,0.132606,0.189827,0.208749,0.430025,0.843661,0.706039,0.650623,0.797073,0.719763,0.055521,0.852340,0.396091,0.429506
知道为什么会发生这种异常吗？
当与服务器的连接由于开销而意外中断时，mclient 会发出“意外的文件结尾”错误。
问题是我同时 运行 负载太多。
我的机器在 Ubuntu 20.04 上有 32 个内核和 128GB RAM，所以我认为我对硬件资源是安全的，但事实并非如此。事实上，开销已经导致 mclient 崩溃。
对于 solve/avoid 这个问题，我建议牢记机器特性的任务开销，以及 运行 繁重的任务顺序而不是并行。
我正在使用 COPY INTO 语句将一个相当大的 csv 文件 (300GB) 加载到 tables 中。经过漫长的等待时间后，我得到一个 "unexpected end of file" 异常和 table if if empty after that.
这是我的导入查询：
COPY d1 FROM '/home/d1_data/d1.csv'
还有我的csv数据文件：
2019-02-01T00:00:10,st0,0.839071,0.179288,0.585304,0.679371,0.492911,0.056175,0.498442,0.938126,0.668068,0.929086,0.081897,0.843644,0.974037,0.159324,0.142218,0.140207,0.625254,0.425917,0.771387,0.096174,0.120735,0.725770,0.139911,0.310633,0.382543,0.896953,0.445951,0.119868,0.424562,0.181185,0.379519,0.105958,0.845021,0.533097,0.723558,0.944910,0.036968,0.112205,0.799767,0.728473,0.968308,0.111421,0.905472,0.980631,0.865119,0.293025,0.973192,0.408123,0.272021,0.125133,0.763793,0.819480,0.600016,0.178615,0.777532,0.081147,0.652687,0.458067,0.767267,0.711449,0.957630,0.115871,0.569370,0.517578,0.093003,0.682874,0.679829,0.485540,0.926170,0.080369,0.570393,0.484541,0.568747,0.626574,0.117149,0.715187,0.655418,0.276893,0.841691,0.173985,0.805234,0.241210,0.858166,0.021120,0.224665,0.238334,0.864353,0.103404,0.868038,0.992483,0.624129,0.755107,0.620674,0.763600,0.199850,0.396798,0.612075,0.515486,0.961466,0.434988
2019-02-01T00:00:20,st0,0.322934,0.755268,0.061692,0.212437,0.231739,0.826009,0.402892,0.546866,0.748315,0.428897,0.634761,0.384299,0.192479,0.391302,0.920955,0.526497,0.150713,0.338057,0.933859,0.137499,0.875741,0.228530,0.297205,0.266878,0.288009,0.060985,0.882594,0.490286,0.870628,0.317989,0.476885,0.132587,0.459073,0.457800,0.380606,0.978631,0.687570,0.353860,0.224363,0.931935,0.272906,0.443753,0.908269,0.173270,0.567581,0.705271,0.659782,0.530196,0.615158,0.107020,0.337759,0.287402,0.113100,0.750601,0.380647,0.338062,0.470644,0.560054,0.916784,0.102615,0.653475,0.234832,0.241591,0.092253,0.984721,0.061122,0.418502,0.268967,0.170532,0.623880,0.505132,0.659034,0.752930,0.888594,0.871888,0.676820,0.938585,0.050625,0.063221,0.559219,0.451311,0.844238,0.915815,0.935894,0.918915,0.271461,0.099396,0.661230,0.405390,0.608056,0.919490,0.483303,0.240281,0.329818,0.181569,0.511471,0.432861,0.463347,0.560382,0.855283
2019-02-01T00:00:30,st0,0.692054,0.538778,0.764992,0.656943,0.006166,0.610429,0.479586,0.639454,0.107885,0.338176,0.535457,0.871265,0.291767,0.955159,0.271295,0.421824,0.772407,0.531340,0.419594,0.776071,0.452270,0.281994,0.479907,0.745093,0.627713,0.774344,0.699013,0.587567,0.878019,0.153955,0.986209,0.704153,0.783832,0.704486,0.200587,0.630304,0.235955,0.429266,0.752330,0.484207,0.394956,0.518921,0.688756,0.720469,0.056679,0.160093,0.502845,0.915870,0.359901,0.744948,0.005774,0.194809,0.180417,0.100580,0.428749,0.621978,0.782535,0.834345,0.960411,0.703126,0.681373,0.894144,0.943699,0.037323,0.294162,0.047351,0.940178,0.396505,0.243780,0.410479,0.257793,0.581372,0.235662,0.441054,0.536284,0.588570,0.946028,0.466676,0.124679,0.803133,0.713820,0.810444,0.810953,0.259700,0.450738,0.995637,0.339662,0.132606,0.189827,0.208749,0.430025,0.843661,0.706039,0.650623,0.797073,0.719763,0.055521,0.852340,0.396091,0.429506
知道为什么会发生这种异常吗？
当与服务器的连接由于开销而意外中断时，mclient 会发出“意外的文件结尾”错误。 问题是我同时 运行 负载太多。
我的机器在 Ubuntu 20.04 上有 32 个内核和 128GB RAM，所以我认为我对硬件资源是安全的，但事实并非如此。事实上，开销已经导致 mclient 崩溃。
对于 solve/avoid 这个问题，我建议牢记机器特性的任务开销，以及 运行 繁重的任务顺序而不是并行。