Hadoop Pig Latin 加载数据总是失败

Question

我正在 Hadoop 的 Pig Latin 中迈出第一步，但我真的受阻了，因为我无法加载任何输入数据，即使它存在

R = LOAD '/home/cloudera/Desktop/vol.csv' USING PigStorage(';')  
AS 
(AnneeVol:int,MoisVol:int,JourVol:int,NumVol:int,AeroDep:chararray,AeroArriv:chararray,DistVol:int);

文件位置

根据运行 :

DUMP R;

我收到这个错误


> Failed Jobs:
JobId   Alias   Feature Message Outputs
job_1590934825774_0010  R   MAP_ONLY    Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://quickstart.cloudera:8020/home/cloudera/pig_lab/input/vol.csv
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:288)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:305)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)
    at org.apache.hadoop.mapreduce.Job.run(Job.java:1307)
    at org.apache.hadoop.mapreduce.Job.run(Job.java:1304)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
    at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
    at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
    at java.lang.Thread.run(Thread.java:745)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.run(MapReduceLauncher.java:270)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://quickstart.cloudera:8020/home/cloudera/pig_lab/input/vol.csv
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
    ... 18 more
    hdfs://quickstart.cloudera:8020/tmp/temp14790577/tmp132115611,


Input(s): Failed to read data from "/home/cloudera/pig_lab/input/vol.csv"

Output(s): Failed to produce result in "hdfs://quickstart.cloudera:8020/tmp/temp14790577/tmp132115611"

如何解决这个问题？谢谢！

Answer 1

该程序尝试在 HDFS 中找到 vol.csv 而不是在您的本地文件系统

Input path does not exist: hdfs://quickstart.cloudera:8020/home/cloudera/pig_lab/input/vol.csv

请检查 core-site.xml 您的默认文件系统。目前它的价值是 hdfs://quickstart.cloudera:8020。这就是它搜索文件 HDFS 的原因。您不必在那里进行任何更改。

只需在路径前添加 file:// 标记，告诉程序从本地文件系统中查找 vol.csv。

R = LOAD 'file:///home/cloudera/Desktop/vol.csv' USING PigStorage(';')  AS (AnneeVol:int,MoisVol:int,JourVol:int,NumVol:int,AeroDep:chararray,AeroArriv:chararray,DistVol:int);

参考：Cloudera blog

如果这没有帮助，要么将文件放在 HDFS 中，然后在您的代码中引用该位置。

hdfs dfs -put /home/cloudera/Desktop/vol.csv hdfs://quickstart.cloudera:8020/user/<hdfs-user>/

然后在你的代码中

R = LOAD '/user/<hdfs-user>/vol.csv' USING PigStorage(';')  AS (AnneeVol:int,MoisVol:int,JourVol:int,NumVol:int,AeroDep:chararray,AeroArriv:chararray,DistVol:int);

Hadoop Pig Latin 加载数据总是失败

Hadoop Pig Latin always fails to Load Data

hadoop

apache-pig